(std rx)

Overview

Regular expressions written as s-expressions, checked and compiled at compile time.

(import (std rx))

(def phone #rx(seq :bos
                   (=> :area (= 3 :ascii-digit))   ; three digits, named :area
                   (lit "-")
                   (=> :line (= 4 :ascii-digit))   ; four digits, named :line
                   :eos))

(rx-match? phone "555-1234")          ; #t
(rx-search #rx((+ :digit)) "a42b")    ; (Some <match on "42">)

A pattern is written #rx(...) in a notation like Scheme’s SRE, not as a string such as "[a-z]+", so nothing is ever escaped twice. It is read, checked and compiled to a .NET pattern at compile time, and a bad pattern is a compile error pointing at the sub-form that is wrong. The pattern language is described with #rx.

Using a pattern. A #rx(...) is an Rx, a compiled regular expression. Compiling is cached by pattern, so evaluating the same #rx(...) again does not build a second .NET regex. rx-match? and rx-match match the whole string; rx-search? and rx-search find the leftmost match anywhere in it; rx-matches finds them all. A match is an RxMatch, read with rx-text, rx-start, rx-end and the rx-group family. rx-replace, rx-replace-with and rx-split rewrite a string. In a match, the patterns Rx and RxIn do the same as rx-match and rx-search.

The engine. The matching is System.Text.RegularExpressions in its non-backtracking mode, which matches in time linear in the length of the string, and the options are fixed: NonBacktracking, ExplicitCapture, CultureInvariant. That mode refuses backreferences, lookarounds, atomic groups and conditionals, the constructs that would tie Bjolang to a backtracking engine for good. So these are not supported, on purpose:

Lookarounds, backreferences and conditionals.
Complementing a whole pattern. A character set can be complemented: (~ :digit).
Case-insensitive matching, which would depend on the runtime’s culture tables.
Patterns built at run time. A pattern is built at compile time, so ,x inside #rx(...) is an error: every pattern is known statically.

Unicode. The alphabet is the Unicode scalar, as it is for char and the string cursors, but .NET’s engine works in UTF-16 code units. So :any and every complemented set are emitted as an alternation that takes a surrogate pair as one unit, and a match never begins or ends inside a pair: an emoji is not split in half.

.NET’s character classes cannot name a scalar above U+FFFF, so a character set cannot hold one: (in "😀") is a compile error. Match it as a literal, "😀" or (lit "😀"), instead.

The emitted pattern never uses ., ^, $, \d, \w, \s or \p{...}, whose meanings .NET decides and which move when the runtime’s Unicode tables are updated. The named sets are written out as explicit ranges from tables in rx.bjo, generated from .NET’s CharUnicodeInfo so that :alpha agrees with char-alphabetic?. A compiled program means the same thing on every runtime.

Reference

Types

`RxMatch`

type alias

One match: the string it was found in, where, and its groups.

Functions

`rx-match`

function

(: rx-match (-> Rx string (Option RxMatch)))
(rx-match rx s)

Matches a pattern against the whole string.

rx: The pattern.
s: The string.
returns: The match, anchored at both ends, or None.

(match (rx-match phone "555-1234")
  ((Some m) (rx-group m :area))   ; (Some "555")
  (None None))

See also: rx-match?, rx-search, Rx

`rx-match?`

function

(: rx-match? (-> Rx string bool))
(rx-match? rx s)

Whether a pattern matches the whole string.

rx: The pattern.
s: The string.
returns: #t if the whole of s matches.

(rx-match? #rx((+ :ascii-digit)) "123")    ; #t
(rx-match? #rx((+ :ascii-digit)) "123a")   ; #f

`rx-search`

function

(: rx-search (-> Rx string (Option RxMatch)))
(rx-search rx s)

The leftmost match anywhere in the string.

rx: The pattern.
s: The string.
returns: The leftmost match, or None.

(rx-search #rx((+ :digit)) "a42b")   ; (Some <match on "42">)

`rx-search?`

function

(: rx-search? (-> Rx string bool))
(rx-search? rx s)

Whether a pattern matches anywhere in the string.

rx: The pattern.
s: The string.
returns: #t if some part of s matches.

`rx-matches`

function

(: rx-matches (-> Rx string (Vec RxMatch)))
(rx-matches rx s)

Every non-overlapping match, left to right.

rx: The pattern.
s: The string.
returns: The matches, in order; empty if there are none.

(vec-map rx-text (rx-matches #rx((+ :digit)) "a1b22c333"))   ; ["1" "22" "333"]

Each search starts where the last match ended. After an empty match it starts one character on, a whole surrogate pair if that is what is there, so it is never inside one.

`rx-replace`

function

(: rx-replace (-> Rx string string string))
(rx-replace rx s replacement)

The string with every match replaced by a fixed text.

rx: The pattern.
s: The string.
replacement: The text to put in place of each match, taken literally.
returns: The new string.

(rx-replace #rx((+ :space)) "a  b   c" " ")   ; "a b c"

There is no $1 substitution syntax: that would be a second pattern language, arriving without being decided on. Use rx-replace-with to build the replacement from the match.

`rx-replace-with`

function

(: rx-replace-with (-> Rx string (-> RxMatch string) string))
(rx-replace-with rx s f)

The string with every match replaced by what a function makes of it.

rx: The pattern.
s: The string.
f: Given each match, answers its replacement.
returns: The new string.

(rx-replace-with #rx(($ (+ :ascii-digit)))
                 "3 and 4"
                 (fun (m) (str "<" (rx-text m) ">")))   ; "<3> and <4>"

`rx-split`

function

(: rx-split (-> Rx string (Vec string)))
(rx-split rx s)

The pieces of the string between the matches.

rx: The separator pattern.
s: The string.
returns: The pieces, in order. The empty pieces before a leading and after a trailing separator are kept; with no match, the whole string.

(rx-split #rx(seq "," (* :space)) "a, b,c")   ; ["a" "b" "c"]

An empty match is not a separator, so a pattern that can match nothing does not split between every character.

`rx-text`

function

(: rx-text (-> RxMatch string))
(rx-text m)

The text a match matched.

m: The match.
returns: The matched part of the string.

`rx-input`

function

(: rx-input (-> RxMatch string))
(rx-input m)

The string a match was found in.

m: The match.
returns: The whole string that was searched, which its cursors index.

`rx-start`

function

(: rx-start (-> RxMatch StringCursor))
(rx-start m)

Where a match begins.

m: The match.
returns: A cursor into rx-input, on the match's first character.
raises System.InvalidOperationException: When the position falls inside a surrogate pair, which only a pattern written by hand for rx-compile can bring about.

`rx-end`

function

(: rx-end (-> RxMatch StringCursor))
(rx-end m)

Where a match ends.

m: The match.
returns: A cursor into rx-input, just after the match's last character.
raises System.InvalidOperationException: When the position falls inside a surrogate pair, which only a pattern written by hand for rx-compile can bring about.

`rx-group`

function

(: rx-group (-> RxMatch Keyword (Option string)))
(rx-group m name)

The text a named group captured.

m: The match.
name: The keyword the group was named with, in (=> :name ...).
returns: The captured text, or None if the group took no part in the match. An unknown name is None too.

(rx-group m :area)   ; (Some "555")

`rx-group-start`

function

(: rx-group-start (-> RxMatch Keyword (Option StringCursor)))
(rx-group-start m name)

Where a named group’s capture begins.

m: The match.
name: The keyword the group was named with.
returns: A cursor into rx-input, or None if the group took no part in the match or there is no such group.
raises System.InvalidOperationException: When the position falls inside a surrogate pair, which only a pattern written by hand for rx-compile can bring about.

`rx-group-end`

function

(: rx-group-end (-> RxMatch Keyword (Option StringCursor)))
(rx-group-end m name)

Where a named group’s capture ends.

m: The match.
name: The keyword the group was named with.
returns: A cursor into rx-input just after the capture, or None if the group took no part in the match or there is no such group.
raises System.InvalidOperationException: When the position falls inside a surrogate pair, which only a pattern written by hand for rx-compile can bring about.

`rx-group/n`

function

(: rx-group/n (-> RxMatch int (Option string)))
(rx-group/n m n)

The text a group captured, by its number.

m: The match.
n: The group's number, counted from zero in the order the groups open. Named and unnamed groups are counted alike.
returns: The captured text, or None if the group took no part in the match or there is no such group.

(match (rx-search #rx(seq ($ (+ :ascii-alpha)) "=" ($ (+ :ascii-digit))) "x=42")
  ((Some m) (rx-group/n m 1))   ; (Some "42")
  (None None))

`rx-group-count`

function

(: rx-group-count (-> RxMatch int))
(rx-group-count m)

How many groups the pattern has.

m: The match.
returns: The number of groups, named and unnamed, whether or not they took part in this match.

`rx-compile`

function

(: rx-compile (-> string string Rx))
(rx-compile pattern names)

Compiles a .NET pattern into an Rx: what #rx(…) expands into.

pattern: The .NET pattern, as #rx writes it.
names: The Bjolang name of each group in order, separated by commas. An unnamed group has an empty name.
returns: The compiled regular expression.
raises System.ArgumentException: When pattern is not a valid .NET pattern.
raises System.NotSupportedException: When pattern uses a construct the non-backtracking engine refuses, such as a backreference or a lookaround.

An Rx is a compiled regular expression. The options are fixed: NonBacktracking, ExplicitCapture and CultureInvariant. Compiling is cached by the pattern and the names, so the same pattern compiled twice is one regex.

It is there for #rx, which checks what it emits. A pattern passed here by hand is not checked, and is not held to the promises the module makes: it may use ., \d and the rest, and may let a match boundary fall inside a surrogate pair. Commas cannot occur in a keyword, so they cannot occur in a name.

`rx-pattern`

function

(: rx-pattern (-> Rx string))
(rx-pattern rx)

The .NET pattern an Rx was compiled from.

rx: The compiled regular expression.
returns: The pattern string, so that a test can check what was emitted and not only what it matches.

Macros

`#rx`

reader extension

#rx(pattern)

A regular expression, checked and compiled at compile time.

pattern: The pattern, in the notation below.

#rx(seq :bos (+ :digit) (lit "-") ($ (** 4 4 :digit)) :eos)
#rx(:digit)              ; one argument is the pattern itself
#rx((+ :digit))
#rx(+ :digit)            ; and the same as #rx((+ :digit))

It expands to (rx-compile "<pattern>" "<names>"), with the .NET pattern and the group names worked out at compile time. Its value is an Rx. #rx(seq a b) reads as (#rx seq a b), so a form may be written with or without its own parentheses.

Two spelling rules. Anything that takes no argument is a keyword: :bos, :any, :digit. Anything that takes arguments is a form headed by a symbol: (seq ...), (+ ...), (in ...). A string or a character is itself: a literal sequence in pattern position, the set of its characters inside a set.

Pattern	Matches
`"text"`, `#\c`	Exactly that text or character.
`(lit "text")`	Exactly that text. It takes one string or character.
`(seq a b ...)`	`a`, then `b`, and so on.
`(or a b ...)`	`a` or `b` or …
`(* r)`, `(+ r)`, `(? r)`	Zero or more, one or more, or zero or one of `r`.
`(*? r)`, `(+? r)`, `(?? r)`	The same, lazily: as few as will do.
`(** least most r)`	`r`, from `least` to `most` times.
`(= n r)`, `(>= n r)`	`r` exactly `n` times, or at least `n` times.
`($ r ...)`	`r`, captured in an unnamed group.
`(=> :name r ...)`	`r`, captured in a group named by the keyword.
`:bos`, `:eos`	The beginning and the end of the string.
`:any`	Any one character, newlines included.
`:nonl`	Any one character but U+000A, the only line break it knows.
a named set, or a set form	One character of the set.

A repetition, $ or => given several patterns takes them in sequence: (+ "a" "b") is (+ (seq "a" "b")). Counts are whole numbers. Groups are numbered from zero in the order they open.

Character sets match one character. They are worked out at compile time, and a set that can match nothing is an error.

Set	Holds
`#\c`, `"abc"`	Those characters.
`(in a b ...)`	Every character in any of the sets; `(or a b ...)` inside a set is the same.
`(/ lo hi ...)`	The ranges from `lo` to `hi`, the endpoints taken in pairs.
`(~ a b ...)`	Every character in none of the sets.
`(- a b ...)`	The characters of `a` that are in none of the others.
`(& a b ...)`	The characters in all of the sets.

(/ #\a #\z #\0 #\9) and (/ "az09") are the same set, the lowercase letters and the digits: the endpoints are taken in pairs however they are written. A range runs upwards. A set member or a range endpoint is a character in U+0000 to U+FFFF; above that, match a literal instead.

The named sets are keywords. Where two names are given they are the same set.

Name	Holds
`:alpha`, `:alphabetic`	Letters, in all of Unicode.
`:digit`, `:numeric`	Every Unicode decimal digit, as `char-numeric?` answers.
`:alnum`, `:alphanumeric`	Letters and decimal digits.
`:upper`, `:upper-case`	Uppercase letters.
`:lower`, `:lower-case`	Lowercase letters.
`:space`, `:whitespace`	Whitespace: U+0009 to U+000D, the space, and Unicode’s others.
`:word`	Letters, digits, marks and connector punctuation such as `_`.
`:control`	U+0000 to U+001F and U+007F to U+009F.
`:hex-digit`	`0` to `9`, `A` to `F` and `a` to `f`.
`:ascii`	U+0000 to U+007F.
`:ascii-digit`	`0` to `9`.
`:ascii-alpha`	`A` to `Z` and `a` to `z`.
`:ascii-alnum`	The ASCII letters and digits.
`:any`	Every character.
`:nonl`	Every character but U+000A.

Mind :digit. It holds every Unicode decimal digit, such as ٩ and ०, not only 0 to 9. A pattern for the ASCII digits wants :ascii-digit, and the same goes for :alpha and :alnum.

`Rx`

macro

(Rx regex pattern)

A match pattern: matches a regular expression against the whole string.

regex: The regular expression: an Rx, such as a #rx(...) or a name bound to one.
pattern: Matched against the (Option RxMatch) that rx-match answers.

(match line
  ((Rx phone (Some m)) (str "phone: " (option-ref-or (rx-group m :area) "?")))
  ((RxIn #rx((+ :digit)) (Some m)) (str "a number: " (rx-text m)))
  (_ "neither"))

Rx is also the name of the type of a compiled regular expression, which rx-compile describes.

A view covers nothing for exhaustiveness, so a match using Rx or RxIn still needs a clause that matches anything.

The regular expression is evaluated each time the clause is tried. The compiled regex is cached, but a #rx(...) written in the clause is still looked up anew each time, so bind the pattern once, at the top level, and name it in the clause:

(def digits #rx((+ :digit)))

(match text
  ((RxIn digits (Some m)) (rx-text m))
  (_ "none"))

`RxIn`

macro

(RxIn regex pattern)

A match pattern: searches the string for a regular expression.

regex: The regular expression: an Rx, such as a #rx(...) or a name bound to one.
pattern: Matched against the (Option RxMatch) that rx-search answers.

As Rx, but finding the leftmost match anywhere in the string. A match using it still needs a clause that matches anything.