1 Lexers
1.1 Creating a Lexer
(lexer [trigger action-expr] ...)
|
| |
| trigger | | = | | re | | | | | | | (eof) | | | | | | | (special) | | | | | | | (special-comment) | | | | | | | | re | | = | | id | | | | | | | string | | | | | | | character | | | | | | | (repetition lo hi re) | | | | | | | (union re ...) | | | | | | | (intersection re ...) | | | | | | | (complement re) | | | | | | | (concatenation re ...) | | | | | | | (char-range char char) | | | | | | | (char-complement re) | | | | | | | (id datum ...) |
|
Produces a function that takes an input-port, matches the
+ re patterns against the buffer, and returns the result of
+ executing the corresponding action-expr. When multiple
+ patterns match, a lexer will choose the longest match, breaking
+ ties in favor of the rule appearing first.
The implementation of syntax-color/racket-lexer
+contains a lexer for the racket language.
+In addition, files in the "examples" sub-directory
+of the "br-parser-tools" collection contain
+simpler example lexers.
An re is matched as follows:
id — expands to the named lexer abbreviation;
+abbreviations are defined via define-lex-abbrev or supplied by modules
+like br-parser-tools/lex-sre.
string — matches the sequence of characters in string.
character — matches a literal character.
(repetition lo hi re) — matches re repeated between lo
+and hi times, inclusive; hi can be +inf.0 for unbounded repetitions.
(union re ...) — matches if any of the sub-expressions match
(intersection re ...) — matches if all of the res match.
(complement re) — matches anything that re does not.
(concatenation re ...) — matches each re in succession.
(char-range char char) — matches any character between the two (inclusive);
+a single character string can be used as a char.
(char-complement re) — matches any character not matched by re.
+The sub-expression must be a set of characters re.
(id datum ...) — expands the lexer macro named id; macros
+are defined via define-lex-trans.
Note that both (concatenation) and "" match the
+empty string, (union) matches nothing,
+(intersection) matches any string, and
+(char-complement (union)) matches any single character.
The regular expression language is not designed to be used directly,
+but rather as a basis for a user-friendly notation written with
+regular expression macros. For example,
+br-parser-tools/lex-sre supplies operators from Olin
+Shivers’s SREs, and br-parser-tools/lex-plt-v200 supplies
+(deprecated) operators from the previous version of this library.
+Since those libraries provide operators whose names match other Racket
+bindings, such as * and +, they normally must be
+imported using a prefix:
(require (prefix-in : br-parser-tools/lex-sre))
The suggested prefix is :, so that :* and
+:+ are imported. Of course, a prefix other than :
+(such as re-) will work too.
Since negation is not a common operator on regular expressions, here
+are a few examples, using : prefixed SRE syntax:
(complement "1")
Matches all strings except the string "1", including
+"11", "111", "0", "01",
+"", and so on.
(complement (:* "1"))
Matches all strings that are not sequences of "1",
+including "0", "00", "11110",
+"0111", "11001010" and so on.
Matches all strings that have 3 consecutive ones, but not those that
+end in "01" and not those that are ones only. These
+include "1110", "0001000111" and "0111"
+but not "", "11", "11101", "111"
+and "11111".
(:: "/*" (complement (:: any-string "*/" any-string)) "*/")
Matches Java/C block comments. "/**/",
+"/******/", "/*////*/", "/*asg4*/" and so
+on. It does not match "/**/*/", "/* */ */" and so
+on. (:: any-string "*/" any-string) matches any string
+that has a "*/" in is, so (complement (:: any-string "*/" any-string)) matches any string without a "*/" in it.
(:: "/*" (:* (complement "*/")) "*/")
Matches any string that starts with "/*" and ends with
+"*/", including "/* */ */ */".
+(complement "*/") matches any string except "*/".
+This includes "*" and "/" separately. Thus
+(:* (complement "*/")) matches "*/" by first
+matching "*" and then matching "/". Any other
+string is matched directly by (complement "*/"). In other
+words, (:* (complement "xx")) = any-string. It is
+usually not correct to place a :* around a
+complement.
The following binding have special meaning inside of a lexer
+ action:
start-pos — a position struct for the first character matched.
end-pos — a position struct for the character after the last character in the match.
lexeme — the matched string.
input-port — the input-port being
+processed (this is useful for matching input with multiple
+lexers).
(return-without-pos x) is a function (continuation) that
+immediately returns the value of x from the lexer. This useful
+in a src-pos lexer to prevent the lexer from adding source
+information. For example:
would wrap the source location information for the comment around
+the value of the recursive call. Using
+((comment) (return-without-pos (get-token input-port)))
+will cause the value of the recursive call to be returned without
+wrapping position around it.
The lexer raises an exception (exn:read) if none of the
+ regular expressions match the input. Hint: If (any-char custom-error-behavior) is the last rule, then there will always
+ be a match, and custom-error-behavior is executed to
+ handle the error situation as desired, only consuming the first
+ character from the input buffer.
In addition to returning characters, input
+ ports can return eof-objects. Custom input ports can
+ also return a special-comment value to indicate a
+ non-textual comment, or return another arbitrary value (a
+ special). The non-re trigger forms handle these
+ cases:
The (eof) rule is matched when the input port
+returns an eof-object value. If no (eof)
+rule is present, the lexer returns the symbol 'eof
+when the port returns an eof-object value.
The (special-comment) rule is matched when the
+input port returns a special-comment structure. If no
+special-comment rule is present, the lexer
+automatically tries to return the next token from the input
+port.
The (special) rule is matched when the input
+port returns a value other than a character,
+eof-object, or special-comment structure. If
+no (special) rule is present, the lexer returns
+(void).
End-of-files, specials, special-comments and special-errors cannot
+ be parsed via a rule using an ordinary regular expression
+ (but dropping down and manipulating the port to handle them
+ is possible in some situations).
Since the lexer gets its source information from the port, use
+ port-count-lines! to enable the tracking of line and
+ column information. Otherwise, the line and column information
+ will return #f.
When peeking from the input port raises an exception (such as by
+ an embedded XML editor with malformed syntax), the exception can
+ be raised before all tokens preceding the exception have been
+ returned.
Each time the racket code for a lexer is compiled (e.g. when a
+ ".rkt" file containing a lexer form is loaded),
+ the lexer generator is run. To avoid this overhead place the
+ lexer into a module and compile the module to a ".zo"
+ bytecode file.
Use of these names outside of a
lexer action is a syntax
+error.
Instances of
position are bound to
start-pos and
+
end-pos. The
offset field contains the offset of
+the character in the input. The
line field contains the
+line number of the character. The
col field contains the
+offset in the current line.
A parameter that the lexer uses as the source location if it
+raises a
exn:fail:read error. Setting this parameter allows
+DrRacket, for example, to open the file containing the error.
1.2 Lexer Abbreviations and Macros
Defines a
lexer abbreviation by associating a regular
+expression to be used in place of the
id in other
+regular expression. The definition of name has the same scoping
+properties as a other syntactic binding (e.g., it can be exported
+from a module).
Defines a
lexer macro, where
trans-expr produces a
+transformer procedure that takes one argument. When
(id datum ...) appears as a regular expression, it is replaced with
+the result of applying the transformer to the expression.
1.3 Lexer SRE Operators
Repetition of re sequence 0 or more times.
Repetition of re sequence 1 or more times.
Zero or one occurrence of re sequence.
Exactly n occurrences of re sequence, where
+n must be a literal exact, non-negative number.
At least n occurrences of re sequence, where
+n must be a literal exact, non-negative number.
Between n and m (inclusive) occurrences of
+re sequence, where n must be a literal exact,
+non-negative number, and m must be literally either
+#f, +inf.0, or an exact, non-negative number; a
+#f value for m is the same as +inf.0.
Both forms concatenate the res.
Intersects the res.
The set difference of the res.
Character-set complement, which each re must match exactly
+one character.
Character ranges, matching characters between successive pairs of
+characters.
1.4 Lexer Legacy Operators
The same as
(complement re ...).
1.5 Tokens
Each action-expr in a lexer form can produce any
+kind of value, but for many purposes, producing a token
+value is useful. Tokens are usually necessary for inter-operating with
+a parser generated by br-parser-tools/parser, but tokens may not
+be the right choice when using lexer in other situations.
Binds
group-id to the group of tokens being defined. For
+each
token-id, a function
+
token-token-id is created that takes any
+value and puts it in a token record specific to
token-id.
+The token value is inspected using
token-id and
+
token-value.
A token cannot be named error, since
+error it has special use in the parser.
Like define-tokens, except a each token constructor
+token-token-id takes no arguments and returns
+(quote token-id).
Returns the name of a token that is represented either by a symbol
+or a token structure.
Returns the value of a token that is represented either by a symbol
+or a token structure, returning #f for a symbol token.
Returns #t if val is a
+token structure, #f otherwise.