mbutterick
/
br-parser-tools
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

A ton of `@scheme*' -> `@racket*' and related updates.

Browse Source 
Also, updates some of the mzlib files to point at `racket/*' libraries
rather than to `scheme/*' ones.

original commit: ac26fe75546b5182d78f18c2cd882f0f440849e2
tokens
Eli Barzilay 14 years ago
parent 861d0d6110
commit 616c2649fd
1 changed files with 225 additions and 228 deletions
Show Stats Download Patch File Download Diff File

453
collects/parser-tools/parser-tools.scrbl
Unescape Escape View File

@ -1,8 +1,5 @@
#lang scribble/doc #lang scribble/doc
@(require scribble/manual @(require scribble/manual scribble/struct scribble/xref scribble/bnf
scribble/struct
scribble/xref
scribble/bnf
(for-label scheme/base (for-label scheme/base
scheme/contract scheme/contract
parser-tools/lex parser-tools/lex
@ -53,107 +50,107 @@ style lexer and parser generators.
(id datum ...)])]{ (id datum ...)])]{
Produces a function that takes an input-port, matches the Produces a function that takes an input-port, matches the
@scheme[re]'s against the buffer, and returns the result of @racket[re]'s against the buffer, and returns the result of
executing the corresponding @scheme[action-expr]. executing the corresponding @racket[action-expr].
@margin-note{The implementation of @schememodname[syntax-color/scheme-lexer] @margin-note{The implementation of @racketmodname[syntax-color/scheme-lexer]
contains a lexer for the @schememodname[scheme] language. contains a lexer for the @racketmodname[racket] language.
In addition, files in the @filepath{examples} sub-directory In addition, files in the @filepath{examples} sub-directory
of the @filepath{parser-tools} collection contain of the @filepath{parser-tools} collection contain
simpler example lexers.} simpler example lexers.}
An @scheme[re] is matched as follows: An @racket[re] is matched as follows:
@itemize[ @itemize[
@item{@scheme[id] --- expands to the named @deftech{lexer abbreviation}; @item{@racket[id] --- expands to the named @deftech{lexer abbreviation};
abbreviations are defined via @scheme[define-lex-abbrev] or supplied by modules abbreviations are defined via @racket[define-lex-abbrev] or supplied by modules
like @schememodname[parser-tools/lex-sre].} like @racketmodname[parser-tools/lex-sre].}
@item{@scheme[string] --- matches the sequence of characters in @scheme[string].} @item{@racket[string] --- matches the sequence of characters in @racket[string].}
@item{@scheme[character] --- matches a literal @scheme[character].} @item{@racket[character] --- matches a literal @racket[character].}
@item{@scheme[(repetition lo hi re)] --- matches @scheme[re] repeated between @scheme[lo] @item{@racket[(repetition lo hi re)] --- matches @racket[re] repeated between @racket[lo]
and @scheme[hi] times, inclusive; @scheme[hi] can be @scheme[+inf.0] for unbounded repetitions.} and @racket[hi] times, inclusive; @racket[hi] can be @racket[+inf.0] for unbounded repetitions.}
@item{@scheme[(union re ...)] --- matches if any of the sub-expressions match} @item{@racket[(union re ...)] --- matches if any of the sub-expressions match}
@item{@scheme[(intersection re ...)] --- matches if all of the @scheme[re]s match.} @item{@racket[(intersection re ...)] --- matches if all of the @racket[re]s match.}
@item{@scheme[(complement re)] --- matches anything that @scheme[re] does not.} @item{@racket[(complement re)] --- matches anything that @racket[re] does not.}
@item{@scheme[(concatenation re ...)] --- matches each @scheme[re] in succession.} @item{@racket[(concatenation re ...)] --- matches each @racket[re] in succession.}
@item{@scheme[(char-range char char)] --- matches any character between the two (inclusive); @item{@racket[(char-range char char)] --- matches any character between the two (inclusive);
a single character string can be used as a @scheme[char].} a single character string can be used as a @racket[char].}
@item{@scheme[(char-complement re)] --- matches any character not matched by @scheme[re]. @item{@racket[(char-complement re)] --- matches any character not matched by @racket[re].
The sub-expression must be a set of characters @scheme[re].} The sub-expression must be a set of characters @racket[re].}
@item{@scheme[(id datum ...)] --- expands the @deftech{lexer macro} named @scheme[id]; macros @item{@racket[(id datum ...)] --- expands the @deftech{lexer macro} named @racket[id]; macros
are defined via @scheme[define-lex-trans].} are defined via @racket[define-lex-trans].}
] ]
Note that both @scheme[(concatenation)] and @scheme[""] match the Note that both @racket[(concatenation)] and @racket[""] match the
empty string, @scheme[(union)] matches nothing, empty string, @racket[(union)] matches nothing,
@scheme[(intersection)] matches any string, and @racket[(intersection)] matches any string, and
@scheme[(char-complement (union))] matches any single character. @racket[(char-complement (union))] matches any single character.
The regular expression language is not designed to be used directly, The regular expression language is not designed to be used directly,
but rather as a basis for a user-friendly notation written with but rather as a basis for a user-friendly notation written with
regular expression macros. For example, regular expression macros. For example,
@schememodname[parser-tools/lex-sre] supplies operators from Olin @racketmodname[parser-tools/lex-sre] supplies operators from Olin
Shivers's SREs, and @schememodname[parser-tools/lex-plt-v200] supplies Shivers's SREs, and @racketmodname[parser-tools/lex-plt-v200] supplies
(deprecated) operators from the previous version of this library. (deprecated) operators from the previous version of this library.
Since those libraries provide operators whose names match other Scheme Since those libraries provide operators whose names match other Racket
bindings, such as @scheme[*] and @scheme[+], they normally must be bindings, such as @racket[*] and @racket[+], they normally must be
imported using a prefix: imported using a prefix:
@schemeblock[ @racketblock[
(require (prefix-in : parser-tools/lex-sre)) (require (prefix-in : parser-tools/lex-sre))
] ]
The suggested prefix is @scheme[:], so that @scheme[:*] and The suggested prefix is @racket[:], so that @racket[:*] and
@scheme[:+] are imported. Of course, a prefix other than @scheme[:] @racket[:+] are imported. Of course, a prefix other than @racket[:]
(such as @scheme[re-]) will work too. (such as @racket[re-]) will work too.
Since negation is not a common operator on regular expressions, here Since negation is not a common operator on regular expressions, here
are a few examples, using @scheme[:] prefixed SRE syntax: are a few examples, using @racket[:] prefixed SRE syntax:
@itemize[ @itemize[
@item{@schemeblock0[(complement "1")] @item{@racketblock0[(complement "1")]
Matches all strings except the string @scheme["1"], including Matches all strings except the string @racket["1"], including
@scheme["11"], @scheme["111"], @scheme["0"], @scheme["01"], @racket["11"], @racket["111"], @racket["0"], @racket["01"],
@scheme[""], and so on.} @racket[""], and so on.}
@item{@schemeblock0[(complement (:* "1"))] @item{@racketblock0[(complement (:* "1"))]
Matches all strings that are not sequences of @scheme["1"], Matches all strings that are not sequences of @racket["1"],
including @scheme["0"], @scheme["00"], @scheme["11110"], including @racket["0"], @racket["00"], @racket["11110"],
@scheme["0111"], @scheme["11001010"] and so on.} @racket["0111"], @racket["11001010"] and so on.}
@item{@schemeblock0[(:& (:: any-string "111" any-string) @item{@racketblock0[(:& (:: any-string "111" any-string)
(complement (:or (:: any-string "01") (:+ "1"))))] (complement (:or (:: any-string "01") (:+ "1"))))]
Matches all strings that have 3 consecutive ones, but not those that Matches all strings that have 3 consecutive ones, but not those that
end in @scheme["01"] and not those that are ones only. These end in @racket["01"] and not those that are ones only. These
include @scheme["1110"], @scheme["0001000111"] and @scheme["0111"] include @racket["1110"], @racket["0001000111"] and @racket["0111"]
but not @scheme[""], @scheme["11"], @scheme["11101"], @scheme["111"] but not @racket[""], @racket["11"], @racket["11101"], @racket["111"]
and @scheme["11111"].} and @racket["11111"].}
@item{@schemeblock0[(:: "/*" (complement (:: any-string "*/" any-string)) "*/")] @item{@racketblock0[(:: "/*" (complement (:: any-string "*/" any-string)) "*/")]
Matches Java/C block comments. @scheme["/**/"], Matches Java/C block comments. @racket["/**/"],
@scheme["/******/"], @scheme["/*////*/"], @scheme["/*asg4*/"] and so @racket["/******/"], @racket["/*////*/"], @racket["/*asg4*/"] and so
on. It does not match @scheme["/**/*/"], @scheme["/* */ */"] and so on. It does not match @racket["/**/*/"], @racket["/* */ */"] and so
on. @scheme[(:: any-string "*/" any-string)] matches any string on. @racket[(:: any-string "*/" any-string)] matches any string
that has a @scheme["*/"] in is, so @scheme[(complement (:: any-string "*/" that has a @racket["*/"] in is, so @racket[(complement (:: any-string "*/"
any-string))] matches any string without a @scheme["*/"] in it.} any-string))] matches any string without a @racket["*/"] in it.}
@item{@schemeblock0[(:: "/*" (:* (complement "*/")) "*/")] @item{@racketblock0[(:: "/*" (:* (complement "*/")) "*/")]
Matches any string that starts with @scheme["/*"] and ends with Matches any string that starts with @racket["/*"] and ends with
@scheme["*/"], including @scheme["/* */ */ */"]. @racket["*/"], including @racket["/* */ */ */"].
@scheme[(complement "*/")] matches any string except @scheme["*/"]. @racket[(complement "*/")] matches any string except @racket["*/"].
This includes @scheme["*"] and @scheme["/"] separately. Thus This includes @racket["*"] and @racket["/"] separately. Thus
@scheme[(:* (complement "*/"))] matches @scheme["*/"] by first @racket[(:* (complement "*/"))] matches @racket["*/"] by first
matching @scheme["*"] and then matching @scheme["/"]. Any other matching @racket["*"] and then matching @racket["/"]. Any other
string is matched directly by @scheme[(complement "*/")]. In other string is matched directly by @racket[(complement "*/")]. In other
words, @scheme[(:* (complement "xx"))] = @scheme[any-string]. It is words, @racket[(:* (complement "xx"))] = @racket[any-string]. It is
usually not correct to place a @scheme[:*] around a usually not correct to place a @racket[:*] around a
@scheme[complement].} @racket[complement].}
] ]
@ -161,18 +158,18 @@ are a few examples, using @scheme[:] prefixed SRE syntax:
action: action:
@itemize[ @itemize[
@item{@scheme[start-pos] --- a @scheme[position] struct for the first character matched.} @item{@racket[start-pos] --- a @racket[position] struct for the first character matched.}
@item{@scheme[end-pos] --- a @scheme[position] struct for the character after the last character in the match.} @item{@racket[end-pos] --- a @racket[position] struct for the character after the last character in the match.}
@item{@scheme[lexeme] --- the matched string.} @item{@racket[lexeme] --- the matched string.}
@item{@scheme[input-port] --- the input-port being @item{@racket[input-port] --- the input-port being
processed (this is useful for matching input with multiple processed (this is useful for matching input with multiple
lexers).} lexers).}
@item{@scheme[(return-without-pos x)] is a function (continuation) that @item{@racket[(return-without-pos x)] is a function (continuation) that
immediately returns the value of @scheme[x] from the lexer. This useful immediately returns the value of @racket[x] from the lexer. This useful
in a src-pos lexer to prevent the lexer from adding source in a src-pos lexer to prevent the lexer from adding source
information. For example: information. For example:
@schemeblock[ @racketblock[
(define get-token (define get-token
(lexer-src-pos (lexer-src-pos
... ...
@ -182,43 +179,43 @@ are a few examples, using @scheme[:] prefixed SRE syntax:
would wrap the source location information for the comment around would wrap the source location information for the comment around
the value of the recursive call. Using the value of the recursive call. Using
@scheme[((comment) (return-without-pos (get-token input-port)))] @racket[((comment) (return-without-pos (get-token input-port)))]
will cause the value of the recursive call to be returned without will cause the value of the recursive call to be returned without
wrapping position around it.} wrapping position around it.}
] ]
The lexer raises an exception @scheme[(exn:read)] if none of the The lexer raises an exception @racket[(exn:read)] if none of the
regular expressions match the input. Hint: If @scheme[(any-char regular expressions match the input. Hint: If @racket[(any-char
_custom-error-behavior)] is the last rule, then there will always _custom-error-behavior)] is the last rule, then there will always
be a match, and @scheme[_custom-error-behavior] is executed to be a match, and @racket[_custom-error-behavior] is executed to
handle the error situation as desired, only consuming the first handle the error situation as desired, only consuming the first
character from the input buffer. character from the input buffer.
In addition to returning characters, input In addition to returning characters, input
ports can return @scheme[eof-object]s. Custom input ports can ports can return @racket[eof-object]s. Custom input ports can
also return a @scheme[special-comment] value to indicate a also return a @racket[special-comment] value to indicate a
non-textual comment, or return another arbitrary value (a non-textual comment, or return another arbitrary value (a
special). The non-@scheme[re] @scheme[trigger] forms handle these special). The non-@racket[re] @racket[trigger] forms handle these
cases: cases:
@itemize[ @itemize[
@item{The @scheme[(eof)] rule is matched when the input port @item{The @racket[(eof)] rule is matched when the input port
returns an @scheme[eof-object] value. If no @scheme[(eof)] returns an @racket[eof-object] value. If no @racket[(eof)]
rule is present, the lexer returns the symbol @scheme['eof] rule is present, the lexer returns the symbol @racket['eof]
when the port returns an @scheme[eof-object] value.} when the port returns an @racket[eof-object] value.}
@item{The @scheme[(special-comment)] rule is matched when the @item{The @racket[(special-comment)] rule is matched when the
input port returns a @scheme[special-comment] structure. If no input port returns a @racket[special-comment] structure. If no
@scheme[special-comment] rule is present, the lexer @racket[special-comment] rule is present, the lexer
automatically tries to return the next token from the input automatically tries to return the next token from the input
port.} port.}
@item{The @scheme[(special)] rule is matched when the input @item{The @racket[(special)] rule is matched when the input
port returns a value other than a character, port returns a value other than a character,
@scheme[eof-object], or @scheme[special-comment] structure. If @racket[eof-object], or @racket[special-comment] structure. If
no @scheme[(special)] rule is present, the lexer returns no @racket[(special)] rule is present, the lexer returns
@scheme[(void)].}] @racket[(void)].}]
End-of-files, specials, special-comments and special-errors cannot End-of-files, specials, special-comments and special-errors cannot
be parsed via a rule using an ordinary regular expression be parsed via a rule using an ordinary regular expression
@ -226,27 +223,27 @@ are a few examples, using @scheme[:] prefixed SRE syntax:
is possible in some situations). is possible in some situations).
Since the lexer gets its source information from the port, use Since the lexer gets its source information from the port, use
@scheme[port-count-lines!] to enable the tracking of line and @racket[port-count-lines!] to enable the tracking of line and
column information. Otherwise, the line and column information column information. Otherwise, the line and column information
will return @scheme[#f]. will return @racket[#f].
When peeking from the input port raises an exception (such as by When peeking from the input port raises an exception (such as by
an embedded XML editor with malformed syntax), the exception can an embedded XML editor with malformed syntax), the exception can
be raised before all tokens preceding the exception have been be raised before all tokens preceding the exception have been
returned. returned.
Each time the scheme code for a lexer is compiled (e.g. when a Each time the racket code for a lexer is compiled (e.g. when a
@filepath{.rkt} file containing a @scheme[lexer] form is loaded), @filepath{.rkt} file containing a @racket[lexer] form is loaded),
the lexer generator is run. To avoid this overhead place the the lexer generator is run. To avoid this overhead place the
lexer into a module and compile the module to a @filepath{.zo} lexer into a module and compile the module to a @filepath{.zo}
bytecode file.} bytecode file.}
@defform[(lexer-src-pos (trigger action-expr) ...)]{ @defform[(lexer-src-pos (trigger action-expr) ...)]{
Like @scheme[lexer], but for each @scheme[_action-result] produced by Like @racket[lexer], but for each @racket[_action-result] produced by
an @scheme[action-expr], returns @scheme[(make-position-token an @racket[action-expr], returns @racket[(make-position-token
_action-result start-pos end-pos)] instead of simply _action-result start-pos end-pos)] instead of simply
@scheme[_action-result].} @racket[_action-result].}
@deftogether[( @deftogether[(
@defidform[start-pos] @defidform[start-pos]
@ -256,30 +253,30 @@ _action-result start-pos end-pos)] instead of simply
@defidform[return-without-pos] @defidform[return-without-pos]
)]{ )]{
Use of these names outside of a @scheme[lexer] action is a syntax Use of these names outside of a @racket[lexer] action is a syntax
error.} error.}
@defstruct[position ([offset exact-positive-integer?] @defstruct[position ([offset exact-positive-integer?]
[line exact-positive-integer?] [line exact-positive-integer?]
[col exact-nonnegative-integer?])]{ [col exact-nonnegative-integer?])]{
Instances of @scheme[position] are bound to @scheme[start-pos] and Instances of @racket[position] are bound to @racket[start-pos] and
@scheme[end-pos]. The @scheme[offset] field contains the offset of @racket[end-pos]. The @racket[offset] field contains the offset of
the character in the input. The @scheme[line] field contains the the character in the input. The @racket[line] field contains the
line number of the character. The @scheme[col] field contains the line number of the character. The @racket[col] field contains the
offset in the current line.} offset in the current line.}
@defstruct[position-token ([token any/c] @defstruct[position-token ([token any/c]
[start-pos position?] [start-pos position?]
[end-pos position?])]{ [end-pos position?])]{
Lexers created with @scheme[src-pos-lexers] return instances of @scheme[position-token].} Lexers created with @racket[src-pos-lexers] return instances of @racket[position-token].}
@defparam[file-path source any/c]{ @defparam[file-path source any/c]{
A parameter that the lexer uses as the source location if it A parameter that the lexer uses as the source location if it
raises a @scheme[exn:fail:rad] error. Setting this parameter allows raises a @racket[exn:fail:rad] error. Setting this parameter allows
DrRacket, for example, to open the file containing the error.} DrRacket, for example, to open the file containing the error.}
@ -289,7 +286,7 @@ error.}
@defform[(char-set string)]{ @defform[(char-set string)]{
A @tech{lexer macro} that matches any character in @scheme[string].} A @tech{lexer macro} that matches any character in @racket[string].}
@defidform[any-char]{A @tech{lexer abbreviation} that matches any character.} @defidform[any-char]{A @tech{lexer abbreviation} that matches any character.}
@ -310,27 +307,27 @@ A @tech{lexer macro} that matches any character in @scheme[string].}
@defidform[iso-control] @defidform[iso-control]
)]{ )]{
@tech{Lexer abbreviations} that match @scheme[char-alphabetic?] @tech{Lexer abbreviations} that match @racket[char-alphabetic?]
characters, @scheme[char-lower-case?] characters, etc.} characters, @racket[char-lower-case?] characters, etc.}
@defform[(define-lex-abbrev id re)]{ @defform[(define-lex-abbrev id re)]{
Defines a @tech{lexer abbreviation} by associating a regular Defines a @tech{lexer abbreviation} by associating a regular
expression to be used in place of the @scheme[id] in other expression to be used in place of the @racket[id] in other
regular expression. The definition of name has the same scoping regular expression. The definition of name has the same scoping
properties as a other syntactic binding (e.g., it can be exported properties as a other syntactic binding (e.g., it can be exported
from a module).} from a module).}
@defform[(define-lex-abbrevs (id re) ...)]{ @defform[(define-lex-abbrevs (id re) ...)]{
Like @scheme[define-lex-abbrev], but defines several @tech{lexer Like @racket[define-lex-abbrev], but defines several @tech{lexer
abbreviations}.} abbreviations}.}
@defform[(define-lex-trans id trans-expr)]{ @defform[(define-lex-trans id trans-expr)]{
Defines a @tech{lexer macro}, where @scheme[trans-expr] produces a Defines a @tech{lexer macro}, where @racket[trans-expr] produces a
transformer procedure that takes one argument. When @scheme[(id transformer procedure that takes one argument. When @racket[(id
_datum ...)] appears as a regular expression, it is replaced with _datum ...)] appears as a regular expression, it is replaced with
the result of applying the transformer to the expression.} the result of applying the transformer to the expression.}
@ -349,56 +346,56 @@ characters, @scheme[char-lower-case?] characters, etc.}
@defform[(* re ...)]{ @defform[(* re ...)]{
Repetition of @scheme[re] sequence 0 or more times.} Repetition of @racket[re] sequence 0 or more times.}
@defform[(+ re ...)]{ @defform[(+ re ...)]{
Repetition of @scheme[re] sequence 1 or more times.} Repetition of @racket[re] sequence 1 or more times.}
@defform[(? re ...)]{ @defform[(? re ...)]{
Zero or one occurrence of @scheme[re] sequence.} Zero or one occurrence of @racket[re] sequence.}
@defform[(= n re ...)]{ @defform[(= n re ...)]{
Exactly @scheme[n] occurrences of @scheme[re] sequence, where Exactly @racket[n] occurrences of @racket[re] sequence, where
@scheme[n] must be a literal exact, non-negative number.} @racket[n] must be a literal exact, non-negative number.}
@defform[(>= n re ...)]{ @defform[(>= n re ...)]{
At least @scheme[n] occurrences of @scheme[re] sequence, where At least @racket[n] occurrences of @racket[re] sequence, where
@scheme[n] must be a literal exact, non-negative number.} @racket[n] must be a literal exact, non-negative number.}
@defform[(** n m re ...)]{ @defform[(** n m re ...)]{
Between @scheme[n] and @scheme[m] (inclusive) occurrences of Between @racket[n] and @racket[m] (inclusive) occurrences of
@scheme[re] sequence, where @scheme[n] must be a literal exact, @racket[re] sequence, where @racket[n] must be a literal exact,
non-negative number, and @scheme[m] must be literally either non-negative number, and @racket[m] must be literally either
@scheme[#f], @scheme[+inf.0], or an exact, non-negative number; a @racket[#f], @racket[+inf.0], or an exact, non-negative number; a
@scheme[#f] value for @scheme[m] is the same as @scheme[+inf.0].} @racket[#f] value for @racket[m] is the same as @racket[+inf.0].}
@defform[(or re ...)]{ @defform[(or re ...)]{
Same as @scheme[(union re ...)].} Same as @racket[(union re ...)].}
@deftogether[( @deftogether[(
@defform[(: re ...)] @defform[(: re ...)]
@defform[(seq re ...)] @defform[(seq re ...)]
)]{ )]{
Both forms concatenate the @scheme[re]s.} Both forms concatenate the @racket[re]s.}
@defform[(& re ...)]{ @defform[(& re ...)]{
Intersects the @scheme[re]s.} Intersects the @racket[re]s.}
@defform[(- re ...)]{ @defform[(- re ...)]{
The set difference of the @scheme[re]s.} The set difference of the @racket[re]s.}
@defform[(~ re ...)]{ @defform[(~ re ...)]{
Character-set complement, which each @scheme[re] must match exactly Character-set complement, which each @racket[re] must match exactly
one character.} one character.}
@defform[(/ char-or-string ...)]{ @defform[(/ char-or-string ...)]{
@ -421,11 +418,11 @@ characters.}
(begin (begin
(require (for-label parser-tools/lex-plt-v200)) (require (for-label parser-tools/lex-plt-v200))
@t{The @schememodname[parser-tools/lex-plt-v200] module re-exports @t{The @racketmodname[parser-tools/lex-plt-v200] module re-exports
@scheme[*], @scheme[+], @scheme[?], and @scheme[&] from @racket[*], @racket[+], @racket[?], and @racket[&] from
@schememodname[parser-tools/lex-sre]. It also re-exports @racketmodname[parser-tools/lex-sre]. It also re-exports
@scheme[:or] as @scheme[:], @scheme[::] as @scheme[|@|], @scheme[:~] @racket[:or] as @racket[:], @racket[::] as @racket[|@|], @racket[:~]
as @scheme[^], and @scheme[:/] as @scheme[-].} as @racket[^], and @racket[:/] as @racket[-].}
@defform[(epsilon)]{ @defform[(epsilon)]{
@ -433,7 +430,7 @@ A @tech{lexer macro} that matches an empty sequence.}
@defform[(~ re ...)]{ @defform[(~ re ...)]{
The same as @scheme[(complement re ...)].}))) The same as @racket[(complement re ...)].})))
@(lex-v200-doc) @(lex-v200-doc)
@ -441,30 +438,30 @@ The same as @scheme[(complement re ...)].})))
@subsection{Tokens} @subsection{Tokens}
Each @scheme[_action-expr] in a @scheme[lexer] form can produce any Each @racket[_action-expr] in a @racket[lexer] form can produce any
kind of value, but for many purposes, producing a @deftech{token} kind of value, but for many purposes, producing a @deftech{token}
value is useful. Tokens are usually necessary for inter-operating with value is useful. Tokens are usually necessary for inter-operating with
a parser generated by @scheme[parser-tools/parser], but tokens not be a parser generated by @racket[parser-tools/parser], but tokens not be
the right choice when using @scheme[lexer] in other situations. the right choice when using @racket[lexer] in other situations.
@defform[(define-tokens group-id (token-id ...))]{ @defform[(define-tokens group-id (token-id ...))]{
Binds @scheme[group-id] to the group of tokens being defined. For Binds @racket[group-id] to the group of tokens being defined. For
each @scheme[token-id], a function each @racket[token-id], a function
@schemeidfont{token-}@scheme[token-id] is created that takes any @racketidfont{token-}@racket[token-id] is created that takes any
value and puts it in a token record specific to @scheme[token-id]. value and puts it in a token record specific to @racket[token-id].
The token value is inspected using @scheme[token-id] and The token value is inspected using @racket[token-id] and
@scheme[token-value]. @racket[token-value].
A token cannot be named @schemeidfont{error}, since A token cannot be named @racketidfont{error}, since
@schemeidfont{error} it has special use in the parser.} @racketidfont{error} it has special use in the parser.}
@defform[(define-empty-tokens group-id (token-id ...) )]{ @defform[(define-empty-tokens group-id (token-id ...) )]{
Like @scheme[define-tokens], except a each token constructor Like @racket[define-tokens], except a each token constructor
@schemeidfont{token-}@scheme[token-id] takes no arguments and returns @racketidfont{token-}@racket[token-id] takes no arguments and returns
@scheme[(@#,scheme[quote] token-id)].} @racket[(@#,racket[quote] token-id)].}
@defproc[(token-name [t (or/c token? symbol?)]) symbol?]{ @defproc[(token-name [t (or/c token? symbol?)]) symbol?]{
@ -476,13 +473,13 @@ the right choice when using @scheme[lexer] in other situations.
@defproc[(token-value [t (or/c token? symbol?)]) any/c]{ @defproc[(token-value [t (or/c token? symbol?)]) any/c]{
Returns the value of a token that is represented either by a symbol Returns the value of a token that is represented either by a symbol
or a token structure, returning @scheme[#f] for a symbol token.} or a token structure, returning @racket[#f] for a symbol token.}
@defproc[(token? [v any/c]) boolean?]{ @defproc[(token? [v any/c]) boolean?]{
Returns @scheme[#t] if @scheme[val] is a Returns @racket[#t] if @racket[val] is a
token structure, @scheme[#f] otherwise.} token structure, @racket[#f] otherwise.}
@; ---------------------------------------------------------------------- @; ----------------------------------------------------------------------
@ -502,7 +499,7 @@ the right choice when using @scheme[lexer] in other situations.
(tokens group-id ...) (tokens group-id ...)
(start non-terminal-id ...) (start non-terminal-id ...)
(end token-id ...) (end token-id ...)
(@#,schemeidfont{error} expr) (@#,racketidfont{error} expr)
(precs (assoc token-id ...) ...) (precs (assoc token-id ...) ...)
(src-pos) (src-pos)
(suppress) (suppress)
@ -518,63 +515,63 @@ the right choice when using @scheme[lexer] in other situations.
@itemize[ @itemize[
@item{@schemeblock0[(grammar (non-terminal-id @item{@racketblock0[(grammar (non-terminal-id
((grammar-id ...) maybe-prec expr) ((grammar-id ...) maybe-prec expr)
...) ...)
...)] ...)]
Declares the grammar to be parsed. Each @scheme[grammar-id] can Declares the grammar to be parsed. Each @racket[grammar-id] can
be a @scheme[token-id] from a @scheme[group-id] named in a be a @racket[token-id] from a @racket[group-id] named in a
@scheme[tokens] declaration, or it can be a @racket[tokens] declaration, or it can be a
@scheme[non-terminal-id] declared in the @scheme[grammar] @racket[non-terminal-id] declared in the @racket[grammar]
declaration. The optional @scheme[prec] declaration works with declaration. The optional @racket[prec] declaration works with
the @scheme[precs] declaration. The @scheme[expr] is a the @racket[precs] declaration. The @racket[expr] is a
``semantic action,'' which is evaluated when the input is found ``semantic action,'' which is evaluated when the input is found
to match its corresponding production. to match its corresponding production.
Each action is scheme code that has the same scope as its Each action is Racket code that has the same scope as its
parser's definition, except that the variables @scheme[$1], ..., parser's definition, except that the variables @racket[$1], ...,
@schemeidfont{$}@math{i} are bound, where @math{i} is the number @racketidfont{$}@math{i} are bound, where @math{i} is the number
of @scheme[grammar-id]s in the corresponding production. Each of @racket[grammar-id]s in the corresponding production. Each
@schemeidfont{$}@math{k} is bound to the result of the action @racketidfont{$}@math{k} is bound to the result of the action
for the @math{k}@superscript{th} grammar symbol on the right of for the @math{k}@superscript{th} grammar symbol on the right of
the production, if that grammar symbol is a non-terminal, or the the production, if that grammar symbol is a non-terminal, or the
value stored in the token if the grammar symbol is a terminal. value stored in the token if the grammar symbol is a terminal.
If the @scheme[src-pos] option is present in the parser, then If the @racket[src-pos] option is present in the parser, then
variables @scheme[$1-start-pos], ..., variables @racket[$1-start-pos], ...,
@schemeidfont{$}@math{i}@schemeidfont{-start-pos} and @racketidfont{$}@math{i}@racketidfont{-start-pos} and
@scheme[$1-end-pos], ..., @racket[$1-end-pos], ...,
@schemeidfont{$}@math{i}@schemeidfont{-end-pos} and are also @racketidfont{$}@math{i}@racketidfont{-end-pos} and are also
available, and they refer to the position structures available, and they refer to the position structures
corresponding to the start and end of the corresponding corresponding to the start and end of the corresponding
@scheme[grammar-symbol]. Grammar symbols defined as empty-tokens @racket[grammar-symbol]. Grammar symbols defined as empty-tokens
have no @schemeidfont{$}@math{k} associated, but do have have no @racketidfont{$}@math{k} associated, but do have
@schemeidfont{$}@math{k}@schemeidfont{-start-pos} and @racketidfont{$}@math{k}@racketidfont{-start-pos} and
@schemeidfont{$}@math{k}@schemeidfont{-end-pos}. @racketidfont{$}@math{k}@racketidfont{-end-pos}.
Also @schemeidfont{$n-start-pos} and @schemeidfont{$n-end-pos} Also @racketidfont{$n-start-pos} and @racketidfont{$n-end-pos}
are bound to the largest start and end positions, (i.e., are bound to the largest start and end positions, (i.e.,
@schemeidfont{$}@math{i}@schemeidfont{-start-pos} and @racketidfont{$}@math{i}@racketidfont{-start-pos} and
@schemeidfont{$}@math{i}@schemeidfont{-end-pos}). @racketidfont{$}@math{i}@racketidfont{-end-pos}).
All of the productions for a given non-terminal must be grouped All of the productions for a given non-terminal must be grouped
with it. That is, no @scheme[non-terminal-id] may appear twice with it. That is, no @racket[non-terminal-id] may appear twice
on the left hand side in a parser.} on the left hand side in a parser.}
@item{@scheme[(tokens group-id ...)] @item{@racket[(tokens group-id ...)]
Declares that all of the tokens defined in each Declares that all of the tokens defined in each
@scheme[group-id]---as bound by @scheme[define-tokens] or @racket[group-id]---as bound by @racket[define-tokens] or
@scheme[define-empty-tokens]---can be used by the parser in the @racket[define-empty-tokens]---can be used by the parser in the
@scheme[grammar] declaration.} @racket[grammar] declaration.}
@item{@scheme[(start non-terminal-id ...)] @item{@racket[(start non-terminal-id ...)]
Declares a list of starting non-terminals for the grammar.} Declares a list of starting non-terminals for the grammar.}
@item{@scheme[(end token-id ...)] @item{@racket[(end token-id ...)]
Specifies a set of tokens from which some member must follow any Specifies a set of tokens from which some member must follow any
valid parse. For example, an EOF token would be specified for a valid parse. For example, an EOF token would be specified for a
@ -582,53 +579,53 @@ the right choice when using @scheme[lexer] in other situations.
that parses entire lines individually.} that parses entire lines individually.}
@item{@scheme[(@#,schemeidfont{error} expr)] @item{@racket[(@#,racketidfont{error} expr)]
The @scheme[expr] should evaluate to a function which will be The @racket[expr] should evaluate to a function which will be
executed for its side-effect whenever the parser encounters an executed for its side-effect whenever the parser encounters an
error. error.
If the @scheme[src-pos] declaration is present, the function If the @racket[src-pos] declaration is present, the function
should accept 5 arguments,: should accept 5 arguments,:
@schemeblock[(lambda (tok-ok? tok-name tok-value _start-pos _end-pos) @racketblock[(lambda (tok-ok? tok-name tok-value _start-pos _end-pos)
....)] ....)]
Otherwise it should accept 3: Otherwise it should accept 3:
@schemeblock[(lambda (tok-ok? tok-name tok-value) @racketblock[(lambda (tok-ok? tok-name tok-value)
....)] ....)]
The first argument will be @scheme[#f] if and only if the error The first argument will be @racket[#f] if and only if the error
is that an invalid token was received. The second and third is that an invalid token was received. The second and third
arguments will be the name and the value of the token at which arguments will be the name and the value of the token at which
the error was detected. The fourth and fifth arguments, if the error was detected. The fourth and fifth arguments, if
present, provide the source positions of that token.} present, provide the source positions of that token.}
@item{@scheme[(precs (assoc token-id ...) ...)] @item{@racket[(precs (assoc token-id ...) ...)]
@italic{OPTIONAL} @italic{OPTIONAL}
Precedence declarations to resolve shift/reduce and Precedence declarations to resolve shift/reduce and
reduce/reduce conflicts as in @exec{yacc}/@exec{bison}. An reduce/reduce conflicts as in @exec{yacc}/@exec{bison}. An
@scheme[assoc] must be one of @scheme[left], @scheme[right] or @racket[assoc] must be one of @racket[left], @racket[right] or
@scheme[nonassoc]. States with multiple shift/reduce or @racket[nonassoc]. States with multiple shift/reduce or
reduce/reduce conflicts (or some combination thereof) are not reduce/reduce conflicts (or some combination thereof) are not
resolved with precedence.} resolved with precedence.}
@item{@scheme[(src-pos)] @italic{OPTIONAL} @item{@racket[(src-pos)] @italic{OPTIONAL}
Causes the generated parser to expect input in the form Causes the generated parser to expect input in the form
@scheme[(make-position-token _token _start-pos _end-pos)] instead @racket[(make-position-token _token _start-pos _end-pos)] instead
of simply @scheme[_token]. Include this option when using the of simply @racket[_token]. Include this option when using the
parser with a lexer generated with @scheme[lexer-src-pos].} parser with a lexer generated with @racket[lexer-src-pos].}
@item{@scheme[(debug filename)] @italic{OPTIONAL} @item{@racket[(debug filename)] @italic{OPTIONAL}
Causes the parser generator to write the LALR table to the file Causes the parser generator to write the LALR table to the file
named @scheme[filename] (unless the file exists), where named @racket[filename] (unless the file exists), where
@scheme[filename] is a literal string. Additionally, if a debug @racket[filename] is a literal string. Additionally, if a debug
file is specified, when a running generated parser encounters a file is specified, when a running generated parser encounters a
parse error on some input file, after the user specified error parse error on some input file, after the user specified error
expression returns, the complete parse stack is printed to expression returns, the complete parse stack is printed to
@ -637,43 +634,43 @@ the right choice when using @scheme[lexer] in other situations.
the LALR table file.} the LALR table file.}
@item{@scheme[(yacc-output filename)] @italic{OPTIONAL} @item{@racket[(yacc-output filename)] @italic{OPTIONAL}
Causes the parser generator to write a grammar file in Causes the parser generator to write a grammar file in
approximately the syntax of @exec{yacc}/@exec{bison}. The file approximately the syntax of @exec{yacc}/@exec{bison}. The file
might not be a valid @exec{yacc} file, because the scheme might not be a valid @exec{yacc} file, because the Racket
grammar can use symbols that are invalid in C.} grammar can use symbols that are invalid in C.}
@item{@scheme[(suppress)] @italic{OPTIONAL} @item{@racket[(suppress)] @italic{OPTIONAL}
Causes the parser generator not to report shift/reduce or Causes the parser generator not to report shift/reduce or
reduce/reduce conflicts.} reduce/reduce conflicts.}
] ]
The result of a @scheme[parser] expression with one @scheme[start] The result of a @racket[parser] expression with one @racket[start]
non-terminal is a function, @scheme[_parse], that takes one non-terminal is a function, @racket[_parse], that takes one
argument. This argument must be a zero argument function, argument. This argument must be a zero argument function,
@scheme[_gen], that produces successive tokens of the input each @racket[_gen], that produces successive tokens of the input each
time it is called. If desired, the @scheme[_gen] may return time it is called. If desired, the @racket[_gen] may return
symbols instead of tokens, and the parser will treat symbols as symbols instead of tokens, and the parser will treat symbols as
tokens of the corresponding name (with @scheme[#f] as a value, so tokens of the corresponding name (with @racket[#f] as a value, so
it is usual to return symbols only in the case of empty tokens). it is usual to return symbols only in the case of empty tokens).
The @scheme[_parse] function returns the value associated with the The @racket[_parse] function returns the value associated with the
parse tree by the semantic actions. If the parser encounters an parse tree by the semantic actions. If the parser encounters an
error, after invoking the supplied error function, it will try to error, after invoking the supplied error function, it will try to
use error productions to continue parsing. If it cannot, it use error productions to continue parsing. If it cannot, it
raises @scheme[exn:fail:read]. raises @racket[exn:fail:read].
If multiple non-terminals are provided in @scheme[start], the If multiple non-terminals are provided in @racket[start], the
@scheme[parser] expression produces a list of parsing functions, @racket[parser] expression produces a list of parsing functions,
one for each non-terminal in the same order. Each parsing function one for each non-terminal in the same order. Each parsing function
is like the result of a parser expression with only one is like the result of a parser expression with only one
@scheme[start] non-terminal, @racket[start] non-terminal,
Each time the scheme code for a @scheme[parser] is compiled Each time the Racket code for a @racket[parser] is compiled
(e.g. when a @filepath{.rkt} file containing a @scheme[parser] form (e.g. when a @filepath{.rkt} file containing a @racket[parser] form
is loaded), the parser generator is run. To avoid this overhead is loaded), the parser generator is run. To avoid this overhead
place the parser into a module and compile the module to a place the parser into a module and compile the module to a
@filepath{.zo} bytecode file.} @filepath{.zo} bytecode file.}
@ -686,12 +683,12 @@ the right choice when using @scheme[lexer] in other situations.
@defproc[(trans [file path-string?]) any/c]{ @defproc[(trans [file path-string?]) any/c]{
Reads a C @exec{yacc}/@exec{bison} grammar from @scheme[file] and Reads a C @exec{yacc}/@exec{bison} grammar from @racket[file] and
produces an s-expression that represents a scheme parser for use with produces an s-expression that represents a Racket parser for use with
@scheme[parser]. @racket[parser].
This function is intended to assist in the manual conversion of This function is intended to assist in the manual conversion of
grammars for use with @scheme[parser], and not as a fully automatic grammars for use with @racket[parser], and not as a fully automatic
conversion tool. It is not entirely robust. For example, if the C conversion tool. It is not entirely robust. For example, if the C
actions in the original grammar have nested blocks, the tool will fail. actions in the original grammar have nested blocks, the tool will fail.

Write Preview
Loading…
Cancel
Save