@ -264,11 +264,11 @@ A @emph{token} is the smallest meaningful element of a source program. Tokens ca
If possible, we also want to attach source location information to each token. Why? Because this information will be incorporated into the syntax objects produced by @racket[parse].
If possible, we also want to attach source location information to each token. Why? Because this information will be incorporated into the syntax objects produced by @racket[parse].
A parser often works in conjunction with a helper function called a @emph{lexer} that converts the raw code of the source program into tokens. The @racketmodname[yaragg-parser-tools/lex] library can help us write a position-sensitive
A parser often works in conjunction with a helper function called a @emph{lexer} that converts the raw code of the source program into tokens. The @racketmodname[yaragg/parser-tools/lex] library can help us write a position-sensitive
tokenizer:
tokenizer:
@interaction[#:eval my-eval
@interaction[#:eval my-eval
(require yaragg-parser-tools/lex)
(require yaragg/parser-tools/lex)
(define (tokenize ip)
(define (tokenize ip)
(port-count-lines! ip)
(port-count-lines! ip)
(define my-lexer
(define my-lexer
@ -307,7 +307,7 @@ Note also from this lexer example:
@item{@racket[parse] accepts as input either a sequence of tokens, or a
@item{@racket[parse] accepts as input either a sequence of tokens, or a
function that produces tokens (which @racket[parse] will call repeatedly to get the next token).}
function that produces tokens (which @racket[parse] will call repeatedly to get the next token).}
@item{As an alternative to the basic @racket[token] structure, a token can also be an instance of the @racket[position-token] structure (also found in @racketmodname[yaragg-parser-tools/lex]). In that case, the token will try to derive its position from that of the position-token.}
@item{As an alternative to the basic @racket[token] structure, a token can also be an instance of the @racket[position-token] structure (also found in @racketmodname[yaragg/parser-tools/lex]). In that case, the token will try to derive its position from that of the position-token.}
@item{@racket[parse] will stop if it gets @racket[void] (or @racket['eof]) as a token.}
@item{@racket[parse] will stop if it gets @racket[void] (or @racket['eof]) as a token.}
@ -939,8 +939,8 @@ bindings. The most important of these is @racket[parse]:
@item{a string}
@item{a string}
@item{a symbol}
@item{a symbol}
@item{an instance produced by @racket[token]}
@item{an instance produced by @racket[token]}
@item{an instance produced by the token constructors of @racketmodname[yaragg-parser-tools/lex]}
@item{an instance produced by the token constructors of @racketmodname[yaragg/parser-tools/lex]}
@item{an instance of @racketmodname[yaragg-parser-tools/lex]'s @racket[position-token] whose
@item{an instance of @racketmodname[yaragg/parser-tools/lex]'s @racket[position-token] whose
@racket[position-token-token] is a @tech{token}.}
@racket[position-token-token] is a @tech{token}.}
]
]
@ -1045,7 +1045,7 @@ The @racketmodname[yaragg/support] module provides functions to interact with
@tt{brag} programs. The most useful is the @racket[token] function, which
@tt{brag} programs. The most useful is the @racket[token] function, which
produces tokens to be parsed.
produces tokens to be parsed.
In addition to the exports shown below, the @racketmodname[yaragg/support] module also provides everything from @racketmodname[yaragg/support], and everything from @racketmodname[yaragg-parser-tools/lex].
In addition to the exports shown below, the @racketmodname[yaragg/support] module also provides everything from @racketmodname[yaragg/support], and everything from @racketmodname[yaragg/parser-tools/lex].
Jack Firth
2 years ago