You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
beautiful-racket/beautiful-racket-demo/jsonic-demo/tokenizer.rkt

15 lines
371 B
Racket

#lang br/quicklang
8 years ago
(require brag/support)
8 years ago
(define (tokenize port)
(define (next-token)
(define our-lexer
(lexer
[(eof) eof]
8 years ago
[(from/to "//" "\n") (next-token)]
[(from/to "@$" "$@")
8 years ago
(token 'SEXP-TOK (trim-ends "@$" lexeme "$@"))]
8 years ago
[any-char (token 'CHAR-TOK lexeme)]))
(our-lexer port))
next-token)
8 years ago
(provide tokenize)