You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
beautiful-racket/beautiful-racket-demo/basic-demo/tokenizer.rkt

17 lines
552 B
Racket

#lang br
8 years ago
(require brag/support)
8 years ago
(provide (all-defined-out))
8 years ago
(define basic-lexer
8 years ago
(lexer-srcloc
8 years ago
[(eof) eof]
[whitespace (token lexeme #:skip? #t)]
8 years ago
[(from/to "rem" "\n") (token 'REM (string-downcase lexeme))]
[(:or "print" "goto" "end") (token (string-downcase lexeme)
(string-downcase lexeme))]
[(:+ numeric) (token 'NUMBER (string->number lexeme))]
[(from/to "\"" "\"") (token 'STRING (trim-ends "\"" lexeme "\""))]))
9 years ago
8 years ago
(define (tokenize ip)
(port-count-lines! ip)
8 years ago
(λ () (basic-lexer ip)))