Compare commits

...

14 Commits

Author SHA1 Message Date
Matthew Butterick 30cbf95e6a doc fix: ? is used for quantification 1 year ago
Matthew Flatt f52c2a80c9
update pkg dependencies to list syntax-color-lib (#33) 2 years ago
Matthew Butterick 81f4fd25e7
lex double-quoted strings with Racket lexer (#32)
Follow up on an idea in #31: use the Racket lexer to lex double-quoted strings in a grammar, thereby giving them all the semantics of Racket strings, and preventing any corner cases. This solution can’t be extended to single-quoted strings, however, because even with readtable tomfoolery to recognize a single quote as an opening delimiter for a string, the string still needs to be closed with a double quote. (I don’t know why this should be so, but it is the documented behavior.) Therefore single-quoted strings still are subject to the homegrown lexing solution and the flaws therein. Still, I don’t see that single-quoted strings have ever been a documented feature of brag (or its predecessor ragg). Perhaps they could be dropped altogether. For now, this solution is satisfying, because double-quoted strings are the dominant notation, and this PR will make them as good as they can be.
2 years ago
Matthew Butterick 168c027031 enlarge test matrix with Racket 8 2 years ago
Matthew Butterick ba5c6c7ab5
support for codepoint escape sequences in strings (closes #29) (#31)
This improves the lexing of escape sequences within strings that appear in a grammar. It relies on Racket’s `read` to interpret these escape sequences rather than a hard-coded hash table. This gives strings in a grammar pretty much the same semantics as standard Racket strings, including support for octal and hex escape sequences for Unicode codepoints.

Though this passes all current tests, there are still some oddball corner cases that can be discovered by sticking together certain combinations of escape sequences (backslashes, double quotes, and codepoints). The better solution would be to peek into the input port for a double quote, and if it’s there, use the standard Racket lexer to pull out the string (this lexer already handles the weirdo cases). We can’t do this, however, because brag also supports single-quoted strings, which need to have the same semantics, and the Racket lexer won’t work with those. So I think we’re stuck with the homegrown solution (for consistency with both kinds of quotes) even at the expense of a few unresolved corner cases. Let’s leave that question for another day, as these cases haven’t surfaced in practical use thus far.
2 years ago
D. Ben Knoble 92b7dcc067
scribble: fix indentation in file examples (#27)
I copied in the actual file contents. It would perhaps be safest to do
some sort of "read the file and splice the contents in," but I'm not
sure how to do that at the moment.
3 years ago
Markus Pfeiffer 6983208426
Fix make-rule-parser (#28)
When using `make-rule-parser` for any rule that is not the start
rule, and applying the resulting parser while giving it a source-path
it would just use the entire grammar from the start rule.
3 years ago
Matthew Butterick d3405dd0ab
Update README.md 3 years ago
Matthew Butterick 243f32a7bb refine docs for `from/stop-before`
Only works with literal tokens, not arbitrary patterns, due to underlying limitations on `complement` pattern matcher
3 years ago
Matthew Butterick 6c161ae31d switch Travis for GH actions 3 years ago
Matthew Butterick 8814be702b force trusty for testing 4 years ago
Matthew Butterick 36f2aa56e5
Update SUBLICENSE.md 4 years ago
Matthew Butterick a7714affa6
Update README.md 4 years ago
Matthew Butterick 7880af4d6f switch to MIT license 4 years ago

@ -0,0 +1,57 @@
name: CI
on: [push, pull_request]
jobs:
run:
name: "Build using Racket '${{ matrix.racket-version }}' (${{ matrix.racket-variant }})"
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
racket-version: ["6.6", "6.7", "6.8", "6.9", "6.10.1", "6.11", "6.12", "7.0", "7.1", "7.2", "7.3", "7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "8.0", "8.1", "8.2", "8.3", "current"]
racket-variant: ["BC", "CS"]
# CS builds are only provided for versions 7.4 and up so avoid
# running the job for prior versions.
exclude:
- {racket-version: "6.6", racket-variant: "CS"}
- {racket-version: "6.7", racket-variant: "CS"}
- {racket-version: "6.8", racket-variant: "CS"}
- {racket-version: "6.9", racket-variant: "CS"}
- {racket-version: "6.10.1", racket-variant: "CS"}
- {racket-version: "6.11", racket-variant: "CS"}
- {racket-version: "6.12", racket-variant: "CS"}
- {racket-version: "7.0", racket-variant: "CS"}
- {racket-version: "7.1", racket-variant: "CS"}
- {racket-version: "7.2", racket-variant: "CS"}
- {racket-version: "7.3", racket-variant: "CS"}
steps:
- name: Checkout
uses: actions/checkout@master
- uses: Bogdanp/setup-racket@v0.11
with:
distribution: 'full'
version: ${{ matrix.racket-version }}
variant: ${{ matrix.racket-variant }}
- name: Install BR parser tools
run: raco pkg install --deps search-auto https://github.com/mbutterick/br-parser-tools.git?path=br-parser-tools-lib
- name: Run the br-parser-tools tests
run: xvfb-run raco test -p br-parser-tools-lib
- name: Install brag-lib
run: raco pkg install --deps search-auto https://github.com/mbutterick/brag.git?path=brag-lib
- name: Run the brag-lib tests
run: xvfb-run raco test -p brag-lib
- name: Install brag
run: raco pkg install --deps search-auto https://github.com/mbutterick/brag.git?path=brag
- name: Run the brag tests
run: xvfb-run raco test -p brag

@ -1,57 +0,0 @@
# adapted from
# https://github.com/greghendershott/travis-racket/blob/master/.travis.yml
# Thanks Greg!
language: c
sudo: false
env:
global:
- RACKET_DIR=~/racket
matrix:
# - RACKET_VERSION=6.0
# - RACKET_VERSION=6.1
# - RACKET_VERSION=6.2
- RACKET_VERSION=6.3
# - RACKET_VERSION=6.4
# - RACKET_VERSION=6.5
- RACKET_VERSION=6.6
# - RACKET_VERSION=6.7
# - RACKET_VERSION=6.8
- RACKET_VERSION=6.9
# - RACKET_VERSION=6.10
# - RACKET_VERSION=6.11
- RACKET_VERSION=6.12
- RACKET_VERSION=7.0
- RACKET_VERSION=7.1
- RACKET_VERSION=7.2
- RACKET_VERSION=HEAD
- RACKET_VERSION=HEADCS
# You may want to test against certain versions of Racket, without
# having them count against the overall success/failure.
matrix:
allow_failures:
#- env: RACKET_VERSION=HEAD
- env: RACKET_VERSION=HEADCS
# Fast finish: Overall build result is determined as soon as any of
# its rows have failed, or, all of its rows that aren't allowed to
# fail have succeeded.
fast_finish: true
before_install:
- "export DISPLAY=:99.0" # needed for testing with `racket/gui`
- "sh -e /etc/init.d/xvfb start" # needed for testing with `racket/gui`
- git clone https://github.com/mbutterick/travis-racket.git
- cat travis-racket/install-racket.sh | bash # pipe to bash not sh!
- export PATH="${RACKET_DIR}/bin:${PATH}" #install-racket.sh can't set for us
script:
- cd .. # Travis did a cd into the dir. Back up, for the next:
# don't rely on package server
- travis_retry raco pkg install --deps search-auto https://github.com/mbutterick/br-parser-tools.git?path=br-parser-tools-lib
- raco test -p br-parser-tools-lib
- travis_retry raco pkg install --deps search-auto https://github.com/mbutterick/brag.git?path=brag-lib
- raco test -p brag-lib
- travis_retry raco pkg install --deps search-auto https://github.com/mbutterick/brag.git?path=brag
- raco test -p brag

@ -1,165 +0,0 @@
GNU LESSER GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
This version of the GNU Lesser General Public License incorporates
the terms and conditions of version 3 of the GNU General Public
License, supplemented by the additional permissions listed below.
0. Additional Definitions.
As used herein, "this License" refers to version 3 of the GNU Lesser
General Public License, and the "GNU GPL" refers to version 3 of the GNU
General Public License.
"The Library" refers to a covered work governed by this License,
other than an Application or a Combined Work as defined below.
An "Application" is any work that makes use of an interface provided
by the Library, but which is not otherwise based on the Library.
Defining a subclass of a class defined by the Library is deemed a mode
of using an interface provided by the Library.
A "Combined Work" is a work produced by combining or linking an
Application with the Library. The particular version of the Library
with which the Combined Work was made is also called the "Linked
Version".
The "Minimal Corresponding Source" for a Combined Work means the
Corresponding Source for the Combined Work, excluding any source code
for portions of the Combined Work that, considered in isolation, are
based on the Application, and not on the Linked Version.
The "Corresponding Application Code" for a Combined Work means the
object code and/or source code for the Application, including any data
and utility programs needed for reproducing the Combined Work from the
Application, but excluding the System Libraries of the Combined Work.
1. Exception to Section 3 of the GNU GPL.
You may convey a covered work under sections 3 and 4 of this License
without being bound by section 3 of the GNU GPL.
2. Conveying Modified Versions.
If you modify a copy of the Library, and, in your modifications, a
facility refers to a function or data to be supplied by an Application
that uses the facility (other than as an argument passed when the
facility is invoked), then you may convey a copy of the modified
version:
a) under this License, provided that you make a good faith effort to
ensure that, in the event an Application does not supply the
function or data, the facility still operates, and performs
whatever part of its purpose remains meaningful, or
b) under the GNU GPL, with none of the additional permissions of
this License applicable to that copy.
3. Object Code Incorporating Material from Library Header Files.
The object code form of an Application may incorporate material from
a header file that is part of the Library. You may convey such object
code under terms of your choice, provided that, if the incorporated
material is not limited to numerical parameters, data structure
layouts and accessors, or small macros, inline functions and templates
(ten or fewer lines in length), you do both of the following:
a) Give prominent notice with each copy of the object code that the
Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the object code with a copy of the GNU GPL and this license
document.
4. Combined Works.
You may convey a Combined Work under terms of your choice that,
taken together, effectively do not restrict modification of the
portions of the Library contained in the Combined Work and reverse
engineering for debugging such modifications, if you also do each of
the following:
a) Give prominent notice with each copy of the Combined Work that
the Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the Combined Work with a copy of the GNU GPL and this license
document.
c) For a Combined Work that displays copyright notices during
execution, include the copyright notice for the Library among
these notices, as well as a reference directing the user to the
copies of the GNU GPL and this license document.
d) Do one of the following:
0) Convey the Minimal Corresponding Source under the terms of this
License, and the Corresponding Application Code in a form
suitable for, and under terms that permit, the user to
recombine or relink the Application with a modified version of
the Linked Version to produce a modified Combined Work, in the
manner specified by section 6 of the GNU GPL for conveying
Corresponding Source.
1) Use a suitable shared library mechanism for linking with the
Library. A suitable mechanism is one that (a) uses at run time
a copy of the Library already present on the user's computer
system, and (b) will operate properly with a modified version
of the Library that is interface-compatible with the Linked
Version.
e) Provide Installation Information, but only if you would otherwise
be required to provide such information under section 6 of the
GNU GPL, and only to the extent that such information is
necessary to install and execute a modified version of the
Combined Work produced by recombining or relinking the
Application with a modified version of the Linked Version. (If
you use option 4d0, the Installation Information must accompany
the Minimal Corresponding Source and Corresponding Application
Code. If you use option 4d1, you must provide the Installation
Information in the manner specified by section 6 of the GNU GPL
for conveying Corresponding Source.)
5. Combined Libraries.
You may place library facilities that are a work based on the
Library side by side in a single library together with other library
facilities that are not Applications and are not covered by this
License, and convey such a combined library under terms of your
choice, if you do both of the following:
a) Accompany the combined library with a copy of the same work based
on the Library, uncombined with any other library facilities,
conveyed under the terms of this License.
b) Give prominent notice with the combined library that part of it
is a work based on the Library, and explaining where to find the
accompanying uncombined form of the same work.
6. Revised Versions of the GNU Lesser General Public License.
The Free Software Foundation may publish revised and/or new versions
of the GNU Lesser General Public License from time to time. Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the
Library as you received it specifies that a certain numbered version
of the GNU Lesser General Public License "or any later version"
applies to it, you have the option of following the terms and
conditions either of that published version or of any later version
published by the Free Software Foundation. If the Library as you
received it does not specify a version number of the GNU Lesser
General Public License, you may choose any version of the GNU Lesser
General Public License ever published by the Free Software Foundation.
If the Library as you received it specifies that a proxy can decide
whether future versions of the GNU Lesser General Public License shall
apply, that proxy's public statement of acceptance of any version is
permanent authorization for you to choose that version for the
Library.

@ -0,0 +1,9 @@
MIT License for `brag` (code only)
© 2017-2020 Matthew Butterick
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

@ -1,19 +1,24 @@
brag [![Build Status](https://travis-ci.org/mbutterick/brag.svg?branch=master)](https://travis-ci.org/mbutterick/brag)
=
## brag ![Build Status](https://github.com/mbutterick/brag/workflows/CI/badge.svg)
Racket DSL for generating parsers from BNF grammars.
Licensed under the LGPL. See `LICENSE`.
Install
-
## Install
`raco pkg install brag`
Documentation
-
## Documentation
http://docs.racket-lang.org/brag/
## License
MIT. See `LICENSE.md`
## Project status
Complete. I will maintain the code but no major updates are planned.

@ -0,0 +1,12 @@
`brag` contains substantial portions of the software [`ragg`](https://github.com/jbclements/ragg)
MIT License for `ragg`
© 2012-2013 Danny Yoo
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

@ -18,7 +18,7 @@
(define-for-syntax (rules->token-types rules)
(define-values (implicit-tokens explicit-tokens) (rules-collect-token-types rules))
(remove-duplicates (append (for/list ([it (in-list implicit-tokens)])
(string->symbol (syntax-e it)))
(string->symbol (syntax-e it)))
(map syntax-e explicit-tokens)) eq?))
(define-syntax (brag-module-begin rules-stx)
@ -38,7 +38,7 @@
(with-syntax ([START-ID (first rule-ids)] ; The first rule, by default, is the start rule.
[((TOKEN-TYPE . TOKEN-TYPE-CONSTRUCTOR) ...)
(for/list ([tt (in-list (rules->token-types rules))])
(cons tt (string->symbol (format "token-~a" tt))))]
(cons tt (string->symbol (format "token-~a" tt))))]
;; Flatten rules to use the yacc-style ruleset that br-parser-tools supports
[GENERATED-RULE-CODES (map flat-rule->yacc-rule (flatten-rules rules))]
;; main exports. Break hygiene so they're also available at top-level / repl
@ -71,41 +71,38 @@
(cons eof token-EOF)
(cons 'TOKEN-TYPE TOKEN-TYPE-CONSTRUCTOR) ...)))
(define-syntax (MAKE-RULE-PARSER rule-id-stx)
(syntax-case rule-id-stx ()
[(_ start-rule)
(and (identifier? #'start-rule)
(member (syntax-e #'start-rule) 'RULE-IDS))
(define-syntax (MAKE-RULE-PARSER stx)
(syntax-case stx ()
[(_ START-RULE-ID)
(and (identifier? #'START-RULE-ID) (member (syntax-e #'START-RULE-ID) 'RULE-IDS))
;; The cfg-parser depends on the start-rule provided in (start ...) to have the same
;; context as the rest of this body. Hence RECOLORED-START-RULE
(with-syntax ([RECOLORED-START-RULE (datum->syntax #'RULES-STX (syntax-e #'start-rule))])
#'(let ([THE-GRAMMAR (cfg-parser (tokens enumerated-tokens)
(src-pos)
(start RECOLORED-START-RULE)
(end EOF)
(error THE-ERROR-HANDLER)
(grammar . GENERATED-RULE-CODES))])
(procedure-rename
(case-lambda [(tokenizer)
(define next-token
(make-permissive-tokenizer tokenizer all-tokens-hash/mutable))
;; little post-processor to support cuts on top rule name
(define parse-tree-stx (THE-GRAMMAR next-token))
(define top-rule-name-stx (syntax-case parse-tree-stx ()
[(TRN . REST) #'TRN]
[_ (error 'malformed-parse-tree)]))
(if (eq? (syntax-property top-rule-name-stx 'hide-or-splice?) 'hide)
;; use `remove-rule-name` so we get the same housekeeping
(remove-rule-name parse-tree-stx)
parse-tree-stx)]
[(source tokenizer)
(parameterize ([current-source source])
(PARSE tokenizer))])
(string->symbol (format "~a-rule-parser" 'start-rule)))))]
(with-syntax ([RECOLORED-START-RULE (datum->syntax #'RULES-STX (syntax-e #'START-RULE-ID))])
#'(let ()
(define (rule-parser tokenizer)
(define rule-grammar (cfg-parser (tokens enumerated-tokens)
(src-pos)
(start RECOLORED-START-RULE)
(end EOF)
(error the-error-handler)
(grammar . GENERATED-RULE-CODES)))
(define next-token (make-permissive-tokenizer tokenizer all-tokens-hash/mutable))
;; here's how we support grammar "cuts" on top rule name
(define parse-tree-stx (rule-grammar next-token))
(syntax-case parse-tree-stx ()
[(TOP-RULE-NAME . _)
(if (eq? (syntax-property #'TOP-RULE-NAME 'hide-or-splice?) 'hide)
(remove-rule-name parse-tree-stx) ; use `remove-rule-name` so we get the same housekeeping
parse-tree-stx)]
[_ (error 'malformed-parse-tree)]))
(case-lambda [(tokenizer) (rule-parser tokenizer)]
[(source tokenizer)
(parameterize ([current-source source])
(rule-parser tokenizer))])))]
[(_ not-a-rule-id)
(raise-syntax-error #f
(format "Rule ~a is not defined in the grammar" (syntax-e #'not-a-rule-id))
rule-id-stx)]))
stx)]))
;; start-id has to be a value, not an expr, because make-rule-parser is a macro
(define PARSE (procedure-rename (MAKE-RULE-PARSER START-ID) 'PARSE))

@ -7,7 +7,7 @@
brag/private/internal-support)
(provide THE-ERROR-HANDLER
(provide the-error-handler
make-permissive-tokenizer
atomic-datum->syntax
positions->srcloc
@ -19,7 +19,7 @@
;; The level of indirection here is necessary since the yacc grammar wants a
;; function value for the error handler up front. We want to delay that decision
;; till parse time.
(define (THE-ERROR-HANDLER tok-ok? tok-name tok-value start-pos end-pos)
(define (the-error-handler tok-ok? tok-name tok-value start-pos end-pos)
(match (positions->srcloc start-pos end-pos)
[(list src line col offset span)
((current-parser-error-handler) tok-name

@ -0,0 +1,6 @@
#lang brag
start: A c def hello-world
A : "\"\101\\" ; A
c : '\'\U0063\\' ; c
def : "*\u64\\\"\\\x65f\"" ; de
hello-world : "\150\145\154\154\157\40\167\157\162\154\144"

@ -0,0 +1,4 @@
#lang brag
start: next
next: "0"

@ -5,7 +5,9 @@
"parser.rkt"
"rule-structs.rkt"
(only-in brag/support from/to)
racket/string)
racket/string
syntax-color/racket-lexer
racket/match)
(provide lex/1 tokenize)
(module+ lex-abbrevs
@ -33,31 +35,49 @@
(define-lex-abbrev id (:& (complement (:+ digit)) (:+ id-char)))
(define-lex-abbrev id-separator (:or ":" "::="))
(define-lex-abbrev esc-chars (union "\\a" "\\b" "\\t" "\\n" "\\v" "\\f" "\\r" "\\e"))
(define (unescape-lexeme lexeme quote-char)
;; convert the literal string representation back into an escape char with lookup table
(define unescapes (hash "a" 7 "b" 8 "t" 9 "n" 10 "v" 11 "f" 12 "r" 13 "e" 27 "\"" 34 "'" 39 "\\" 92))
(define pat (regexp (format "(?<=^~a\\\\).(?=~a$)" quote-char quote-char)))
(cond
[(regexp-match pat lexeme)
=> (λ (m) (string quote-char (integer->char (hash-ref unescapes (car m))) quote-char))]
[else lexeme]))
(define lex/1
(define (unescape-double-quoted-lexeme lexeme start-pos end-pos)
;; use `read` so brag strings have all the notational semantics of Racket strings
(with-handlers ([exn:fail:read?
(λ (e) ((current-parser-error-handler)
#f
'error
lexeme
(position->pos start-pos)
(position->pos end-pos)))])
(list->string `(#\" ,@(string->list (read (open-input-string lexeme))) #\"))))
(define (convert-to-double-quoted lexeme)
;; brag supports single-quoted strings, for some reason
;; (Racket does not. A single quote denotes a datum)
;; let's convert a single-quoted string into standard double-quoted style
;; so we can use Racket's `read` function on it.
;; and thereby support all the standard Racket string elements:
;; https://docs.racket-lang.org/reference/reader.html#%28part._parse-string%29
(define outside-quotes-removed (string-trim lexeme "'"))
(define single-quotes-unescaped (string-replace outside-quotes-removed "\\'" "'"))
(define double-quotes-escaped (string-replace single-quotes-unescaped "\"" "\\\""))
(define double-quotes-on-ends (string-append "\"" double-quotes-escaped "\""))
double-quotes-on-ends)
(define-lex-abbrev backslash "\\")
(define-lex-abbrev single-quote "'")
(define-lex-abbrev escaped-single-quote (:: backslash single-quote))
(define-lex-abbrev double-quote "\"")
(define-lex-abbrev escaped-double-quote (:: backslash double-quote))
(define-lex-abbrev escaped-backslash (:: backslash backslash))
(define brag-lex
(lexer-src-pos
;; handle whitespace & escape chars within quotes as literal tokens: "\n" "\t" '\n' '\t'
;; match the escaped version, and then unescape them before they become token-LITs
[(:: "'"
(:or (:* (:or "\\'" esc-chars (:~ "'" "\\"))) "\\\\")
"'")
(token-LIT (unescape-lexeme lexeme #\'))]
[(:: "\""
(:or (:* (:or "\\\"" esc-chars (:~ "\"" "\\"))) "\\\\")
"\"")
(token-LIT (unescape-lexeme lexeme #\"))]
[(:or "()" "Ø" "") (token-EMPTY lexeme)]
;; we delegate lexing of double-quoted strings to the Racket lexer (see below)
;; single-quoted string has to be handled manually (see lex/1 for details)
[(:: single-quote
(intersection
(:* (:or escaped-single-quote escaped-backslash (:~ single-quote)))
(complement (:: any-string backslash escaped-single-quote any-string)))
single-quote)
(token-LIT (unescape-double-quoted-lexeme (convert-to-double-quoted lexeme) start-pos end-pos))]
[(:or "()" "Ø" "")
(token-EMPTY lexeme)]
["("
(token-LPAREN lexeme)]
["["
@ -109,6 +129,28 @@
(position->pos start-pos)
(position->pos end-pos-2)))]))
(define (lex/1-with-racket-lexer ip [conversion-proc values])
;; delegate lexing of strings to the default Racket lexer
(define-values (line-start col-start pos-start) (port-next-location ip))
(define str (read ip))
(define-values (line-end col-end pos-end) (port-next-location ip))
(make-position-token (token-LIT (string-append "\"" str "\""))
(make-position pos-start line-start col-start)
(make-position pos-end line-end col-end)))
(define (lex/1 ip)
(match (peek-bytes 1 0 ip)
[#"\"" (lex/1-with-racket-lexer ip)]
;; it would be nice to also handle single-quoted strings with the Racket lexer
;; but we can only change the opening delimiter with the readtable.
;; for whatever reason, the closing delimiter still has to be a double quote.
;; "mapping a character to the same action as a " means that the character starts a string, but the string is still terminated with a closing ". "
;; https://docs.racket-lang.org/reference/readtables.html#%28def._%28%28quote._~23~25kernel%29._make-readtable%29%29
#;[#"'" (parameterize ([current-readtable (make-readtable (current-readtable)
#\' #\" #f)])
'lex-single-quoted-string
(lex/1-with-racket-lexer ip convert-to-double-quoted))]
[_ (brag-lex ip)]))
;; This is the helper for the error production.
(define lex-nonwhitespace
@ -120,8 +162,6 @@
[(eof)
(values "" end-pos)]))
;; position->pos: position -> pos
;; Converts position structures from br-parser-tools/lex to our own pos structures.
(define (position->pos a-pos)

@ -0,0 +1,10 @@
#lang racket/base
(require brag/examples/codepoints
rackunit)
(check-equal? (parse-to-datum '("\"A\\" "'c\\" "*d\\\"\\ef\"" "hello world"))
'(start (A "\"A\\")
(c "'c\\")
(def "*d\\\"\\ef\"")
(hello-world "hello world")))

@ -51,11 +51,13 @@
(check-equal? (l "]")
'(RBRACKET "]" 1 2))
;; 220111: lexer now converts single-quoted lexemes
;; to standard Racket-style double-quoted string literal
(check-equal? (l "'hello'")
'(LIT "'hello'" 1 8))
'(LIT "\"hello\"" 1 8))
(check-equal? (l "'he\\'llo'")
'(LIT "'he\\'llo'" 1 10))
'(LIT "\"he'llo\"" 1 10))
(check-equal? (l "/")
'(HIDE "/" 1 2))

@ -0,0 +1,17 @@
#lang racket/base
(require rackunit
brag/support
brag/examples/subrule)
(define parse-next (make-rule-parser next))
(define parse-start (make-rule-parser start))
(check-equal? (syntax->datum (parse #f "0")) '(start (next "0")))
(check-equal? (syntax->datum (parse #f "0")) (syntax->datum (parse "0")))
(check-equal? (syntax->datum (parse-start #f "0")) '(start (next "0")))
(check-equal? (syntax->datum (parse-start #f "0")) (syntax->datum (parse-start "0")))
(check-equal? (syntax->datum (parse-next #f "0")) '(next "0"))
(check-equal? (syntax->datum (parse-next #f "0")) (syntax->datum (parse-next "0")))

@ -4,6 +4,7 @@
(define deps '(["base" #:version "6.3"]
"br-parser-tools-lib"
"rackunit-lib"))
"rackunit-lib"
"syntax-color-lib"))
(define implies '("br-parser-tools-lib"))

@ -538,16 +538,24 @@ Here's the definition for
brag/examples/simple-line-drawing/semantics
#:read my-read
#:read-syntax my-read-syntax
#:info my-get-info
#:whole-body-readers? #t
(require brag/examples/simple-line-drawing/lexer
brag/examples/simple-line-drawing/grammar)
brag/examples/simple-line-drawing/grammar)
(define (my-read in)
(syntax->datum (my-read-syntax #f in)))
(syntax->datum (my-read-syntax #f in)))
(define (my-read-syntax src ip)
(list (parse src (tokenize ip))))
(list (parse src (tokenize ip))))
(define (my-get-info key default default-filter)
(case key
[(color-lexer)
(dynamic-require 'syntax-color/default-lexer 'default-lexer)]
[else
(default-filter key default)]))
}|
}
@ -564,43 +572,43 @@ compilation:
(require (for-syntax racket/base syntax/parse))
(provide #%module-begin
;; We reuse Racket's treatment of raw datums, specifically
;; for strings and numbers:
#%datum
;; And otherwise, we provide definitions of these three forms.
;; During compiliation, Racket uses these definitions to
;; rewrite into for loops, displays, and newlines.
drawing rows chunk)
;; We reuse Racket's treatment of raw datums, specifically
;; for strings and numbers:
#%datum
;; And otherwise, we provide definitions of these three forms.
;; During compiliation, Racket uses these definitions to
;; rewrite into for loops, displays, and newlines.
drawing rows chunk)
;; Define a few compile-time functions to do the syntax rewriting:
(begin-for-syntax
(define (compile-drawing drawing-stx)
(syntax-parse drawing-stx
[({~literal drawing} rows-stxs ...)
(define (compile-drawing drawing-stx)
(syntax-parse drawing-stx
[({~literal drawing} row-stxs ...)
(syntax/loc drawing-stx
(begin rows-stxs ...))]))
(syntax/loc drawing-stx
(begin row-stxs ...))]))
(define (compile-rows rows-stx)
(syntax-parse rows-stx
[({~literal rows}
({~literal repeat} repeat-number)
chunks ...
";")
(define (compile-rows row-stx)
(syntax-parse row-stx
[({~literal rows}
({~literal repeat} repeat-number)
chunks ...
";")
(syntax/loc rows-stx
(for ([i repeat-number])
chunks ...
(newline)))]))
(syntax/loc row-stx
(for ([i repeat-number])
chunks ...
(newline)))]))
(define (compile-chunk chunk-stx)
(syntax-parse chunk-stx
[({~literal chunk} chunk-size chunk-string)
(define (compile-chunk chunk-stx)
(syntax-parse chunk-stx
[({~literal chunk} chunk-size chunk-string)
(syntax/loc chunk-stx
(for ([k chunk-size])
(display chunk-string)))])))
(syntax/loc chunk-stx
(for ([k chunk-size])
(display chunk-string)))])))
;; Wire up the use of "drawing", "rows", and "chunk" to these
@ -681,8 +689,8 @@ continues till the end of the line.
A @deftech{multiline comment} begins with @litchar{(*} and ends with @litchar{*)}.
An @deftech{identifier} is a sequence of letters, numbers, or
characters in the set @racket["-.!$%&/<=>?^_~@"]. It must not contain
@litchar{*}, @litchar{+}, or @litchar|{{}| and @litchar|{}}|, as those characters are used to denote quantification.
characters in the set @racket["-.!$%&/<=>^_~@"]. It must not contain
@litchar{*}, @litchar{+}, @litchar{?}, or @litchar|{{}| and @litchar|{}}|, as those characters are used to denote quantification.
A @deftech{pattern} is one of the following:
@itemize[
@ -1176,11 +1184,11 @@ In other words, this matches @emph{all} the @racket[re]s in order, whereas @rack
@defform[(from/to open close)]{
A string that is bounded by @racket[open] and @racket[close]. Matching is non-greedy (meaning, it stops at the first occurence of @racket[close]). The resulting lexeme includes @racket[open] and @racket[close]. To remove them, see @racket[trim-ends].}
A string that is bounded by literal tokens @racket[open] and @racket[close]. Matching is non-greedy (meaning, it stops at the first occurence of @racket[close]). The resulting lexeme includes @racket[open] and @racket[close]. To remove them, see @racket[trim-ends].}
@defform[(from/stop-before open close)]{
Like @racket[from/to], a string that is bounded by @racket[open] and @racket[close], except that @racket[close] is not included in the resulting lexeme. Matching is non-greedy (meaning, it stops at the first occurence of @racket[close]).}
Like @racket[from/to], a string that is bounded by literal tokens @racket[open] and @racket[close], except that @racket[close] is not included in the resulting lexeme. Matching is non-greedy (meaning, it stops at the first occurence of @racket[close]).}
@subsection{Differences with @tt{ragg}}

Loading…
Cancel
Save