Compare commits

..

2 Commits

Author SHA1 Message Date
Matthew Butterick 532d21423c thinking 5 years ago
Matthew Butterick 385954d59f Revert "Revert "subparsers""
This reverts commit c792b12d71.
5 years ago

@ -1,57 +0,0 @@
name: CI
on: [push, pull_request]
jobs:
run:
name: "Build using Racket '${{ matrix.racket-version }}' (${{ matrix.racket-variant }})"
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
racket-version: ["6.6", "6.7", "6.8", "6.9", "6.10.1", "6.11", "6.12", "7.0", "7.1", "7.2", "7.3", "7.4", "7.5", "7.6", "7.7", "7.8", "7.9", "8.0", "8.1", "8.2", "8.3", "current"]
racket-variant: ["BC", "CS"]
# CS builds are only provided for versions 7.4 and up so avoid
# running the job for prior versions.
exclude:
- {racket-version: "6.6", racket-variant: "CS"}
- {racket-version: "6.7", racket-variant: "CS"}
- {racket-version: "6.8", racket-variant: "CS"}
- {racket-version: "6.9", racket-variant: "CS"}
- {racket-version: "6.10.1", racket-variant: "CS"}
- {racket-version: "6.11", racket-variant: "CS"}
- {racket-version: "6.12", racket-variant: "CS"}
- {racket-version: "7.0", racket-variant: "CS"}
- {racket-version: "7.1", racket-variant: "CS"}
- {racket-version: "7.2", racket-variant: "CS"}
- {racket-version: "7.3", racket-variant: "CS"}
steps:
- name: Checkout
uses: actions/checkout@master
- uses: Bogdanp/setup-racket@v0.11
with:
distribution: 'full'
version: ${{ matrix.racket-version }}
variant: ${{ matrix.racket-variant }}
- name: Install BR parser tools
run: raco pkg install --deps search-auto https://github.com/mbutterick/br-parser-tools.git?path=br-parser-tools-lib
- name: Run the br-parser-tools tests
run: xvfb-run raco test -p br-parser-tools-lib
- name: Install brag-lib
run: raco pkg install --deps search-auto https://github.com/mbutterick/brag.git?path=brag-lib
- name: Run the brag-lib tests
run: xvfb-run raco test -p brag-lib
- name: Install brag
run: raco pkg install --deps search-auto https://github.com/mbutterick/brag.git?path=brag
- name: Run the brag tests
run: xvfb-run raco test -p brag

@ -0,0 +1,57 @@
# adapted from
# https://github.com/greghendershott/travis-racket/blob/master/.travis.yml
# Thanks Greg!
language: c
sudo: false
env:
global:
- RACKET_DIR=~/racket
matrix:
# - RACKET_VERSION=6.0
# - RACKET_VERSION=6.1
# - RACKET_VERSION=6.2
- RACKET_VERSION=6.3
# - RACKET_VERSION=6.4
# - RACKET_VERSION=6.5
- RACKET_VERSION=6.6
# - RACKET_VERSION=6.7
# - RACKET_VERSION=6.8
- RACKET_VERSION=6.9
# - RACKET_VERSION=6.10
# - RACKET_VERSION=6.11
- RACKET_VERSION=6.12
- RACKET_VERSION=7.0
- RACKET_VERSION=7.1
- RACKET_VERSION=7.2
- RACKET_VERSION=HEAD
- RACKET_VERSION=HEADCS
# You may want to test against certain versions of Racket, without
# having them count against the overall success/failure.
matrix:
allow_failures:
#- env: RACKET_VERSION=HEAD
- env: RACKET_VERSION=HEADCS
# Fast finish: Overall build result is determined as soon as any of
# its rows have failed, or, all of its rows that aren't allowed to
# fail have succeeded.
fast_finish: true
before_install:
- "export DISPLAY=:99.0" # needed for testing with `racket/gui`
- "sh -e /etc/init.d/xvfb start" # needed for testing with `racket/gui`
- git clone https://github.com/mbutterick/travis-racket.git
- cat travis-racket/install-racket.sh | bash # pipe to bash not sh!
- export PATH="${RACKET_DIR}/bin:${PATH}" #install-racket.sh can't set for us
script:
- cd .. # Travis did a cd into the dir. Back up, for the next:
# don't rely on package server
- travis_retry raco pkg install --deps search-auto https://github.com/mbutterick/br-parser-tools.git?path=br-parser-tools-lib
- raco test -p br-parser-tools-lib
- travis_retry raco pkg install --deps search-auto https://github.com/mbutterick/brag.git?path=brag-lib
- raco test -p brag-lib
- travis_retry raco pkg install --deps search-auto https://github.com/mbutterick/brag.git?path=brag
- raco test -p brag

@ -0,0 +1,165 @@
GNU LESSER GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
This version of the GNU Lesser General Public License incorporates
the terms and conditions of version 3 of the GNU General Public
License, supplemented by the additional permissions listed below.
0. Additional Definitions.
As used herein, "this License" refers to version 3 of the GNU Lesser
General Public License, and the "GNU GPL" refers to version 3 of the GNU
General Public License.
"The Library" refers to a covered work governed by this License,
other than an Application or a Combined Work as defined below.
An "Application" is any work that makes use of an interface provided
by the Library, but which is not otherwise based on the Library.
Defining a subclass of a class defined by the Library is deemed a mode
of using an interface provided by the Library.
A "Combined Work" is a work produced by combining or linking an
Application with the Library. The particular version of the Library
with which the Combined Work was made is also called the "Linked
Version".
The "Minimal Corresponding Source" for a Combined Work means the
Corresponding Source for the Combined Work, excluding any source code
for portions of the Combined Work that, considered in isolation, are
based on the Application, and not on the Linked Version.
The "Corresponding Application Code" for a Combined Work means the
object code and/or source code for the Application, including any data
and utility programs needed for reproducing the Combined Work from the
Application, but excluding the System Libraries of the Combined Work.
1. Exception to Section 3 of the GNU GPL.
You may convey a covered work under sections 3 and 4 of this License
without being bound by section 3 of the GNU GPL.
2. Conveying Modified Versions.
If you modify a copy of the Library, and, in your modifications, a
facility refers to a function or data to be supplied by an Application
that uses the facility (other than as an argument passed when the
facility is invoked), then you may convey a copy of the modified
version:
a) under this License, provided that you make a good faith effort to
ensure that, in the event an Application does not supply the
function or data, the facility still operates, and performs
whatever part of its purpose remains meaningful, or
b) under the GNU GPL, with none of the additional permissions of
this License applicable to that copy.
3. Object Code Incorporating Material from Library Header Files.
The object code form of an Application may incorporate material from
a header file that is part of the Library. You may convey such object
code under terms of your choice, provided that, if the incorporated
material is not limited to numerical parameters, data structure
layouts and accessors, or small macros, inline functions and templates
(ten or fewer lines in length), you do both of the following:
a) Give prominent notice with each copy of the object code that the
Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the object code with a copy of the GNU GPL and this license
document.
4. Combined Works.
You may convey a Combined Work under terms of your choice that,
taken together, effectively do not restrict modification of the
portions of the Library contained in the Combined Work and reverse
engineering for debugging such modifications, if you also do each of
the following:
a) Give prominent notice with each copy of the Combined Work that
the Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the Combined Work with a copy of the GNU GPL and this license
document.
c) For a Combined Work that displays copyright notices during
execution, include the copyright notice for the Library among
these notices, as well as a reference directing the user to the
copies of the GNU GPL and this license document.
d) Do one of the following:
0) Convey the Minimal Corresponding Source under the terms of this
License, and the Corresponding Application Code in a form
suitable for, and under terms that permit, the user to
recombine or relink the Application with a modified version of
the Linked Version to produce a modified Combined Work, in the
manner specified by section 6 of the GNU GPL for conveying
Corresponding Source.
1) Use a suitable shared library mechanism for linking with the
Library. A suitable mechanism is one that (a) uses at run time
a copy of the Library already present on the user's computer
system, and (b) will operate properly with a modified version
of the Library that is interface-compatible with the Linked
Version.
e) Provide Installation Information, but only if you would otherwise
be required to provide such information under section 6 of the
GNU GPL, and only to the extent that such information is
necessary to install and execute a modified version of the
Combined Work produced by recombining or relinking the
Application with a modified version of the Linked Version. (If
you use option 4d0, the Installation Information must accompany
the Minimal Corresponding Source and Corresponding Application
Code. If you use option 4d1, you must provide the Installation
Information in the manner specified by section 6 of the GNU GPL
for conveying Corresponding Source.)
5. Combined Libraries.
You may place library facilities that are a work based on the
Library side by side in a single library together with other library
facilities that are not Applications and are not covered by this
License, and convey such a combined library under terms of your
choice, if you do both of the following:
a) Accompany the combined library with a copy of the same work based
on the Library, uncombined with any other library facilities,
conveyed under the terms of this License.
b) Give prominent notice with the combined library that part of it
is a work based on the Library, and explaining where to find the
accompanying uncombined form of the same work.
6. Revised Versions of the GNU Lesser General Public License.
The Free Software Foundation may publish revised and/or new versions
of the GNU Lesser General Public License from time to time. Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the
Library as you received it specifies that a certain numbered version
of the GNU Lesser General Public License "or any later version"
applies to it, you have the option of following the terms and
conditions either of that published version or of any later version
published by the Free Software Foundation. If the Library as you
received it does not specify a version number of the GNU Lesser
General Public License, you may choose any version of the GNU Lesser
General Public License ever published by the Free Software Foundation.
If the Library as you received it specifies that a proxy can decide
whether future versions of the GNU Lesser General Public License shall
apply, that proxy's public statement of acceptance of any version is
permanent authorization for you to choose that version for the
Library.

@ -1,9 +0,0 @@
MIT License for `brag` (code only)
© 2017-2020 Matthew Butterick
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

@ -1,24 +1,19 @@
## brag ![Build Status](https://github.com/mbutterick/brag/workflows/CI/badge.svg)
brag [![Build Status](https://travis-ci.org/mbutterick/brag.svg?branch=master)](https://travis-ci.org/mbutterick/brag)
=
Racket DSL for generating parsers from BNF grammars.
## Install
Licensed under the LGPL. See `LICENSE`.
`raco pkg install brag`
## Documentation
http://docs.racket-lang.org/brag/
Install
-
## License
MIT. See `LICENSE.md`
`raco pkg install brag`
## Project status
Complete. I will maintain the code but no major updates are planned.
Documentation
-
http://docs.racket-lang.org/brag/

@ -1,12 +0,0 @@
`brag` contains substantial portions of the software [`ragg`](https://github.com/jbclements/ragg)
MIT License for `ragg`
© 2012-2013 Danny Yoo
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

@ -1,11 +1,13 @@
#lang racket/base
(require (for-syntax racket/base
racket/list
racket/syntax
"codegen.rkt"
"runtime.rkt"
"flatten.rkt")
br-parser-tools/lex
br-parser-tools/cfg-parser
racket/promise
(prefix-in bs: brag/support)
racket/set)
@ -18,7 +20,7 @@
(define-for-syntax (rules->token-types rules)
(define-values (implicit-tokens explicit-tokens) (rules-collect-token-types rules))
(remove-duplicates (append (for/list ([it (in-list implicit-tokens)])
(string->symbol (syntax-e it)))
(string->symbol (syntax-e it)))
(map syntax-e explicit-tokens)) eq?))
(define-syntax (brag-module-begin rules-stx)
@ -34,25 +36,33 @@
(check-all-rules-satisfiable! rules)
(define rule-ids (map rule-id rules))
(define rule-id-datums (map syntax-e rule-ids))
(with-syntax ([START-ID (first rule-ids)] ; The first rule, by default, is the start rule.
[((TOKEN-TYPE . TOKEN-TYPE-CONSTRUCTOR) ...)
(for/list ([tt (in-list (rules->token-types rules))])
(cons tt (string->symbol (format "token-~a" tt))))]
;; Flatten rules to use the yacc-style ruleset that br-parser-tools supports
[GENERATED-RULE-CODES (map flat-rule->yacc-rule (flatten-rules rules))]
;; main exports. Break hygiene so they're also available at top-level / repl
[(PARSE PARSE-TO-DATUM PARSE-TREE MAKE-RULE-PARSER ALL-TOKEN-TYPES)
(map (λ (sym) (datum->syntax rules-stx sym))
'(parse parse-to-datum parse-tree make-rule-parser all-token-types))]
[TOKEN (datum->syntax rules-stx 'token)] ; for repl
[RULE-IDS (map syntax-e rule-ids)]
[RULES-STX rules-stx])
(with-syntax* ([START-ID (first rule-ids)] ; The first rule, by default, is the start rule.
[((TOKEN-TYPE . TOKEN-TYPE-CONSTRUCTOR) ...)
(for/list ([tt (in-list (rules->token-types rules))])
(cons tt (string->symbol (format "token-~a" tt))))]
;; Flatten rules to use the yacc-style ruleset that br-parser-tools supports
[GENERATED-RULE-CODES (map flat-rule->yacc-rule (flatten-rules rules))]
;; main exports. Break hygiene so they're also available at top-level / repl
[(PARSE PARSE-TO-DATUM PARSE-TREE MAKE-RULE-PARSER ALL-TOKEN-TYPES)
(map (λ (sym) (datum->syntax rules-stx sym))
'(parse parse-to-datum parse-tree make-rule-parser all-token-types))]
[TOKEN (datum->syntax rules-stx 'token)] ; for repl
[(RULE-ID ...) rule-id-datums]
[(PARSE-RULE-ID ...)
(map (λ (dat) (format-id rules-stx "parse-~a" dat)) rule-id-datums)]
[(PARSE-RULE-ID-TO-DATUM ...)
(map (λ (dat) (format-id rules-stx "parse-~a-to-datum" dat)) rule-id-datums)]
[(PARSE-START-ID . _) #'(PARSE-RULE-ID ...)]
[(PARSE-START-ID-TO-DATUM . _) #'(PARSE-RULE-ID-TO-DATUM ...)]
[RULES-STX rules-stx])
;; this stx object represents the top level of a #lang brag module.
;; so any `define`s are automatically available at the repl.
;; and only identifiers explicitly `provide`d are visible on import.
#'(#%module-begin
(provide PARSE PARSE-TO-DATUM PARSE-TREE MAKE-RULE-PARSER ALL-TOKEN-TYPES)
(provide PARSE PARSE-TO-DATUM PARSE-TREE MAKE-RULE-PARSER ALL-TOKEN-TYPES
PARSE-RULE-ID ... PARSE-RULE-ID-TO-DATUM ...)
;; handle brag/support `token` with special identifier
;; so it doesn't conflict with brag's internal `token` macro
@ -70,44 +80,62 @@
;; the permissive tokenizer even nicer to work with.
(cons eof token-EOF)
(cons 'TOKEN-TYPE TOKEN-TYPE-CONSTRUCTOR) ...)))
(define-syntax (MAKE-RULE-PARSER stx)
(syntax-case stx ()
[(_ START-RULE-ID)
(and (identifier? #'START-RULE-ID) (member (syntax-e #'START-RULE-ID) 'RULE-IDS))
(module mrp racket/base
(define-syntax (MAKE-RULE-PARSER rule-id-stx)
(syntax-case rule-id-stx ()
[(_ start-rule)
(and (identifier? #'start-rule)
(member (syntax-e #'start-rule) '(RULE-ID ...)))
;; The cfg-parser depends on the start-rule provided in (start ...) to have the same
;; context as the rest of this body. Hence RECOLORED-START-RULE
(with-syntax ([RECOLORED-START-RULE (datum->syntax #'RULES-STX (syntax-e #'START-RULE-ID))])
#'(let ()
(define (rule-parser tokenizer)
(define rule-grammar (cfg-parser (tokens enumerated-tokens)
(src-pos)
(start RECOLORED-START-RULE)
(end EOF)
(error the-error-handler)
(grammar . GENERATED-RULE-CODES)))
(define next-token (make-permissive-tokenizer tokenizer all-tokens-hash/mutable))
;; here's how we support grammar "cuts" on top rule name
(define parse-tree-stx (rule-grammar next-token))
(syntax-case parse-tree-stx ()
[(TOP-RULE-NAME . _)
(if (eq? (syntax-property #'TOP-RULE-NAME 'hide-or-splice?) 'hide)
(remove-rule-name parse-tree-stx) ; use `remove-rule-name` so we get the same housekeeping
parse-tree-stx)]
[_ (error 'malformed-parse-tree)]))
(case-lambda [(tokenizer) (rule-parser tokenizer)]
[(source tokenizer)
(parameterize ([current-source source])
(rule-parser tokenizer))])))]
(with-syntax ([RECOLORED-START-RULE (datum->syntax #'RULES-STX (syntax-e #'start-rule))])
#`(let ([THE-GRAMMAR (cfg-parser (tokens enumerated-tokens)
(src-pos)
(start RECOLORED-START-RULE)
(end EOF)
(error THE-ERROR-HANDLER)
(grammar . GENERATED-RULE-CODES))])
(procedure-rename
(case-lambda [(tokenizer)
(define next-token
(make-permissive-tokenizer tokenizer all-tokens-hash/mutable))
;; little post-processor to support cuts on top rule name
(define parse-tree-stx (THE-GRAMMAR next-token))
(define top-rule-name-stx (syntax-case parse-tree-stx ()
[(TRN . REST) #'TRN]
[_ (error 'malformed-parse-tree)]))
(if (eq? (syntax-property top-rule-name-stx 'hide-or-splice?) 'hide)
;; use `remove-rule-name` so we get the same housekeeping
(remove-rule-name parse-tree-stx)
parse-tree-stx)]
[(source tokenizer)
(parameterize ([current-source source])
(PARSE tokenizer))])
(string->symbol (format "~a-rule-parser" 'start-rule)))))]
[(_ not-a-rule-id)
(raise-syntax-error #f
(format "Rule ~a is not defined in the grammar" (syntax-e #'not-a-rule-id))
stx)]))
rule-id-stx)])))
;; start-id has to be a value, not an expr, because make-rule-parser is a macro
(define PARSE (procedure-rename (MAKE-RULE-PARSER START-ID) 'PARSE))
(define (PARSE-TO-DATUM x) (syntax->datum (PARSE x)))
(define PARSE-TREE PARSE-TO-DATUM))))]))
(module RULE-ID racket/base
(require (submod ".." mrp))
(define p (MAKE-RULE-PARSER RULE-ID))
(provide p)) ...
(define PARSE-RULE-ID
(procedure-rename
(let ([func-p (delay (dynamic-require '(submod #,(syntax-source rules-stx) RULE-ID) 'p))])
(λ args (apply (force func-p) args)))
'PARSE-RULE-ID)) ...
(define (PARSE-RULE-ID-TO-DATUM x)
(syntax->datum (PARSE-RULE-ID x))) ...
;; start-id has to be a value, not an expr, because make-rule-parser is a macro
(define PARSE (procedure-rename PARSE-START-ID 'PARSE))
(define (PARSE-TO-DATUM x) (syntax->datum (PARSE x)))
(define PARSE-TREE PARSE-TO-DATUM))))]))

@ -0,0 +1,21 @@
#lang racket
(module mac racket
(provide m)
(println 'wasting-time)
(define-syntax-rule (m arg) arg))
(module mod racket
(require (submod ".." mac))
(define (modm arg) (m arg))
(provide modm))
(require rackunit)
(define f (let ([func-p (delay (dynamic-require '(submod "proof-of-concept.rkt" mod) 'modm))])
(λ (x) ((force func-p) x))))
f
(f 100)
(f 100)

@ -7,7 +7,7 @@
brag/private/internal-support)
(provide the-error-handler
(provide THE-ERROR-HANDLER
make-permissive-tokenizer
atomic-datum->syntax
positions->srcloc
@ -19,7 +19,7 @@
;; The level of indirection here is necessary since the yacc grammar wants a
;; function value for the error handler up front. We want to delay that decision
;; till parse time.
(define (the-error-handler tok-ok? tok-name tok-value start-pos end-pos)
(define (THE-ERROR-HANDLER tok-ok? tok-name tok-value start-pos end-pos)
(match (positions->srcloc start-pos end-pos)
[(list src line col offset span)
((current-parser-error-handler) tok-name

@ -1,6 +0,0 @@
#lang brag
start: A c def hello-world
A : "\"\101\\" ; A
c : '\'\U0063\\' ; c
def : "*\u64\\\"\\\x65f\"" ; de
hello-world : "\150\145\154\154\157\40\167\157\162\154\144"

@ -0,0 +1,5 @@
#lang brag
top : foo
foo : bar
bar : "x"

@ -1,4 +0,0 @@
#lang brag
start: next
next: "0"

@ -5,9 +5,7 @@
"parser.rkt"
"rule-structs.rkt"
(only-in brag/support from/to)
racket/string
syntax-color/racket-lexer
racket/match)
racket/string)
(provide lex/1 tokenize)
(module+ lex-abbrevs
@ -35,49 +33,31 @@
(define-lex-abbrev id (:& (complement (:+ digit)) (:+ id-char)))
(define-lex-abbrev id-separator (:or ":" "::="))
(define (unescape-double-quoted-lexeme lexeme start-pos end-pos)
;; use `read` so brag strings have all the notational semantics of Racket strings
(with-handlers ([exn:fail:read?
(λ (e) ((current-parser-error-handler)
#f
'error
lexeme
(position->pos start-pos)
(position->pos end-pos)))])
(list->string `(#\" ,@(string->list (read (open-input-string lexeme))) #\"))))
(define (convert-to-double-quoted lexeme)
;; brag supports single-quoted strings, for some reason
;; (Racket does not. A single quote denotes a datum)
;; let's convert a single-quoted string into standard double-quoted style
;; so we can use Racket's `read` function on it.
;; and thereby support all the standard Racket string elements:
;; https://docs.racket-lang.org/reference/reader.html#%28part._parse-string%29
(define outside-quotes-removed (string-trim lexeme "'"))
(define single-quotes-unescaped (string-replace outside-quotes-removed "\\'" "'"))
(define double-quotes-escaped (string-replace single-quotes-unescaped "\"" "\\\""))
(define double-quotes-on-ends (string-append "\"" double-quotes-escaped "\""))
double-quotes-on-ends)
(define-lex-abbrev backslash "\\")
(define-lex-abbrev single-quote "'")
(define-lex-abbrev escaped-single-quote (:: backslash single-quote))
(define-lex-abbrev double-quote "\"")
(define-lex-abbrev escaped-double-quote (:: backslash double-quote))
(define-lex-abbrev escaped-backslash (:: backslash backslash))
(define brag-lex
(define-lex-abbrev esc-chars (union "\\a" "\\b" "\\t" "\\n" "\\v" "\\f" "\\r" "\\e"))
(define (unescape-lexeme lexeme quote-char)
;; convert the literal string representation back into an escape char with lookup table
(define unescapes (hash "a" 7 "b" 8 "t" 9 "n" 10 "v" 11 "f" 12 "r" 13 "e" 27 "\"" 34 "'" 39 "\\" 92))
(define pat (regexp (format "(?<=^~a\\\\).(?=~a$)" quote-char quote-char)))
(cond
[(regexp-match pat lexeme)
=> (λ (m) (string quote-char (integer->char (hash-ref unescapes (car m))) quote-char))]
[else lexeme]))
(define lex/1
(lexer-src-pos
;; we delegate lexing of double-quoted strings to the Racket lexer (see below)
;; single-quoted string has to be handled manually (see lex/1 for details)
[(:: single-quote
(intersection
(:* (:or escaped-single-quote escaped-backslash (:~ single-quote)))
(complement (:: any-string backslash escaped-single-quote any-string)))
single-quote)
(token-LIT (unescape-double-quoted-lexeme (convert-to-double-quoted lexeme) start-pos end-pos))]
[(:or "()" "Ø" "")
(token-EMPTY lexeme)]
;; handle whitespace & escape chars within quotes as literal tokens: "\n" "\t" '\n' '\t'
;; match the escaped version, and then unescape them before they become token-LITs
[(:: "'"
(:or (:* (:or "\\'" esc-chars (:~ "'" "\\"))) "\\\\")
"'")
(token-LIT (unescape-lexeme lexeme #\'))]
[(:: "\""
(:or (:* (:or "\\\"" esc-chars (:~ "\"" "\\"))) "\\\\")
"\"")
(token-LIT (unescape-lexeme lexeme #\"))]
[(:or "()" "Ø" "") (token-EMPTY lexeme)]
["("
(token-LPAREN lexeme)]
["["
@ -129,28 +109,6 @@
(position->pos start-pos)
(position->pos end-pos-2)))]))
(define (lex/1-with-racket-lexer ip [conversion-proc values])
;; delegate lexing of strings to the default Racket lexer
(define-values (line-start col-start pos-start) (port-next-location ip))
(define str (read ip))
(define-values (line-end col-end pos-end) (port-next-location ip))
(make-position-token (token-LIT (string-append "\"" str "\""))
(make-position pos-start line-start col-start)
(make-position pos-end line-end col-end)))
(define (lex/1 ip)
(match (peek-bytes 1 0 ip)
[#"\"" (lex/1-with-racket-lexer ip)]
;; it would be nice to also handle single-quoted strings with the Racket lexer
;; but we can only change the opening delimiter with the readtable.
;; for whatever reason, the closing delimiter still has to be a double quote.
;; "mapping a character to the same action as a " means that the character starts a string, but the string is still terminated with a closing ". "
;; https://docs.racket-lang.org/reference/readtables.html#%28def._%28%28quote._~23~25kernel%29._make-readtable%29%29
#;[#"'" (parameterize ([current-readtable (make-readtable (current-readtable)
#\' #\" #f)])
'lex-single-quoted-string
(lex/1-with-racket-lexer ip convert-to-double-quoted))]
[_ (brag-lex ip)]))
;; This is the helper for the error production.
(define lex-nonwhitespace
@ -162,6 +120,8 @@
[(eof)
(values "" end-pos)]))
;; position->pos: position -> pos
;; Converts position structures from br-parser-tools/lex to our own pos structures.
(define (position->pos a-pos)

@ -19,6 +19,7 @@
"test-simple-arithmetic-grammar.rkt"
"test-simple-line-drawing.rkt"
"test-start-and-atok.rkt"
"test-subparser.rkt"
"test-top-level-cut.rkt"
"test-weird-grammar.rkt"
"test-whitespace.rkt"

@ -1,10 +0,0 @@
#lang racket/base
(require brag/examples/codepoints
rackunit)
(check-equal? (parse-to-datum '("\"A\\" "'c\\" "*d\\\"\\ef\"" "hello world"))
'(start (A "\"A\\")
(c "'c\\")
(def "*d\\\"\\ef\"")
(hello-world "hello world")))

@ -51,13 +51,11 @@
(check-equal? (l "]")
'(RBRACKET "]" 1 2))
;; 220111: lexer now converts single-quoted lexemes
;; to standard Racket-style double-quoted string literal
(check-equal? (l "'hello'")
'(LIT "\"hello\"" 1 8))
'(LIT "'hello'" 1 8))
(check-equal? (l "'he\\'llo'")
'(LIT "\"he'llo\"" 1 10))
'(LIT "'he\\'llo'" 1 10))
(check-equal? (l "/")
'(HIDE "/" 1 2))

@ -1,17 +0,0 @@
#lang racket/base
(require rackunit
brag/support
brag/examples/subrule)
(define parse-next (make-rule-parser next))
(define parse-start (make-rule-parser start))
(check-equal? (syntax->datum (parse #f "0")) '(start (next "0")))
(check-equal? (syntax->datum (parse #f "0")) (syntax->datum (parse "0")))
(check-equal? (syntax->datum (parse-start #f "0")) '(start (next "0")))
(check-equal? (syntax->datum (parse-start #f "0")) (syntax->datum (parse-start "0")))
(check-equal? (syntax->datum (parse-next #f "0")) '(next "0"))
(check-equal? (syntax->datum (parse-next #f "0")) (syntax->datum (parse-next "0")))

@ -0,0 +1,9 @@
#lang racket/base
(require brag/examples/subparser
brag/support
rackunit)
(check-equal? (parse-top-to-datum "x") (parse-to-datum "x"))
(check-equal? (parse-top-to-datum "x") '(top (foo (bar "x"))))
(check-equal? (parse-foo-to-datum "x") '(foo (bar "x")))
(check-equal? (parse-bar-to-datum "x") '(bar "x"))

@ -4,7 +4,6 @@
(define deps '(["base" #:version "6.3"]
"br-parser-tools-lib"
"rackunit-lib"
"syntax-color-lib"))
"rackunit-lib"))
(define implies '("br-parser-tools-lib"))

@ -538,24 +538,16 @@ Here's the definition for
brag/examples/simple-line-drawing/semantics
#:read my-read
#:read-syntax my-read-syntax
#:info my-get-info
#:whole-body-readers? #t
(require brag/examples/simple-line-drawing/lexer
brag/examples/simple-line-drawing/grammar)
brag/examples/simple-line-drawing/grammar)
(define (my-read in)
(syntax->datum (my-read-syntax #f in)))
(syntax->datum (my-read-syntax #f in)))
(define (my-read-syntax src ip)
(list (parse src (tokenize ip))))
(define (my-get-info key default default-filter)
(case key
[(color-lexer)
(dynamic-require 'syntax-color/default-lexer 'default-lexer)]
[else
(default-filter key default)]))
(list (parse src (tokenize ip))))
}|
}
@ -572,43 +564,43 @@ compilation:
(require (for-syntax racket/base syntax/parse))
(provide #%module-begin
;; We reuse Racket's treatment of raw datums, specifically
;; for strings and numbers:
#%datum
;; And otherwise, we provide definitions of these three forms.
;; During compiliation, Racket uses these definitions to
;; rewrite into for loops, displays, and newlines.
drawing rows chunk)
;; We reuse Racket's treatment of raw datums, specifically
;; for strings and numbers:
#%datum
;; And otherwise, we provide definitions of these three forms.
;; During compiliation, Racket uses these definitions to
;; rewrite into for loops, displays, and newlines.
drawing rows chunk)
;; Define a few compile-time functions to do the syntax rewriting:
(begin-for-syntax
(define (compile-drawing drawing-stx)
(syntax-parse drawing-stx
[({~literal drawing} row-stxs ...)
(define (compile-drawing drawing-stx)
(syntax-parse drawing-stx
[({~literal drawing} rows-stxs ...)
(syntax/loc drawing-stx
(begin row-stxs ...))]))
(syntax/loc drawing-stx
(begin rows-stxs ...))]))
(define (compile-rows row-stx)
(syntax-parse row-stx
[({~literal rows}
({~literal repeat} repeat-number)
chunks ...
";")
(define (compile-rows rows-stx)
(syntax-parse rows-stx
[({~literal rows}
({~literal repeat} repeat-number)
chunks ...
";")
(syntax/loc row-stx
(for ([i repeat-number])
chunks ...
(newline)))]))
(syntax/loc rows-stx
(for ([i repeat-number])
chunks ...
(newline)))]))
(define (compile-chunk chunk-stx)
(syntax-parse chunk-stx
[({~literal chunk} chunk-size chunk-string)
(define (compile-chunk chunk-stx)
(syntax-parse chunk-stx
[({~literal chunk} chunk-size chunk-string)
(syntax/loc chunk-stx
(for ([k chunk-size])
(display chunk-string)))])))
(syntax/loc chunk-stx
(for ([k chunk-size])
(display chunk-string)))])))
;; Wire up the use of "drawing", "rows", and "chunk" to these
@ -689,8 +681,8 @@ continues till the end of the line.
A @deftech{multiline comment} begins with @litchar{(*} and ends with @litchar{*)}.
An @deftech{identifier} is a sequence of letters, numbers, or
characters in the set @racket["-.!$%&/<=>^_~@"]. It must not contain
@litchar{*}, @litchar{+}, @litchar{?}, or @litchar|{{}| and @litchar|{}}|, as those characters are used to denote quantification.
characters in the set @racket["-.!$%&/<=>?^_~@"]. It must not contain
@litchar{*}, @litchar{+}, or @litchar|{{}| and @litchar|{}}|, as those characters are used to denote quantification.
A @deftech{pattern} is one of the following:
@itemize[
@ -931,7 +923,9 @@ bindings. The most important of these is @racket[parse]:
@racket[token-source]. The optional @racket[source-path] argument is used to enrich the
syntax-location fields.
The @deftech{token source} can either be a sequence, or a 0-arity function that
@tt{brag} also exports a function called @racket[parse-rule-id] for each @racket[_rule-id] in the grammar. So a grammar with rules called @tt{top}, @tt{foo}, and @tt{bar} would export @racket[parse-top], @racket[parse-foo], and @racket[parse-bar]. These take the same arguments as @racket[parse], but use the corresponding rule for the initial production, instead of the first rule.
The @deftech{token source} can either be a sequence, or a zero-arity function that
produces @tech{tokens}.
A @deftech{token} in @tt{brag} can be any of the following values:
@ -982,6 +976,8 @@ Thus, it's only the presence of @tech{rule identifier}s in a rule's
(-> token))])
list?]{
Same as @racket[parse], but the result is converted into a plain datum. Useful for testing or debugging a parser.
@tt{brag} also exports a function called @racket[parse-rule-id-to-datum] for each @racket[_rule-id] in the grammar. So a grammar with rules called @tt{top}, @tt{foo}, and @tt{bar} would export @racket[parse-top-to-datum], @racket[parse-foo-to-datum], and @racket[parse-bar-to-datum]. These take the same arguments as @racket[parse-to-datum], but use the corresponding rule for the initial production, instead of the first rule.
}
@ -1184,11 +1180,11 @@ In other words, this matches @emph{all} the @racket[re]s in order, whereas @rack
@defform[(from/to open close)]{
A string that is bounded by literal tokens @racket[open] and @racket[close]. Matching is non-greedy (meaning, it stops at the first occurence of @racket[close]). The resulting lexeme includes @racket[open] and @racket[close]. To remove them, see @racket[trim-ends].}
A string that is bounded by @racket[open] and @racket[close]. Matching is non-greedy (meaning, it stops at the first occurence of @racket[close]). The resulting lexeme includes @racket[open] and @racket[close]. To remove them, see @racket[trim-ends].}
@defform[(from/stop-before open close)]{
Like @racket[from/to], a string that is bounded by literal tokens @racket[open] and @racket[close], except that @racket[close] is not included in the resulting lexeme. Matching is non-greedy (meaning, it stops at the first occurence of @racket[close]).}
Like @racket[from/to], a string that is bounded by @racket[open] and @racket[close], except that @racket[close] is not included in the resulting lexeme. Matching is non-greedy (meaning, it stops at the first occurence of @racket[close]).}
@subsection{Differences with @tt{ragg}}

Loading…
Cancel
Save