23 KiB
Defining new #lang Languages
When loading a module as a source program that starts
#lang language
the language determines the way that the rest of the module is parsed
at the reader level. The reader-level parse must produce a module form
as a syntax object. As always, the second sub-form after module
specifies the module language that controls the meaning of the module’s
body forms. Thus, a language specified after #lang controls both the
reader-level and expander-level parsing of a module.
1 Designating a `#lang` Language
2 Using `#lang reader`
3 Using `#lang s-exp syntax/module-reader`
4 Installing a Language
5 Source-Handling Configuration
6 Module-Handling Configuration
1. Designating a #lang Language
The syntax of a language intentionally overlaps with the syntax of a
module path as used in require or as a module language, so that names
like racket, racket/base, slideshow, or scribble/manual can be
used both as #lang languages and as module paths.
At the same time, the syntax of language is far more restricted than a
module path, because only a-z, A-Z, 0-9, / (not at the
start or end), _, -, and + are allowed in a language name.
These restrictions keep the syntax of #lang as simple as possible.
Keeping the syntax of #lang simple, in turn, is important because the
syntax is inherently inflexible and non-extensible; the #lang protocol
allows a language to refine and define syntax in a practically
unconstrained way, but the #lang protocol itself must remain fixed so
that various different tools can “boot” into the extended world.
Fortunately, the #lang protocol provides a natural way to refer to
languages in ways other than the rigid language syntax: by defining a
language that implements its own nested protocol. We have already seen
one example in \[missing\]: the s-exp language allows a
programmer to specify a module language using the general module path
syntax. Meanwhile, s-exp takes care of the reader-level
responsibilities of a #lang language.
Unlike racket, s-exp cannot be used as a module path with require.
Although the syntax of language for #lang overlaps with the syntax
of module paths, a language is not used directly as a module path.
Instead, a language obtains a module path by trying two locations:
first, it looks for a reader submodule of the main module for
language. If this is not a valid module path, then language is
suffixed with /lang/reader. (If neither is a valid module path, an
error is raised.) The resulting module supplies read and
read-syntax functions using a protocol that is similar to the one for
#reader.
+[missing] introduces
#reader.
A consequence of the way that a #lang language is turned into a
module path is that the language must be installed in a collection,
similar to the way that "racket" or "slideshow" are collections that
are distributed with Racket. Again, however, there’s an escape from this
restriction: the reader language lets you specify a reader-level
implementation of a language using a general module path.
2. Using #lang`` ``reader
The reader language for #lang is similar to s-exp, in that it acts
as a kind of meta-language. Whereas s-exp lets a programmer specify a
module language at the expander layer of parsing, reader lets a
programmer specify a language at the reader level.
A #lang reader must be followed by a module path, and the specified
module must provide two functions: read and read-syntax. The
protocol is the same as for a #reader implementation, but for #lang,
the read and read-syntax functions must produce a module form that
is based on the rest of the input file for the module.
The following "literal.rkt" module implements a language that treats
its entire body as literal text and exports the text as a data string:
"literal.rkt"
#lang racket
(require syntax/strip-context)
(provide (rename-out [literal-read read]
[literal-read-syntax read-syntax]))
(define (literal-read in)
(syntax->datum
(literal-read-syntax #f in)))
(define (literal-read-syntax src in)
(with-syntax ([str (port->string in)])
(strip-context
#'(module anything racket
(provide data)
(define data 'str)))))
The "literal.rkt" language uses strip-context on the generated
module expression, because a read-syntax function should return a
syntax object with no lexical context. Also, the "literal.rkt"
language creates a module named anything, which is an arbitrary
choice; the language is intended to be used in a file, and the longhand
module name is ignored when it appears in a required file.
The "literal.rkt" language can be used in a module "tuvalu.rkt":
"tuvalu.rkt"
#lang reader "literal.rkt"
Technology!
System!
Perfect!
Importing "tuvalu.rkt" binds data to a string version of the module
content:
> (require "tuvalu.rkt")
> data
"\nTechnology!\nSystem!\nPerfect!\n"
3. Using #lang`` ``s-exp`` ``syntax/module-reader
Parsing a module body is usually not as trivial as in "literal.rkt". A
more typical module parser must iterate to parse multiple forms for a
module body. A language is also more likely to extend Racket
syntax—perhaps through a readtable—instead of replacing Racket syntax
completely.
The syntax/module-reader module language abstracts over common parts
of a language implementation to simplify the creation of new languages.
In its most basic form, a language implemented with
syntax/module-reader simply specifies the module language to be used
for the language, in which case the reader layer of the language is the
same as Racket. For example, with
"raquet-mlang.rkt"
#lang racket
(provide (except-out (all-from-out racket) lambda)
(rename-out [lambda function]))
and
"raquet.rkt"
#lang s-exp syntax/module-reader
"raquet-mlang.rkt"
then
#lang reader "raquet.rkt"
(define identity (function (x) x))
(provide identity)
implements and exports the identity function, since
"raquet-mlang.rkt" exports lambda as function.
The syntax/module-reader language accepts many optional specifications
to adjust other features of the language. For example, an alternate
read and read-syntax for parsing the language can be specified with
#:read and #:read-syntax, respectively. The following
"dollar-racket.rkt" language uses "dollar.rkt" see \[missing\]
to build a language that is like racket but with a $ escape to
simple infix arithmetic:
"dollar-racket.rkt"
#lang s-exp syntax/module-reader
racket
#:read $-read
#:read-syntax $-read-syntax
(require (prefix-in $- "dollar.rkt"))
The require form appears at the end of the module, because all of the
keyword-tagged optional specifications for syntax/module-reader must
appear before any helper imports or definitions.
The following module uses "dollar-racket.rkt" to implement a cost
function using a $ escape:
"store.rkt"
#lang reader "dollar-racket.rkt"
(provide cost)
; Cost of ‘n' $1 rackets with 7% sales
; tax and shipping-and-handling fee ‘h':
(define (cost n h)
$n*107/100+h$)
4. Installing a Language
So far, we have used the reader meta-language to access languages like
"literal.rkt" and "dollar-racket.rkt". If you want to use something
like #lang literal directly, then you must move "literal.rkt" into a
Racket collection named "literal" see also \[missing\].
Specifically, move "literal.rkt" to a reader submodule of
"literal/main.rkt" for any directory name "literal", like so:
"literal/main.rkt"
#lang racket
(module reader racket
(require syntax/strip-context)
(provide (rename-out [literal-read read]
[literal-read-syntax read-syntax]))
(define (literal-read in)
(syntax->datum
(literal-read-syntax #f in)))
(define (literal-read-syntax src in)
(with-syntax ([str (port->string in)])
(strip-context
#'(module anything racket
(provide data)
(define data 'str))))))
Then, install the "literal" directory as a package:
cd /path/to/literal ; raco pkg install
After moving the file and installing the package, you can use literal
directly after #lang:
#lang literal
Technology!
System!
Perfect!
See [missing] for more information on using
raco.
You can also make your language available for others to install by using
the Racket package manager see \[missing\]. After you create a
"literal" package and register it with the Racket package catalog
see \[missing\], others can install it using raco pkg:
raco pkg install literal
Once installed, others can invoke the language the same way: by using
#lang literal at the top of a source file.
If you use a public source repository e.g., GitHub, you can link
your package to the source. As you improve the package, others can
update their version using raco pkg:
raco pkg update literal
See [missing] for more information about the Racket package manager.
5. Source-Handling Configuration
The Racket distribution includes a Scribble language for writing prose documents, where Scribble extends the normal Racket to better support text. Here is an example Scribble document:
#lang scribble/base
@(define (get-name) "Self-Describing Document")
@title[(get-name)]
The title of this document is “@(get-name).”
If you put that program in DrRacket’s definitions area and click Run,
then nothing much appears to happen. The scribble/base language just
binds and exports doc as a description of a document, similar to the
way that "literal.rkt" exports a string as data.
Simply opening a module with the language scribble/base in DrRacket,
however, causes a Scribble HTML button to appear. Furthermore, DrRacket
knows how to colorize Scribble syntax by coloring green those parts of
the document that correspond to literal text. The language name
scribble/base is not hard-wired into DrRacket. Instead, the
implementation of the scribble/base language provides button and
syntax-coloring information in response to a query from DrRacket.
If you have installed the literal language as described in Installing
a Language, then you can adjust "literal/main.rkt" so that DrRacket
treats the content of a module in the literal language as plain text
instead of erroneously as Racket syntax:
"literal/main.rkt"
#lang racket
(module reader racket
(require syntax/strip-context)
(provide (rename-out [literal-read read]
[literal-read-syntax read-syntax])
get-info)
(define (literal-read in)
(syntax->datum
(literal-read-syntax #f in)))
(define (literal-read-syntax src in)
(with-syntax ([str (port->string in)])
(strip-context
#'(module anything racket
(provide data)
(define data 'str)))))
(define (get-info in mod line col pos)
(lambda (key default)
(case key
[(color-lexer)
(dynamic-require 'syntax-color/default-lexer
'default-lexer)]
[else default]))))
This revised literal implementation provides a get-info function.
The get-info function is called by read-language (which DrRacket
calls) with the source input stream and location information, in case
query results should depend on the content of the module after the
language name which is not the case for `literal`. The result of
get-info is a function of two arguments. The first argument is always
a symbol, indicating the kind of information that a tool requests from
the language; the second argument is the default result to be returned
if the language does not recognize the query or has no information for
it.
After DrRacket obtains the result of get-info for a language, it calls
the function with a 'color-lexer query; the result should be a
function that implements syntax-coloring parsing on an input stream. For
literal, the syntax-color/default-lexer module provides a
default-lexer syntax-coloring parser that is suitable for plain text,
so literal loads and returns that parser in response to a
'color-lexer query.
The set of symbols that a programming tool uses for queries is entirely
between the tool and the languages that choose to cooperate with it. For
example, in addition to 'color-lexer, DrRacket uses a
'drracket:toolbar-buttons query to determine which buttons should be
available in the toolbar to operate on modules using the language.
The syntax/module-reader language lets you specify get-info handling
through a #:info optional specification. The protocol for an #:info
function is slightly different from the raw get-info protocol; the
revised protocol allows syntax/module-reader the possibility of
handling future language-information queries automatically.
6. Module-Handling Configuration
Suppose that the file "death-list-5.rkt" contains
"death-list-5.rkt"
#lang racket
(list "O-Ren Ishii"
"Vernita Green"
"Budd"
"Elle Driver"
"Bill")
If you require "death-list-5.rkt" directly, then it prints the list
in the usual Racket result format:
> (require "death-list-5.rkt")
'("O-Ren Ishii" "Vernita Green" "Budd" "Elle Driver" "Bill")
However, if "death-list-5.rkt" is required by a "kiddo.rkt" that is
implemented with scheme instead of racket:
"kiddo.rkt"
#lang scheme
(require "death-list-5.rkt")
then, if you run "kiddo.rkt" file in DrRacket or if you run it
directly with racket, "kiddo.rkt" causes "death-list-5.rkt" to
print its list in traditional Scheme format, without the leading quote:
("O-Ren Ishii" "Vernita Green" "Budd" "Elle Driver" "Bill")
The "kiddo.rkt" example illustrates how the format for printing a
result value can depend on the main module of a program instead of the
language that is used to implement it.
More broadly, certain features of a language are only invoked when a
module written in that language is run directly with racket (as
opposed to being imported into another module). One example is
result-printing style as shown above. Another example is REPL
behavior. These features are part of what’s called the run-time
configuration of a language.
Unlike the syntax-coloring property of a language (as described in
Source-Handling Configuration), the run-time configuration is a
property of a module per se as opposed to a property of the source
text representing the module. For that reason, the run-time
configuration for a module needs to be available even if the module is
compiled to bytecode form and the source is unavailable. Therefore,
run-time configuration cannot be handled by the get-info function
we’re exporting from the language’s parser module.
Instead, it will be handled by a new configure-runtime submodule that
we’ll add inside the parsed module form. When a module is run directly
with racket, racket looks for a configure-runtime submodule. If it
exists, racket runs it. But if the module is imported into another
module, the 'configure-runtime submodule is ignored. (And if the
configure-runtime submodule doesn’t exist, racket just evaluates the
module as usual.) That means that the configure-runtime submodule can
be used for any special setup tasks that need to happen when the module
is run directly.
Going back to the literal language (see Source-Handling
Configuration), we can adjust the language so that directly running a
literal module causes it to print out its string, while using a
literal module in a larger program simply provides data without
printing. To make this work, we will need an extra module. (For clarity
here, we will implement this module as a separate file. But it could
equally well be a submodule of an existing file.)
.... (the main installation or the user’s space)
|- "literal"
|- "main.rkt" (with reader submodule)
|- "show.rkt" (new)
-
The
"literal/show.rkt"module will provide ashowfunction to be applied to the string content of aliteralmodule, and also provide ashow-enabledparameter that controls whethershowactually prints the result. -
The new
configure-runtimesubmodule in"literal/main.rkt"will set theshow-enabledparameter to#t. The net effect is thatshowwill print the strings that it’s given, but only when a module using theliterallanguage is run directly (because only then will theconfigure-runtimesubmodule be invoked).
These changes are implemented in the following revised
"literal/main.rkt":
"literal/main.rkt"
#lang racket
(module reader racket
(require syntax/strip-context)
(provide (rename-out [literal-read read]
[literal-read-syntax read-syntax])
get-info)
(define (literal-read in)
(syntax->datum
(literal-read-syntax #f in)))
(define (literal-read-syntax src in)
(with-syntax ([str (port->string in)])
(strip-context
#'(module anything racket
(module configure-runtime racket
(require literal/show)
(show-enabled #t))
(require literal/show)
(provide data)
(define data 'str)
(show data)))))
(define (get-info in mod line col pos)
(lambda (key default)
(case key
[(color-lexer)
(dynamic-require 'syntax-color/default-lexer
'default-lexer)]
[else default]))))
Then the "literal/show.rkt" module must provide the show-enabled
parameter and show function:
"literal/show.rkt"
#lang racket
(provide show show-enabled)
(define show-enabled (make-parameter #f))
(define (show v)
(when (show-enabled)
(display v)))
With all of the pieces for literal in place, try running the following
variant of "tuvalu.rkt" directly and through a require from another
module:
"tuvalu.rkt"
#lang literal
Technology!
System!
Perfect!
When run directly, we’ll see the result printed like so, because our
configure-runtime submodule will have set the show-enabled parameter
to #t:
Technology! System! Perfect!
But when imported into another module, printing will be suppressed,
because the configure-runtime submodule will not be invoked, and
therefore the show-enabled parameter will remain at its default value
of #f.