# Defining new `#lang` Languages When loading a module as a source program that starts `#lang` `language` the `language` determines the way that the rest of the module is parsed at the reader level. The reader-level parse must produce a `module` form as a syntax object. As always, the second sub-form after `module` specifies the module language that controls the meaning of the module’s body forms. Thus, a `language` specified after `#lang` controls both the reader-level and expander-level parsing of a module. 1 Designating a `#lang` Language 2 Using `#lang reader` 3 Using `#lang s-exp syntax/module-reader` 4 Installing a Language 5 Source-Handling Configuration 6 Module-Handling Configuration ## 1. Designating a `#lang` Language The syntax of a `language` intentionally overlaps with the syntax of a module path as used in `require` or as a module language, so that names like `racket`, `racket/base`, `slideshow`, or `scribble/manual` can be used both as `#lang` languages and as module paths. At the same time, the syntax of `language` is far more restricted than a module path, because only `a`-`z`, `A`-`Z`, `0`-`9`, `/` \(not at the start or end\), `_`, `-`, and `+` are allowed in a `language` name. These restrictions keep the syntax of `#lang` as simple as possible. Keeping the syntax of `#lang` simple, in turn, is important because the syntax is inherently inflexible and non-extensible; the `#lang` protocol allows a `language` to refine and define syntax in a practically unconstrained way, but the `#lang` protocol itself must remain fixed so that various different tools can “boot” into the extended world. Fortunately, the `#lang` protocol provides a natural way to refer to languages in ways other than the rigid `language` syntax: by defining a `language` that implements its own nested protocol. We have already seen one example \(in \[missing\]\): the `s-exp` `language` allows a programmer to specify a module language using the general module path syntax. Meanwhile, `s-exp` takes care of the reader-level responsibilities of a `#lang` language. Unlike `racket`, `s-exp` cannot be used as a module path with `require`. Although the syntax of `language` for `#lang` overlaps with the syntax of module paths, a `language` is not used directly as a module path. Instead, a `language` obtains a module path by trying two locations: first, it looks for a `reader` submodule of the main module for `language`. If this is not a valid module path, then `language` is suffixed with `/lang/reader`. \(If neither is a valid module path, an error is raised.\) The resulting module supplies `read` and `read-syntax` functions using a protocol that is similar to the one for `#reader`. > +\[missing\] introduces `#reader`. A consequence of the way that a `#lang` `language` is turned into a module path is that the language must be installed in a collection, similar to the way that `"racket"` or `"slideshow"` are collections that are distributed with Racket. Again, however, there’s an escape from this restriction: the `reader` language lets you specify a reader-level implementation of a language using a general module path. ## 2. Using `#lang`` ``reader` The `reader` language for `#lang` is similar to `s-exp`, in that it acts as a kind of meta-language. Whereas `s-exp` lets a programmer specify a module language at the expander layer of parsing, `reader` lets a programmer specify a language at the reader level. A `#lang reader` must be followed by a module path, and the specified module must provide two functions: `read` and `read-syntax`. The protocol is the same as for a `#reader` implementation, but for `#lang`, the `read` and `read-syntax` functions must produce a `module` form that is based on the rest of the input file for the module. The following `"literal.rkt"` module implements a language that treats its entire body as literal text and exports the text as a `data` string: `"literal.rkt"` ```racket #lang racket (require syntax/strip-context) (provide (rename-out [literal-read read] [literal-read-syntax read-syntax])) (define (literal-read in) (syntax->datum (literal-read-syntax #f in))) (define (literal-read-syntax src in) (with-syntax ([str (port->string in)]) (strip-context #'(module anything racket (provide data) (define data 'str))))) ``` The `"literal.rkt"` language uses `strip-context` on the generated `module` expression, because a `read-syntax` function should return a syntax object with no lexical context. Also, the `"literal.rkt"` language creates a module named `anything`, which is an arbitrary choice; the language is intended to be used in a file, and the longhand module name is ignored when it appears in a `require`d file. The `"literal.rkt"` language can be used in a module `"tuvalu.rkt"`: `"tuvalu.rkt"` ```racket #lang reader "literal.rkt" Technology! System! Perfect! ``` Importing `"tuvalu.rkt"` binds `data` to a string version of the module content: ```racket > (require "tuvalu.rkt") > data "\nTechnology!\nSystem!\nPerfect!\n" ``` ## 3. Using `#lang`` ``s-exp`` ``syntax/module-reader` Parsing a module body is usually not as trivial as in `"literal.rkt"`. A more typical module parser must iterate to parse multiple forms for a module body. A language is also more likely to extend Racket syntax—perhaps through a readtable—instead of replacing Racket syntax completely. The `syntax/module-reader` module language abstracts over common parts of a language implementation to simplify the creation of new languages. In its most basic form, a language implemented with `syntax/module-reader` simply specifies the module language to be used for the language, in which case the reader layer of the language is the same as Racket. For example, with `"raquet-mlang.rkt"` ```racket #lang racket (provide (except-out (all-from-out racket) lambda) (rename-out [lambda function])) ``` and `"raquet.rkt"` ```racket #lang s-exp syntax/module-reader "raquet-mlang.rkt" ``` then ```racket #lang reader "raquet.rkt" (define identity (function (x) x)) (provide identity) ``` implements and exports the `identity` function, since `"raquet-mlang.rkt"` exports `lambda` as `function`. The `syntax/module-reader` language accepts many optional specifications to adjust other features of the language. For example, an alternate `read` and `read-syntax` for parsing the language can be specified with `#:read` and `#:read-syntax`, respectively. The following `"dollar-racket.rkt"` language uses `"dollar.rkt"` \(see \[missing\]\) to build a language that is like `racket` but with a `$` escape to simple infix arithmetic: `"dollar-racket.rkt"` ```racket #lang s-exp syntax/module-reader racket #:read $-read #:read-syntax $-read-syntax (require (prefix-in $- "dollar.rkt")) ``` The `require` form appears at the end of the module, because all of the keyword-tagged optional specifications for `syntax/module-reader` must appear before any helper imports or definitions. The following module uses `"dollar-racket.rkt"` to implement a `cost` function using a `$` escape: `"store.rkt"` ```racket #lang reader "dollar-racket.rkt" (provide cost) ; Cost of ‘n' $1 rackets with 7% sales ; tax and shipping-and-handling fee ‘h': (define (cost n h) $n*107/100+h$) ``` ## 4. Installing a Language So far, we have used the `reader` meta-language to access languages like `"literal.rkt"` and `"dollar-racket.rkt"`. If you want to use something like `#lang literal` directly, then you must move `"literal.rkt"` into a Racket collection named `"literal"` \(see also \[missing\]\). Specifically, move `"literal.rkt"` to a `reader` submodule of `"literal/main.rkt"` for any directory name `"literal"`, like so: `"literal/main.rkt"` ```racket #lang racket (module reader racket (require syntax/strip-context) (provide (rename-out [literal-read read] [literal-read-syntax read-syntax])) (define (literal-read in) (syntax->datum (literal-read-syntax #f in))) (define (literal-read-syntax src in) (with-syntax ([str (port->string in)]) (strip-context #'(module anything racket (provide data) (define data 'str)))))) ``` Then, install the `"literal"` directory as a package:   `cd /path/to/literal ; raco pkg install` After moving the file and installing the package, you can use `literal` directly after `#lang`: ```racket #lang literal Technology! System! Perfect! ``` > See \[missing\] for more information on using `raco`. You can also make your language available for others to install by using the Racket package manager \(see \[missing\]\). After you create a `"literal"` package and register it with the Racket package catalog \(see \[missing\]\), others can install it using `raco pkg`:   `raco pkg install literal` Once installed, others can invoke the language the same way: by using `#lang literal` at the top of a source file. If you use a public source repository \(e.g., GitHub\), you can link your package to the source. As you improve the package, others can update their version using `raco pkg`:   `raco pkg update literal` > See \[missing\] for more information about the Racket package manager. ## 5. Source-Handling Configuration The Racket distribution includes a Scribble language for writing prose documents, where Scribble extends the normal Racket to better support text. Here is an example Scribble document: `#lang scribble/base` `@(define (get-name) "Self-Describing Document")` `@title[(get-name)]` `The title of this document is “@(get-name).”` If you put that program in DrRacket’s definitions area and click Run, then nothing much appears to happen. The `scribble/base` language just binds and exports `doc` as a description of a document, similar to the way that `"literal.rkt"` exports a string as `data`. Simply opening a module with the language `scribble/base` in DrRacket, however, causes a Scribble HTML button to appear. Furthermore, DrRacket knows how to colorize Scribble syntax by coloring green those parts of the document that correspond to literal text. The language name `scribble/base` is not hard-wired into DrRacket. Instead, the implementation of the `scribble/base` language provides button and syntax-coloring information in response to a query from DrRacket. If you have installed the `literal` language as described in Installing a Language, then you can adjust `"literal/main.rkt"` so that DrRacket treats the content of a module in the `literal` language as plain text instead of \(erroneously\) as Racket syntax: `"literal/main.rkt"` ```racket #lang racket (module reader racket (require syntax/strip-context) (provide (rename-out [literal-read read] [literal-read-syntax read-syntax]) get-info) (define (literal-read in) (syntax->datum (literal-read-syntax #f in))) (define (literal-read-syntax src in) (with-syntax ([str (port->string in)]) (strip-context #'(module anything racket (provide data) (define data 'str))))) (define (get-info in mod line col pos) (lambda (key default) (case key [(color-lexer) (dynamic-require 'syntax-color/default-lexer 'default-lexer)] [else default])))) ``` This revised `literal` implementation provides a `get-info` function. The `get-info` function is called by `read-language` \(which DrRacket calls\) with the source input stream and location information, in case query results should depend on the content of the module after the language name \(which is not the case for `literal`\). The result of `get-info` is a function of two arguments. The first argument is always a symbol, indicating the kind of information that a tool requests from the language; the second argument is the default result to be returned if the language does not recognize the query or has no information for it. After DrRacket obtains the result of `get-info` for a language, it calls the function with a `'color-lexer` query; the result should be a function that implements syntax-coloring parsing on an input stream. For `literal`, the `syntax-color/default-lexer` module provides a `default-lexer` syntax-coloring parser that is suitable for plain text, so `literal` loads and returns that parser in response to a `'color-lexer` query. The set of symbols that a programming tool uses for queries is entirely between the tool and the languages that choose to cooperate with it. For example, in addition to `'color-lexer`, DrRacket uses a `'drracket:toolbar-buttons` query to determine which buttons should be available in the toolbar to operate on modules using the language. The `syntax/module-reader` language lets you specify `get-info` handling through a `#:info` optional specification. The protocol for an `#:info` function is slightly different from the raw `get-info` protocol; the revised protocol allows `syntax/module-reader` the possibility of handling future language-information queries automatically. ## 6. Module-Handling Configuration Suppose that the file `"death-list-5.rkt"` contains `"death-list-5.rkt"` ```racket #lang racket (list "O-Ren Ishii" "Vernita Green" "Budd" "Elle Driver" "Bill") ``` If you `require` `"death-list-5.rkt"` directly, then it prints the list in the usual Racket result format: ```racket > (require "death-list-5.rkt") '("O-Ren Ishii" "Vernita Green" "Budd" "Elle Driver" "Bill") ``` However, if `"death-list-5.rkt"` is required by a `"kiddo.rkt"` that is implemented with `scheme` instead of `racket`: `"kiddo.rkt"` ```racket #lang scheme (require "death-list-5.rkt") ``` then, if you run `"kiddo.rkt"` file in DrRacket or if you run it directly with `racket`, `"kiddo.rkt"` causes `"death-list-5.rkt"` to print its list in traditional Scheme format, without the leading quote: `("O-Ren Ishii" "Vernita Green" "Budd" "Elle Driver" "Bill")` The `"kiddo.rkt"` example illustrates how the format for printing a result value can depend on the main module of a program instead of the language that is used to implement it. More broadly, certain features of a language are only invoked when a module written in that language is run directly with `racket` \(as opposed to being imported into another module\). One example is result-printing style \(as shown above\). Another example is REPL behavior. These features are part of what’s called the _run-time configuration_ of a language. Unlike the syntax-coloring property of a language \(as described in Source-Handling Configuration\), the run-time configuration is a property of a _module_ per se as opposed to a property of the _source text_ representing the module. For that reason, the run-time configuration for a module needs to be available even if the module is compiled to bytecode form and the source is unavailable. Therefore, run-time configuration cannot be handled by the `get-info` function we’re exporting from the language’s parser module. Instead, it will be handled by a new `configure-runtime` submodule that we’ll add inside the parsed `module` form. When a module is run directly with `racket`, `racket` looks for a `configure-runtime` submodule. If it exists, `racket` runs it. But if the module is imported into another module, the `'configure-runtime` submodule is ignored. \(And if the `configure-runtime` submodule doesn’t exist, `racket` just evaluates the module as usual.\) That means that the `configure-runtime` submodule can be used for any special setup tasks that need to happen when the module is run directly. Going back to the `literal` language \(see Source-Handling Configuration\), we can adjust the language so that directly running a `literal` module causes it to print out its string, while using a `literal` module in a larger program simply provides `data` without printing. To make this work, we will need an extra module. \(For clarity here, we will implement this module as a separate file. But it could equally well be a submodule of an existing file.\) ```racket .... (the main installation or the user’s space) |- "literal" |- "main.rkt" (with reader submodule) |- "show.rkt" (new) ``` * The `"literal/show.rkt"` module will provide a `show` function to be applied to the string content of a `literal` module, and also provide a `show-enabled` parameter that controls whether `show` actually prints the result. * The new `configure-runtime` submodule in `"literal/main.rkt"` will set the `show-enabled` parameter to `#t`. The net effect is that `show` will print the strings that it’s given, but only when a module using the `literal` language is run directly \(because only then will the `configure-runtime` submodule be invoked\). These changes are implemented in the following revised `"literal/main.rkt"`: `"literal/main.rkt"` ```racket #lang racket (module reader racket (require syntax/strip-context) (provide (rename-out [literal-read read] [literal-read-syntax read-syntax]) get-info) (define (literal-read in) (syntax->datum (literal-read-syntax #f in))) (define (literal-read-syntax src in) (with-syntax ([str (port->string in)]) (strip-context #'(module anything racket (module configure-runtime racket (require literal/show) (show-enabled #t)) (require literal/show) (provide data) (define data 'str) (show data))))) (define (get-info in mod line col pos) (lambda (key default) (case key [(color-lexer) (dynamic-require 'syntax-color/default-lexer 'default-lexer)] [else default])))) ``` Then the `"literal/show.rkt"` module must provide the `show-enabled` parameter and `show` function: `"literal/show.rkt"` ```racket #lang racket (provide show show-enabled) (define show-enabled (make-parameter #f)) (define (show v) (when (show-enabled) (display v))) ``` With all of the pieces for `literal` in place, try running the following variant of `"tuvalu.rkt"` directly and through a `require` from another module: `"tuvalu.rkt"` ```racket #lang literal Technology! System! Perfect! ``` When run directly, we’ll see the result printed like so, because our `configure-runtime` submodule will have set the `show-enabled` parameter to `#t`: `Technology! System! Perfect!` But when imported into another module, printing will be suppressed, because the `configure-runtime` submodule will not be invoked, and therefore the `show-enabled` parameter will remain at its default value of `#f`.