Pollen: the book is a program

Pollen is a publishing system that helps authors create beautiful and functional web-based books. Pollen includes tools for writing, designing, programming, testing, and publishing.

I used Pollen to create my book Butterick’s Practical Typography. Sure, go take a look. Is it better than the last digital book you encountered? Yes it is. Would you like your book to look like that? If so, keep reading.

That’s what Pollen is for.

Not that you need to be a programmer to use Pollen. On the contrary, the Pollen language is markup-based, so you can write & edit text naturally. But when you want to automate repetitive tasks, add cross-references, or pull in data from other sources, you can access a full programming language from within the text.

That language is Racket. I chose Racket because while the idea for Pollen had been with me for several years, it simply wasn’t possible to build it with other languages. So if it’s unfamiliar to you, don’t panic. It was unfamiliar to me. Once you see what you can do with Pollen & Racket, you may be persuaded. I was.

Or, if you can find a better digital publishing tool, use that. Personally, I’m never going back to the way I used to work.

1 Installation

Install Racket, which includes DrRacket.

2 Quick start

2.1 You had me at Hello World

Launch DrRacket. Open a new document.

Change the top line to #lang pollen. This will be the first line of any source file where you want to invoke the Pollen language.

Type Hello World underneath.

Run the file by clicking the Run button, or typing cmd-R.

The result window will show Hello World.

Congratulations, you’ve made your first Pollen document.

If you like, change the text and run the file again. The new text will appear in the result window.

Pollen is a programming language that operates in text mode by default. Meaning, all plain text in the source file is considered valid input, and gets passed through intact.

2.2 The plot thickens

Start a new Pollen document. Remember to change the top line.

Underneath, type Hello ◊(+ 1 2) Worlds. The character before the left parenthesis is called a lozenge. Type it by [doing such and such].

Ask yourself: what are you likely to get when you run the file?

OK, now run the file.

The result will be Hello 3 Worlds. Hopefully, that’s what you expected.

Feel free to change the numbers inside the parenthesized expression and run the file again. The printed sum will change. You can also change the + sign to a * sign and make really big numbers. If you want to see your first stupid Pollen trick, type Hello ◊(/ 38 57) of a World and watch what happens.

Erase everything but the top line.

Type this: ◊(define name "Roxy") Hello ◊name.

What do you suppose you’ll get this time?

Run the file. You’ll see Hello Roxy.

The lozenge character (◊) tells Pollen to interpret what follows as code rather than plain text. This character is therefore the gateway to all the programming functions available in Pollen. In the first case, it denoted a math expression. In the second case, it denoted the definition of a variable, and then the variable itself.

2.3 Making an HTML page with Pollen

By default, Pollen operates in preprocessor mode. That means it evaluates all the expressions in your document, renders each as text, and then outputs the whole document as a text file.

In this tutorial, you’re going to make an HTML file. But you can use Pollen as a preprocessor for any kind of text file.

That means Pollen can act as a preprocessor for CSS, JavaScript, XML — and even source files for other programming languages.

2.3.1 Making an HTML page with Pollen in decoder mode

2.3.2 Using the Pollen dashboard

3 Why I made Pollen

The nerds have already raced ahead to the quick tutorial. That’s okay. Because software isn’t just data structures and functions. It’s ideas, and choices, and policies. It’s design.

I created Pollen to overcome certain tool limitations that surfaced repeatedly in my work. If you agree with my characterization of those problems, then you’ll probably like the solution that Pollen offers.

If not, you probably won’t.

3.1 The web-development problem

I made my first web page in 1994, shortly after the web was invented. I opened my text editor (at the time, BBEdit) and pecked out <html><body>Hello world</body></html>, then loaded it in Mosaic. So did a million others.

If you weren’t around then, you didn’t miss much. Everything about the web was horrible: the web browsers, the computers running the browsers, the dial-up connections feeding the browsers, and of course HTML itself. At that point, the desktop-software experience was already slick and refined. By comparison, using the web felt like banging rocks together.

That’s no longer true. The web is now 20 years old. During that time, most parts of the web have improved dramatically — the connections are faster, the browsers are more sophisticated, the screens have more pixels.

But one part has not: the way we make web pages. Over the years, tools promising to simplify HTML development have come and mostly gone — from PageMill to Dreamweaver. Meanwhile, true web jocks have remained loyal to the original HTML power tool: the humble text editor.

In one way, this makes sense. Web pages are mostly made of text — HTML, CSS, JavaScript, and so on — and thus the simplest way to mainpulate them is with a text editor. While HTML and CSS are not programming languages, they lend themselves to semantic and logical structure that’s most easily expressed by editing them as text. Text-based editing also makes debugging and performance improvements easier.

But text-based editing is also limited. Though the underlying description of a web page is notionally human-readable, it’s largely optimized to be readable by other software (namely, web browsers). HTML markup in particular is verbose and easily mistyped. And isn’t it fatally dull to manage all the boilerplate, like surrounding every paragraph with <p>...</p>? Yes, it is.

For these reasons, much of web development should lend itself to automation. But in practice, tools that enable this automation have been slow to arrive, and most come hobbled with unacceptable deficiencies.

3.1.1 Why not a content management system, like WordPress?

I used WordPress to make the original version of Typography for Lawyers (the precursor to Butterick’s Practical Typography). Even WordPress founder Matt Mullenweg thought it was “a cool use of WordPress for a mini-book.” Thanks, Matt. At the time, WordPress was the best tool for the job.

But at the risk of having my Gravatar revoked, I’ll tell you I became disenchanted with WordPress because:

It’s a resource hog.

Performance is questionable.

There’s always a new security problem.

No source control.

PHP.

3.1.2 Why not a CSS preprocessor, like Sass or LESS?

A CSS preprocessor automates the generation of CSS data. These preprocessors do save time & effort, so using one is better than not using one. My objection is that they ask you to incur much of the overhead of learning a programming language but without delivering the benefits. Because unlike a general-purpose programming language, Sass and LESS can only manipulate CSS. Better to learn a programming language that can manipulate anything.

3.1.3 Why not a static blog generator, like Jekyll or Pelican?

3.1.4 Why not a dynamic templating system, like Bottle?

4 Source formats

This puts Pollen into automatic mode, where the source file is interpreted according to the file extension.

If the file extension is “.pm”, the source is interpreted as pollen/markup.

If the file extension is “.pp”, the source is interpreted as pollen/pre (“pre” stands for “preprocessor”).

If the file extension is “.pmd”, the source is interpreted as pollen/markdown.

5 ◊ command overview

5.1 The golden rule

Pollen uses a special character — the lozenge, which looks like this: ◊ — to mark commands within a Pollen source file. So when you put a ◊ in your source, whatever comes next will be treated as a command. If you don’t, it will just be interpreted as plain text.

5.2 The lozenge glyph (◊)

I chose the lozenge as the command marker because a) it appears in almost every font, b) it’s barely used in ordinary typesetting, c) it’s not used in any programming language that I know of, and d) its shape and color allow it to stand out easily in code without being distracting.

Here’s how you type it:

Mac: option + shift + V
Windows:
Ubuntu:

Still, if you don’t want to use the lozenge as your command marker, you can use something else. Set Pollen’s world:command-marker value to whatever character you want.

Scribble uses the @ sign as a delimiter. It’s not a bad choice if you only work with Racket files. But as you use Pollen to work on other kinds of text-based files that commonly contain @ signs — HTML pages especially — it gets cumbersome. So I changed it.

But don’t knock the lozenge till you try it.

5.3 The two command modes: text mode & Racket mode

Pollen commands can be entered in one of two modes: text mode or Racket mode. Both modes start with a lozenge (◊):

◊ ‹command name› [ ‹Racket arguments ...› ] { ‹text argument› }

◊ ( ‹Racket expression› )

Text-mode commands

A text-mode command has the three possible parts after the ◊:

• The command name appears immediately after the ◊. Typically it’s a short word.

• The Racket arguments appear between square brackets. Pollen is partly an interface to the Racket programming language. These arguments are entered using Racket conventions — e.g., a string of text needs to be put in quotes as a "string of text". If you like programming, you’ll end up using these frequently. If you don’t, you won’t.

• The text argument appears between braces (aka curly brackets). You can put any ordinary text here. Unlike with the Racket arguments, you don’t put quotes around the text.

Each of the three parts is optional. You can also nest commands within each other. However:

• You can never have spaces between the three parts.

• Whatever parts you use must always appear in the order above.

Here are a few examples of correct text-mode commands:

#lang pollen

◊variable-name

◊tag{Text inside the tag.}

◊tag['attr: "value"]{Text inside the tag}

◊get-customer-id["Brennan Huff"]

◊tag{His ID is ◊get-customer-id["Brennan Huff"].}

And some incorrect examples:

#lang pollen

◊tag {Text inside the tag.} ; space between first and second parts

◊tag[Text inside the tag] ; text argument needs to be within braces

◊tag{Text inside the tag}['attr: "value"] ; wrong order

The next section describes each of these parts in detail.

Racket-mode commands

If you’re familiar with Racket expressions, you can use the Racket-mode commands to embed them within Pollen source files. It’s simple: any Racket expression can become a Pollen command by adding ◊ to the front. So in Racket, this code:

#lang racket

(define song "Revolution")

(format "~a #~a" song (* 3 3))

Can be converted to Pollen like so:

#lang pollen

◊(define song "Revolution")

◊(format "~a #~a" song (* 3 3))

And in DrRacket, they produce the same output:

Revolution #9

Beyond that, there’s not much to say about Racket mode — any valid expression you can write in Racket will also be a valid Racket-mode Pollen command.

The relationship of text mode and Racket mode

Even if you don’t plan to write a lot of Racket-mode commands, you should be aware that under the hood, Pollen is converting all commands in text mode to Racket mode. So a text-mode command that looks like this:

◊headline[#:size 'enormous]{Man Bites Dog!}

Is actually being turned into a Racket-mode command like this:

(headline #:size 'enormous "Man Bites Dog!")

Thus a text-mode command is just an alternate way of writing a Racket-mode command. (More broadly, all of Pollen is just an alternate way of using Racket.)

The corollary is that you can always write Pollen commands using whichever mode is more convenient or readable. For instance, the earlier example, written in the Racket mode:

#lang pollen

◊(define song "Revolution")

◊(format "~a #~a" song (* 3 3))

Can be rewritten using text mode:

#lang pollen

◊define[song]{Revolution}

◊format["~a #~a" song (* 3 3)]

And it will work the same way.

5.3.1 The command name

In Pollen, you’ll typically use the command name for one of four purposes:

• To invoke a tag function.

• To invoke another function.

• To insert the value of a variable.

• To insert a comment.

5.3.1.1 Invoking tag functions

By default, Pollen treats every command name as a tag function. As the name implies, a tag function creates a tagged X-expression with the command name as the tag, and the text argument as the content.

#lang pollen

◊strong{Fancy Sauce, $1}

'(strong "Fancy Sauce, $1")

To streamline markup, Pollen doesn’t restrict you to a certain set of tags, nor does it make you define your tag functions ahead of time. Just type a tag, and you can start using it.

The one restriction is that you can’t invent names for tag functions that are already being used for other commands. For instance, map is a name permanently reserved by the Racket function map. It’s also a rarely-used HTML tag. But gosh, you really want to use it. Problem is, if you invoke it directly, Pollen will think you mean the other map:

#lang pollen

◊map{Fancy Sauce, $1}

map: arity mismatch;
the expected number of arguments does not match the given number
  given: 1
  arguments...:
    "Fancy Sauce, $1"

What to do? Read on.

5.3.1.2 Invoking other functions

Though every command name starts out as a tag function, it doesn’t necessarily end there. You have two options for invoking other functions: defining your own , or invoking others from Racket.

Defining your own functions

Use the define command to create your own function for a command name. After that, when you use the command name, you’ll get the new behavior. For instance, recall this example showing the default tag-function behavior:

#lang pollen

◊strong{Fancy Sauce, $1}

'(strong "Fancy Sauce, $1")

We can define strong to do something else, like add to the text:

#lang pollen

◊(define (strong text) `(strong ,(format "Hey! Listen up! ~a" text)))

◊strong{Fancy Sauce, $1}

'(strong "Hey! Listen up! Fancy Sauce, $1")

The replacement function has to accept any arguments that might get passed along, but it doesn’t have to do anything with them. For instance, this function definition won’t work because strong is going to get a text argument that it’s not defined to handle:

#lang pollen

◊(define (strong) '(fib "1 1 2 3 5 8 13 ..."))

◊strong{Fancy Sauce, $1}

strong: arity mismatch;
the expected number of arguments does not match the given number
  expected: 0
  given: 1
  arguments...:
    "Fancy Sauce, $1"

Whereas in this version, strong accepts an argument called text, but then ignores it:

#lang pollen

◊(define (strong text) '(fib "1 1 2 3 5 8 13 ..."))

◊strong{Fancy Sauce, $1}

'(fib "1 1 2 3 5 8 13 ...")

You can attach any behavior to a command name. As your project evolves, you can also update the behavior of a command name. In that way, Pollen commands become a set of hooks to which you can attach more elaborate processing.

Using Racket functions

You aren’t limited to functions you define. Any function from Racket, or any Racket library, can be invoked directly by using it as a command name. Here’s the function range, which creates a list of numbers:

#lang pollen

◊range[1 20]

'(range 1 20)

Hold on — that’s not what we want. Where’s the list of numbers? The problem here is that we didn’t explicitly import the racket/list library, which contains the definition for range. (If you need to find out what library contains a certain function, the Racket documentation will tell you.) Without racket/list, Pollen just thinks we’re trying to use range as a tag function (and if we had been, then '(range 1 20) would’ve been the right result).

We fix this by using the require command to bring in the racket/list library, which contains the range we want:

#lang pollen

◊(require racket/list)

◊range[1 20]

'(1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19)

Of course, you can also invoke Racket functions indirectly, by attaching them to functions you define for command names:

#lang pollen

◊(require racket/list)

◊(define (rick start finish) (range start finish))

◊rick[1 20]

'(1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19)

Let’s return to the problem that surfaced in the last section — the fact that some command names can’t be used as tag functions because they’re already being used for other things. You can work around this by defining your own tag function with a non-conflicting name.

For instance, suppose we want to use map as a tag even though Racket is using it for its own function called map. First, we invent a command name that doesn’t conflict. Let’s call it my-map. As you learned above, Pollen will treat a new command name as a tag function by default:

#lang pollen

◊my-map{How I would love this to be a map.}

'(my-map "How I would love this to be a map.")

But my-map is not the tag we want. We need to define my-map to be a tag function for map. We can do this with the Pollen helper make-tag-function. That function lives in pollen/tag, so we require that too:

#lang pollen

◊(require pollen/tag)

◊(define my-map (make-tag-function 'map))

◊my-map{How I would love this to be a map.}

'(map "How I would love this to be a map.")

Problem solved.

5.3.1.3 Inserting the value of a variable

A Pollen command name usually refers to a function, but it can also refer to a variable, which is a data value. Once you define the variable, you can insert it into your source by using the ◊ notation without any other arguments:

#lang pollen

◊(define foo "bar")

The value of foo is ◊foo

The value of foo is bar

Be careful — if you include arguments, even blank ones, Pollen will treat the command name as a function. This won’t work, because a variable is not a function:

To understand what happens here, recall the relationship between Pollen’s command modes. The text-mode command ◊foo[] becomes the Racket-mode command (foo), which after variable substitution becomes ("bar"). If you try to evaluate ("bar") — e.g., in DrRacket — you’ll get the same error.

#lang pollen

◊(define foo "bar")

The value of foo is ◊foo[]

application: not a procedure;
expected a procedure that can be applied to arguments
  given: "bar"
  arguments...: [none]

The reason we can simply drop ◊foo into the text argument of another Pollen command is that the variable foo holds a string (i.e., a text value). When appropriate, Pollen will convert a variable to a string in a sensible way. For instance, numbers are easily converted:

#lang pollen

◊(define zam 42)

The value of zam is ◊zam

The value of zam is 42

If the variable holds a container datatype (like a list, hash, or vector), Pollen will produce the Racket text representation of the item. Here, zam is a list of integers:

#lang pollen

◊(define zam (list 1 2 3))

The value of zam is ◊zam

The value of zam is '(1 2 3)

This feature is included for your convenience as an author. But in general, your readers won’t want to see the Racket representation of a container. So in these cases, you should convert to a string manually in some sensible way. Here, the integers in the list are converted to strings, which are then combined using string-join from the racket/string library:

#lang pollen

◊(require racket/string)

◊(define zam (list 1 2 3))

The value of zam is ◊string-join[(map number->string zam)]{ and }

The value of zam is 1 and 2 and 3

Pollen will still produce an error if you try to convert an esoteric value to a string. Here, zam is the addition function (+):

#lang pollen

◊(define zam +)

The value of zam is ◊zam

Pollen decoder: can’t convert #<procedure:+> to string

One special case to know about. In the examples above, there’s a word space between the variable and the other text. But suppose you need to insert a variable into text so that there’s no space in between. The simple ◊ notation won’t work, because it won’t be clear where the variable name ends and the text begins.

For instance, suppose we want to use a variable edge next to the string px:

#lang pollen

◊(define edge 100)

p { margin-left: ◊edgepx; }

Pollen decoder: can’t convert #<procedure:...t/pollen/tag.rkt:6:2> to string

The example fails because Pollen reads the whole string after the ◊ as the single variable name edgepx. Since edgepx isn’t defined, it’s treated as a tag function, and since Pollen can’t convert a function to a string, we get an error.

In these situations, surround the variable name with vertical bars ◊|like so| to explicitly indicate where the variable name ends. The bars are not treated as part of the name, nor are they included in the result. Once we do that, we get what we intended:

#lang pollen

◊(define edge 100)

p { margin-left: ◊|edge|px; }

p { margin-left: 100px; }

If you use this notation when you don’t need to, nothing bad will happen. The vertical bars are always ignored.

#lang pollen

◊(define edge 100)

The value of edge is ◊|edge| pixels}

The value of edge is 100 pixels

5.3.1.4 Inserting a comment

Two options.

To comment out the rest of a single line, use a lozenge followed by a semicolon ◊;.

#lang pollen

◊span{This is not a comment}

◊span{Nor is this} ◊;span{But this is}

'(span "This is not a comment")
'(span "Nor is this")

To comment out a multiline block, use the lozenge–semicolon signal ◊; with curly braces, ◊;{like so}.

#lang pollen

◊;{

◊span{This is not a comment}

◊span{Nor is this} ◊;span{But this is}

}

Actually, it's all a comment now

Actually, it's all a comment now

5.3.2 The Racket arguments

The middle part of a text-mode Pollen command contains the Racket arguments [between square brackets.] Most often, you’ll see these used to pass extra information to commands that operate on text.

For instance, tag functions. Recall from before that any not-yet-defined command name in Pollen is treated as a tag function:

#lang pollen

◊title{The Beginning of the End}

'(title "The Beginning of the End")

But what if you wanted to add attributes to this tag, so that it comes out like this?

'(title ((class "red")(id "first")) "The Beginning of the End")

You can do it with Racket arguments.

Here’s the hard way. You can type out your list of attributes in Racket format and drop them into the brackets as a single argument:

#lang pollen

◊title['((class "red")(id "first"))]{The Beginning of the End}

'(title ((class "red") (id "first")) "The Beginning of the End")

But that’s a lot of parentheses to think about. So here’s the easy way. Anytime you use a tag function, there’s a shortcut for inserting attributes. You can enter them as a series of symbol / string pairs between the Racket-argument brackets. The only caveat is that the symbols have to begin with a quote mark ' and end with a colon :. So taken together, they look like this:

#lang pollen

◊title['class: "red" 'id: "first"]{The Beginning of the End}

'(title ((class "red") (id "first")) "The Beginning of the End")

Racket arguments can be any valid Racket expressions. For instance, this will also work:

#lang pollen

◊title['class: (format "~a" (* 6 7)) 'id: "first"]{The Beginning of the End}

'(title ((class "42") (id "first")) "The Beginning of the End")

Since Pollen commands are really just Racket arguments underneath, you can use those too. Here, we’ll define a variable called name and use it in the Racket arguments of title:

#lang pollen

◊(define name "Brennan")

◊title['class: "red" 'id: ◊name]{The Beginning of the End}

'(title ((class "read") (id "Brennan")) "The Beginning of the End")

You can also use this area for keyword arguments. Keyword arguments can be used to provide options for a particular Pollen command, to avoid redundancy. Suppose that instead of using the h1 ... h6 tags, you want to consolidate them into one command called heading and select the level separately. You can do this with a keyword, in this case #:level, which is passed as a Racket argument:

#lang pollen

◊(define (heading #:level which text)

`(,(string->symbol (format "h~a" which)) ,text))

◊heading[#:level 1]{Major league}

◊heading[#:level 2]{Minor league}

◊heading[#:level 6]{Trivial league}

'(h1 "Major league")
'(h2 "Minor league")
'(h6 "Trivial league")

5.3.3 The text argument

The third part of a text-mode Pollen command is the text argument. The text argument {appears between curly braces}. It can contain any text you want. The text argument can also contain other Pollen commands with their own text arguments. And they can contain other Pollen commands ... and so on, all the way down.

#lang pollen

◊div{Do it again. ◊div{And again. ◊div{And yet again.}}}

'(div "Do it again. " (div "And again. " (div "And yet again.")))

Three small details to know about the text argument.

First, the only character that needs special handling in a text argument is the lozenge ◊. A lozenge ordinarily marks a new command. So if you want an actual lozenge to appear in the text, you have to escape it by typing ◊"◊".

#lang pollen

◊definition{This is the lozenge: ◊"◊"}

'(definition "This is the lozenge: ◊")

Second, the whitespace-trimming policy. Here’s the short version: if there’s a carriage return at either end of the text argument, it is trimmed, and whitespace at the end of each line is selectively trimmed in an intelligent way. So this text argument, with carriage returns on the ends:

#lang pollen

◊div{

Roomy!

I agree.

}

'(div "Roomy!" "\n" "\n" "I agree.")

Yields the same result as this one:

#lang pollen

◊div{Roomy!

I agree.}

'(div "Roomy!" "\n" "\n" "I agree.")

For the long version, please see (part "Spaces, Newlines, and Indentation").

Third, within a multiline text argument, newline characters become individual strings that are not merged with adjacent text. So what you end up with is a list of strings, not a single string. That’s why in the last example, we got this:

'(div "Roomy!" "\n" "\n" "I agree.")

Instead of this:

'(div "Roomy!\n\nI agree.")

Under most circumstances, these two tagged X-expressions will behave the same way. The biggest exception is with functions. A function that operates on multiline text arguments needs to be able to handle an indefinite number of strings. For instance, this jejune function only accepts a single argument. It will work with a single-line text argument, because that produces a single string:

#lang pollen

◊(define (jejune text)

`(jejune ,text))

◊jejune{Irrational confidence}

'(jejune "Irrational confidence")

But watch what happens with a multiline text argument:

#lang pollen

◊(define (jejune text)

`(jejune ,text))

◊jejune{Deeply

chastened}

jejune: arity mismatch;
the expected number of arguments does not match the given number
  expected: 1
  given: 3
  arguments...:
   "Deeply"
   "\n"
   "chastened"

The answer is to use a rest argument in the function, which takes the “rest” of the arguments — however many there may be — and combines them into a single list. If we rewrite jejune with a rest argument, we can fix the problem:

#lang pollen

◊(define (jejune . texts)

`(jejune ,@texts))

◊jejune{Deeply

chastened}

'(jejune "Deeply" "\n" "chastened")

5.4 Further reading

The Pollen language is a variant of Racket’s own text-processing language, called Scribble. So many things that are true about Scribble are also true about Pollen. For the sake of clarity & brevity, I’ve omitted them from this summary. But if you want the full story:

[insert]

6 Module reference

6.1 Cache

The slowest part of a render is parsing and decoding the source file. Often, previewing a single source file necessarily means decoding others (for instance templates, or other source files that are linked into the main source file). But usually, only one source file is changing at a time. Therefore, Pollen stores copies of the exports of source files — namely, whatever is stored in doc and metas — in the cache so they can be reused.

The cache is a hash table that uses the complete path of a source file as its keys. The value associated with each of these keys is a subcache — another hash table with keys 'doc, 'metas (for storing the exports of the source file) and 'mod-time (for storing the modification time, provided by file-or-directory-modify-seconds).

The only keys supported are 'doc, 'metas, and 'mod-time.

If you want the speed benefit of the cache, you should always use cached-require to get data from Pollen source files. That doesn’t mean you can’t still use functions like require, local-require, and dynamic-require. They’ll just be slower.

6.2 Decode

The doc export of a Pollen markup file is a simple X-expression. Decoding refers to any post-processing of this X-expression. The pollen/decode module provides tools for creating decoders.

The decode step can happen separately from the compilation of the file. But you can also attach a decoder to the markup file’s root node, so the decoding happens automatically when the markup is compiled, and thus automatically incorporated into doc. (Following this approach, you could also attach multiple decoders to different tags within doc.)

You can, of course, embed function calls within Pollen markup. But since markup is optimized for authors, decoding is useful for operations that can or should be moved out of the authoring layer.

One example is presentation and layout. For instance, detect-paragraphs is a decoder function that lets authors mark paragraphs in their source simply by using two carriage returns.

Another example is conversion of output into a particular data format. Most Pollen functions are optimized for HTML output, but one could write a decoder that targets another format.

This function doesn’t do much on its own. Rather, it provides the hooks upon which harder-working functions can be hung.

Recall from (part "Pollen mechanics") that any tag can have a function attached to it. By default, the tagged-xexpr from a source file is tagged with root. So the typical way to use decode is to attach your decoding functions to it, and then define root to invoke your decode function. Then it will be automatically applied to every doc during compile.

For instance, here’s how decode is attached to root in Butterick’s Practical Typography. There’s not much to it —

(define (root . items)

(decode (make-txexpr 'root null items)

#:txexpr-elements-proc detect-paragraphs

#:block-txexpr-proc

(λ(bx) (wrap-hanging-quotes (nonbreaking-last-space bx)))

#:string-proc (compose1 smart-quotes smart-dashes)))

This illustrates another important point: even though decode presents an imposing list of arguments, you’re unlikely to use all of them at once. These represent possibilities, not requirements. For instance, let’s see what happens when decode is invoked without any of its optional arguments.

Right — nothing. That’s because the default value for the decoding arguments is the identity function, (λ (x) x). So all the input gets passed through intact unless another action is specified.

The *-proc arguments of decode take procedures that are applied to specific categories of elements within txexpr.

The txexpr-tag-proc argument is a procedure that handles X-expression tags.

The txexpr-attrs-proc argument is a procedure that handles lists of X-expression attributes. (The txexpr module, included at no extra charge with Pollen, includes useful helper functions for dealing with these attribute lists.)

Note that txexpr-attrs-proc will change the attributes of every tagged X-expression, even those that don’t have attributes. This is useful, because sometimes you want to add attributes where none existed before. But be careful, because the behavior may make your processing function overinclusive.

The txexpr-elements-proc argument is a procedure that operates on the list of elements that represents the content of each tagged X-expression. Note that each element of an X-expression is subject to two passes through the decoder: once now, as a member of the list of elements, and also later, through its type-specific decoder (i.e., string-proc, symbol-proc, and so on).

So why do you need txexpr-elements-proc? Because some types of element decoding depend on context, thus it’s necessary to handle the elements as a group. For instance, the doubling function above, though useless, requires handling the element list as a whole, because elements are being added.

A more useful example: paragraph detection. The behavior is not merely a map across each element:

The block-txexpr-proc argument and the inline-txexpr-proc arguments are procedures that operate on tagged X-expressions. If the X-expression meets the block-txexpr? test, it is processed by block-txexpr-proc. Otherwise, it is processed by inline-txexpr-proc. Thus every tagged X-expression will be handled by one or the other. Of course, if you want block and inline elements to be handled the same way, you can set block-txexpr-proc and inline-txexpr-proc to be the same procedure.

The string-proc, symbol-proc, valid-char-proc, and cdata-proc arguments are procedures that operate on X-expressions that are strings, symbols, valid-chars, and CDATA, respectively. Deliberately, the output contracts for these procedures accept any kind of X-expression (meaning, the procedure can change the X-expression type).

Finally, the tags-to-exclude argument is a list of tags that will be exempted from decoding. Though you could get the same result by testing the input within the individual decoding functions, that’s tedious and potentially slower.

The tags-to-exclude argument is useful if you’re decoding source that’s destined to become HTML. According to the HTML spec, material within a <style> or <script> block needs to be preserved literally. In this example, if the CSS and JavaScript blocks are capitalized, they won’t work. So exclude '(style script), and problem solved.

6.2.1 Blocks

Because it’s convenient, Pollen categorizes tagged X-expressions into two categories: block and inline. Why is it convenient? When using decode, you often want to treat the two categories differently. Not that you have to. But this is how you can.

(address article aside audio blockquote body canvas dd div dl fieldset figcaption figure footer form h1 h2 h3 h4 h5 h6 header hgroup noscript ol output p pre section table tfoot ul video)

Pollen tries to do the right thing without being told. But this is the rare case where you have to be explicit. If you introduce a tag into your markup that you want treated as a block, you must use this function to identify it, or you will get spooky behavior later on.

For instance, detect-paragraphs knows that block elements in the markup shouldn’t be wrapped in a p tag. So if you introduce a new block element called bloq without registering it as a block, misbehavior will follow:

But once you register bloq as a block, order is restored:

If you find the idea of registering block tags unbearable, good news. The project-block-tags include the standard HTML block tags by default. So if you just want to use things like div and p and h1–h6, you’ll get the right behavior for free.

6.2.2 Typography

An assortment of typography & layout functions, designed to be used with decode. These aren’t hard to write. So if you like these, use them. If not, make your own.