#lang scribble/manual
@(require (for-label pollen/render racket/base pollen/core txexpr pollen/setup pollen/template pollen/pagetree sugar))
@(require "mb-tools.rkt")
@title[#:tag "second-tutorial"]{Second tutorial: Markdown, templates, & pagetrees}
In this tutorial, you'll use Pollen to publish a multiple-page article written in Markdown. You'll learn about:
@itemlist[
@item{Using Markdown files with the preprocessor}
@item{X-expressions}
@item{Markdown authoring mode}
@item{Templates}
@item{Pagetrees}
]
If you want the shortest possible introduction to Pollen, try the @secref["quick-tour"].
@section[#:tag-prefix "tutorial-2"]{Prerequisites}
I'll assume you've completed the @seclink["first-tutorial"]{first tutorial} and you understand how to create source files in DrRacket and view them in the project server. I won't be spelling out those tasks as I did before.
@section[#:tag "the-case-against-markdown"]{Optional reading: the case against Markdown}
I recognize that people like Markdown. I want people to like Pollen too, so that's why Pollen supports Markdown.
But just to be clear about my own views:
I'm mystified by the popularity of Markdown among writers. I can agree that it's a clever and readable way of notating basic HTML. And sure, that makes it great for things like web comments, where speed and simplicity are primary virtues.
In longer-form writing, however, its shortcomings become evident. Like programming languages, the best writing tools maximize expressive possibilities, and minimize constraints. But Markdown is hugely constrained. First and worst, Markdown isn't semantic. It only knows about formatting, and in that regard, isn't that much of an improvement on tools like Microsoft Word. Second, even as a formatting-notation tool, it's limited to a small subset of the already-small set of formatting tags permitted in HTML. Third, it can't be extended by an author.
An animating principle of Pollen, as explained in the @secref["Backstory"], is that after 20 years, we ought to move beyond thinking of HTML as a source format. Since Markdown is just well-disguised HTML, a vote for Markdown is really a vote to continue the status quo (albeit with fewer angle brackets). For me, that's not good enough. I'm ready for the tools to expand to fit my ideas. I refuse to keep cutting down my ideas to fit the tools.
All that said, if you genuinely prefer Markdown, I'm not looking to pry it from your fingers. Pollen has great Markdown support (due entirely to Greg Hendershott's excellent @link["https://github.com/greghendershott/markdown/"]{Markdown parser} for Racket). It makes Markdown more useful.
But let's make a deal, Markdown fans. Having met you more than halfway, will you at least consider that @seclink["pollen-vs-xml"]{Pollen markup} might be a better option for you than Markdown? Because it can notate anything that's in your brain, not just a subset of HTML? And if @secref["the-book-is-a-program"], the source for that book should look more like your brain, and less like HTML (or XML or LaTeX or ...)?
That's all I ask.
@section{Markdown in Pollen: two options}
There are two ways to use Markdown within Pollen:
@itemlist[#:style 'ordered
@item{Send Markdown files through the preprocessor.}
@item{Use Markdown authoring mode.}]
The preprocessor approach is better if you want to end up with a set of Markdown files that can be passed along to a HTML converter (or other Markdown-to-______ converter) downstream.
The authoring-mode approach is better if you want to end up with something other than Markdown, e.g., finished HTML files.
But we'll look at both.
@subsection{Using Markdown with the preprocessor}
Because Markdown is a text-based format, you can use the Pollen preprocessor to add programmatic features to existing Markdown files. (See @secref["Working_with_the_preprocessor"] in the @seclink["first-tutorial"]{first tutorial} if you need a refresher.)
Suppose we have a Markdown file called @filepath{brennan.md} that we want to use with the preprocessor. Create this file in DrRacket, save it, and start the project server in that directory.
@fileblock["brennan.md"
@codeblock[#:keep-lang-line? #f]{
#lang pollen
My name is _Brennan_, and I enjoy:
+ boring sauce
+ 24 fish nuggets
}]
You'll be able to see this file in the project server, but for now, it's just a static file. Pollen isn't doing anything to it.
Let's change that. Consistent with the usual preprocessor practice, add @tt{#lang pollen} as the first line, and append the @filepath{.pp} file extension, so our new preprocessor-ready file looks like this:
@fileblock["brennan.md.pp"
@codeblock{
#lang pollen
My name is _Brennan_, and I enjoy:
+ boring sauce
+ 24 fish nuggets
}]
Go back to the project server and you'll see the new filename. When you click on it, Pollen will render a new @filepath{brennan.md} file, but it will look the same as the one you had before.
Now we'll change some of the values using Pollen commands:
@fileblock["brennan.md.pp"
@codeblock{
#lang pollen
◊(define sauce-type "fancy")
◊(define nugget-type "chicken")
◊(define nugget-quantity (* 2 2 3))
My name is _Brennan_, and I enjoy:
+ ◊sauce-type sauce
+ ◊nugget-quantity ◊nugget-type nuggets
}]
When you reload this file in the project server, @filepath{brennan.md} will be regenerated, and will now look like this:
@terminal{
My name is _Brennan_, and I enjoy:
+ fancy sauce
+ 12 chicken nuggets}
Instead of running Markdown files through the preprocessor, you can also use Markdown authoring mode within Pollen. This is the better choice if you want to end up with rendered HTML files.
But first, let's pause to clarify the general concept of an authoring mode.
@subsection{Authoring mode}
Though the preprocessor is useful, it limits you to inserting chunks of text at various positions into an existing file.
Pollen's @defterm{authoring mode}, by contrast, parses the whole source file into a special data structure called an @defterm{X-expression}. You can then process the whole X-expression any way you like, and output to any format you like — or multiple formats — using a @defterm{template}.
Compared to the preprocessor, authoring mode offers more abstraction and flexibility. Of course, it also requires a bit more effort to set up.
Pollen offers two authoring modes: one that uses Markdown syntax (which we'll cover later in this tutorial) and the other that uses a free-form markup syntax (which we'll cover in the @seclink["third-tutorial"]{third tutorial}). In both cases, the basic process is the same: 1) compile the source into an X-expression, and then 2) convert that X-expression to the target file format using a template.
@subsection{X-expressions}
@(noskip-note)
I avoid nerdy jargon when I can. But in this case, the thing is called an @defterm{X-expression} throughout the Racket documentation, for good reasons. So I use the term too. Better to acclimate you now.
An X-expression is a way of representing markup-based data in code. X-expressions are indigenous to Lisp-based languages like Pollen and Racket. They don't exist in Python or JavaScript or Ruby.
Let's start with the part you're familiar with. By ``markup-based data,'' I mean things like HTML and XML and SVG. The idea is that you have text-based data surrounded by @defterm{tags}. Each tag can also have its own @defterm{attributes} that are made of keys and values. Tags can contain other tags, thus creating a tree-like structure. Right? You know what I mean:
@terminal{
Hello world
Nice to see you.
}
An X-expression is just a simplified, generalized method of notation for these data structures — much like Markdown is a simplified method of notation for HTML. To see the relationship, we'll convert one into the other.
First, we change the angle brackets to parentheses, and only use them on the outside of tags:
@terminal{(body
(h1 Hello world /h1)
(p class="first" Nice to (i see /i) you. /p)
/body)}
Then we get rid of the closing tags, which are superfluous, since each closing parenthesis suffices to mark the end of a tag:
@terminal{(body
(h1 Hello world)
(p class="first" Nice to (i see) you.))}
However, this creates ambiguity between the name of the tag and the content. So we'll put the content within double quotes:
@terminal{(body
(h1 "Hello world")
(p class="first" "Nice to" (i "see") "you."))}
As for the @code{class} attribute, we need to distinguish it from both the markup tags and the content, so we'll move it into another pair of parentheses:
@terminal{(body
(h1 "Hello world")
(p ((class "first")) "Nice to" (i "see") "you."))}
Whitespace isn't significant, so the X-expression can also be written without linebreaks:
@terminal{(body (h1 "Hello world") (p ((class "first")) "Nice to" (i "see") "you."))}
Skipping a few boring details, that's basically all there is to it.
So why is it called an X-expression? Lisp languages are built out of units called S-expressions, which look like this:
@terminal{(and (txexpr? x) (memq (get-tag x) (setup:block-tags)))}
S-expressions use prefix notation, where each pair of parentheses contains a list. The first element in the list names a function, and the other elements are the arguments to that function. (This is a review of @secref["Racket_basics__if_you_re_not_familiar_"].) X-expressions just adapt S-expression notation to represent markup, hence the name (the @defterm{X} is short for @defterm{XML-like}).
For handling markup-based data, X-expressions have some clear advantages over other methods:
@itemlist[
@item{@bold{Readability.} X-expressions retain the semantics of markup-based data while dispensing with the infamous verbosity.}
@item{@bold{A hybrid between a tree and a string.} Most programming languages represent markup-based data either as a string or as an XML tree. Neither is a good choice. The string captures none of the internal structure of the data. An XML tree captures the structure, but conceals the sequential nature of the data elements. The X-expression shows both.}
@item{@bold{An ideal match for an expression-based programming language.} Aside from some notational details, X-expressions are just a subset of S-expressions generally, which are the building block of Racket. Processing X-expressions in Racket maximizes flexibility and minimizes @link["https://projects.csail.mit.edu/gsb/old-archive/gsb-archive/gsb2000-02-11.html"]{yak-shaving}.}
]
@margin-note{Given the close kinship between XML-ish data structures and Lisp-ish programming languages, I have no explanation why, during the Internet era, they have not been paired more often. They're like peanut butter and jelly.}
In Pollen's authoring modes, your source file is compiled into an X-expression, which is then injected into a template & converted to output. As a first example, we'll look at Markdown authoring mode.
@subsection{Markdown authoring mode}
Let's start putting together our multiple-page article. For simplicity, I'm going to use unrealistically short sample texts. But you can use whatever Markdown content you want.
We want to use Markdown authoring mode to make a file that will ultimately be HTML. So consistent with Pollen file-naming conventions (see @secref["Saving___naming_your_source_file"]), we'll start with our desired output filename, @filepath{article.html}, and then append the Markdown authoring suffix, which is @filepath{.pmd}. So in DrRacket, start a new file called @filepath{article.html.pmd} and put some Markdown in it:
@fileblock["article.html.pmd"
@codeblock{
#lang pollen
Deep Thought
============
I am **so** happy to be writing this.
}]
Before you preview this file in the project server, click the @onscreen{Run} button in DrRacket just to see what the file produces. You'll see something like this:
@repl-output{
'(root
(h1 ((id "deep-thought")) "Deep Thought")
(p "I am " (strong "so") " happy to be writing this."))
}
You should now be able to recognize this as an X-expression. In authoring mode, Pollen compiles your Markdown into the corresponding HTML entities, but then provides the data as an X-expression rather than finished HTML.
From what you learned in the last section, it should be evident that this X-expression will convert to HTML that looks like this:
@terminal{
Deep Thought
I am so happy to be writing this.
}
``But what's this @code{root} tag? That's not HTML.'' An X-expression that holds other X-expressions must have a root tag. So in the spirit of obviousness, every X-expression produced by Pollen in authoring mode will start with @code{root}. If you don't need it, you can discard it (we'll cover this below, in @secref[#:tag-prefixes '("tutorial-2")]{Templates}). Though as you'll learn in the @seclink["third-tutorial"]{third tutorial}, @code{root} also creates a useful hook for further processing — it's not a superfluous accessory.
@subsection{Review: authoring mode vs. preprocessor mode}
Before we proceed, let's quickly review how authoring mode is different from preprocessor mode. To do that, let's take the same Markdown content, but this time put it into a preprocessor source file called @filepath{article.md.pp}.
@fileblock["article.md.pp"
@codeblock{
#lang pollen
Deep Thought
============
I am **so** happy to be writing this.
}]
When you run this file in DrRacket, you'll see:
@repl-output{
Deep Thought
============
I am **so** happy to be writing this.
}
This result makes sense, right? To recap: when you use Markdown source in preprocessor mode, Pollen gives you Markdown. When you use Markdown source in authoring mode, Pollen gives you an X-expression.
So how do you convert an X-expression into HTML? Read on.
@section[#:tag-prefix "tutorial-2"]{Templates}
In Pollen, a @defterm{template} lets you convert an X-expression to your target output format. If you've used other web-publishing systems, templates are probably a familiar idea. Templates in Pollen are both similar and different.
First, the two major similarities:
@itemlist[
@item{At its simplest, a template holds @bold{boilerplate material} that you want to reuse across multiple output files. For instance, in a set of HTML pages, you might have layout and navigation elements that stay the same, while the content changes. In that case, you could put the layout and navigation in the template, and keep the content in your Pollen source files. When you want to add a new page, you can make a new source file and just use it with the existing template. Moreover, if you want to change the layout and navigation globally, you can just change the template, rather than changing the source files.}
@item{Pollen templates, like others, can also have @bold{conditional elements} — meaning, you can embed simple code in your templates that allows them to change based on the content in the page. For instance, a template could show or hide ``previous page'' and ``next page'' links depending on whether there's actually a previous or next page. (Jump ahead to the @seclink["third-tutorial"]{third tutorial} if this excites you.)}
]
And two major differences:
@itemlist[
@item{There's @bold{no special template language} you need to learn, with magic syntax and whatnot. Instead, you can use all the same Pollen commands in a template that you can in authoring mode or preprocessor mode.}
@item{Within a template, you have to @bold{explicitly convert} the X-expression to the target format. This is a feature, not a bug. By avoiding assumptions about the target format, Pollen templates can be used to generate any kind of file (even binary formats like PDF). But the cost of this flexibility is that you need to tell Pollen what you want.}
]
@margin-note{``So a template is also a Pollen source file?'' Not quite. More accurately, it's a fragment of Pollen source that is completed by adding the X-expression that comes out of one of your source files. Because of this, there are a few extra limitations on the code you can put in a template, though with easy workarounds.}
To see how this works, let's return to the source file we started in the last section:
@fileblock["article.html.pmd"
@codeblock{
#lang pollen
Deep Thought
============
I am **so** happy to be writing this.
}]
Last time, I had you run this file in DrRacket to see the X-expression it produced. This time, load it in the project server. You'll see something like this:
@browser{
@bold{@larger{Deep Thought}}
I am @bold{so} happy to be writing this.
}
Here, you're seeing the X-expression from your source combined with an HTML template, which adds the necessary boilerplate for the finished HTML:
@terminal{
Deep Thought
I am so happy to be writing this.
}
But wait — where did the template come from? If you try to render a file to HTML without first setting up a template, Pollen helps you out and uses its @defterm{fallback template} for HTML. The fallback template is just a minimal template that's used as a last resort. Under ordinary circumstances, seeing the fallback template probably signals a problem (e.g., Pollen couldn't find the template you asked for). For now, we can rely on it to get a preview of our file.
Still, we can learn a few things about how to make an HTML template by studying the fallback template.
@subsection{The @tt{doc} export and the @tt{->html} function}
To understand the necessary parts of a template, let's look at a simple one — the fallback template that Pollen uses for HTML files, called @filepath{fallback.html}.
@fileblock["fallback.html"
@codeblock[#:keep-lang-line? #f]{
#lang pollen
◊(->html (html (head (meta #:charset "UTF-8")) (body doc)))
}]
The fallback template has three key ingredients:
The first ingredient is an X-expression that represents a basic HTML page:
@codeblock[#:keep-lang-line? #f]{
#lang pollen
(html (head (meta #:charset "UTF-8")) (body))
}
That X-expression is equivalent to this HTML string:
@terminal{}
But how do we get from X-expression to HTML? Here's how: within the template, we'll explicitly convert this X-expression to HTML. We do this by sending the X-expression through our second key ingredient, the function @racket[->html]:
@codeblock[#:keep-lang-line? #f]{
#lang pollen
◊(->html (html (head (meta #:charset "UTF-8")) (body)))
}
@racket[->html] is in the @racketmodname[pollen/template] module, but this module is automatically imported into every template, so there's nothing extra we need to do.
Finally, we need to include the X-expression from our source file. By convention, every Pollen source file makes its output available through an exported variable called @code{doc}. A source file in preprocessor mode puts its text result in @code{doc}. And a source file in authoring mode puts its X-expression result in @code{doc}. So we put the third key ingredient, the variable @code{doc}, inside our @code{body} tag.
@codeblock[#:keep-lang-line? #f]{
#lang pollen
◊(->html (html (head (meta #:charset "UTF-8")) (body doc)))
}
To summarize: this template contains a skeletal HTML page (in X-expression format). We drop @code{doc} into the template to indicate where the X-expression of our source file should be inserted. Finally, we convert the whole X-expression to HTML with @racket[->html].
``So I have to convert my HTML template to an X-expression?'' No. That's optional. You can also put hard-coded HTML in your template. Here's an equivalent way of writing @filepath{fallback.html.p}, with explicit HTML:
@fileblock["fallback.html.p" @codeblock[#:keep-lang-line? #f]{
#lang pollen
◊(->html doc)
}]
Notice that we still need to use the @racket[->html] function, but this time, instead of surrounding a larger X-expression, it just goes around @code{doc}. The rest of the template code gets passed through as plain text.
What matters most is ending up with HTML. Otherwise, there is no difference between these two approaches. In the first, you're describing the whole template using an X-expression and converting the whole thing with @racket[->html]. In the second, you're hard-coding the boilerplate HTML, so the only part that needs to be converted with @racket[->html] is @code{doc}.
Use whichever method works best for you. I often prefer the second method, because I like to build HTML layouts by hand using placeholder content to make sure all the fiddly bits work. Then it's easy to replace the placeholder content with @racket[(->html doc)], and it becomes a template.
On the other hand, when you use X-expressions to generate HTML, the result is always perfect. You'll never have to comb through a template looking for a missing closing @code{} tag.
@subsection{Making a custom template}
Having learned about the three key template ingredients, we'll now make a custom template for @filepath{article.html.pmd}.
In general, template files can have any name you want. But by default, Pollen will first look for a file in your project directory called @filepath{template.ext}, where @filepath{ext} matches the output-file extension of the source file. So if your source file is @filepath{database.xml.pmd}, Pollen will look for @filepath{template.xml}. And for @filepath{article.html.pmd}, Pollen will look for @filepath{template.html}.
@margin-note{If you want to use a template with a different name, or apply a special template to a particular source file, you can specify a template from within any source file by inserting the line @code{◊(define-meta template "my-template-name.html")}. For more, see @racket[get-template-for].}
Furthermore, it's also fine to use a Pollen source file that will result in @filepath{template.html}. Following the fallback-template example above, we'll use the null extension and name our template @filepath{template.html.p} — Pollen will convert this to the @filepath{template.html} it needs.
Beyond that, all we need to do make sure our template has the three key ingredients we saw in the fallback template. Pollen will automatically apply it to @filepath{article.html.pmd} when we view it in the project server.
In your project directory, create a new file called @filepath{template.html.p}:
@fileblock["template.html.p"
@codeblock[#:keep-lang-line? #f]{
#lang pollen
Custom template
◊(->html doc)
}]
The @filepath{p} extension is Pollen's @seclink["Null___p_extension_"]{@defterm{null extension}}, which does what you might guess — nothing. The template would still work if it were called @filepath{template.html}. But using the null extension is a virtuous habit, because it keeps your templates distinct from output files (both for your sake, and your operating system, which may get flummoxed by things like @filepath{template.pdf}, which would be a text-based file with a binary extension.
Otherwise, this is the same as the fallback template we saw in the previous section, but written out in HTML, and with a @code{title} element added. Refresh @filepath{article.html} in the project server. Does it look different? No — it won't, because the template is basically the same. But because of the new @code{title} field, you should notice that the title shown in the browser window is @onscreen{Custom template}. This shows that Pollen is correctly relying on our new template file, rather than the fallback template.
Let's keep going, and add a @code{style} block to our custom template:
@fileblock["template.html.p"
@codeblock[#:keep-lang-line? #f]{
#lang pollen
Custom template
◊(->html doc)
}]
Refresh @filepath{article.html} in the project server again. Now you'll see that the heading has a gray background, and one word in the text is red.
Feel free to add other settings to @filepath{template.html.p}, or update the text in @filepath{article.html.pmd}, and see how the page changes in the browser. As you'd expect, the project server keeps an eye on both your source files and your template files, and if one changes, it regenerates the output file.
@subsection{Inserting specific source data into templates}
In the previous example, we used @code{doc} to insert the entire content of the source file — as an X-expression — into the template.
But what if you want to only insert part of your source file into the template? For instance, you'll look like a dork if the title on each page is @onscreen{Custom template}. So let's fix that.
When you're working in a template, Pollen provides a @racket[select] function that lets you extract the content of a specific tag, like so: @code{◊(select tag-name doc)}, which means ``get the content of @racketvarfont{tag-name} out of @code{doc} and put it here.''
Let's suppose that we'd rather use the name of the article — @italic{Deep Thought} — as the page title. To do this, we'll put a @racket[select] command inside the @code{} tag.
To make @racket[select] work, we also need to know the tag name that contains the title. If we have a little Markdown expertise, we might already know that this part of our Markdown source:
@codeblock[#:keep-lang-line? #f]{
#lang pollen
Deep Thought
============
}
is going to produce a tag named @code{h1}.
What if we don't have all the Markdown conversions memorized? No problem. We can still figure out the tag name by running the @filepath{article.html.pmd} source file in DrRacket and looking at the resulting X-expression:
@repl-output{'(root (h1 ((id "my-article")) "Deep Thought") (p "I am "
(strong "so") " happy to be writing this."))}
Either way, now we know that the text @italic{Deep Thought} lives in the @code{h1} tag. So we update our template accordingly (for brevity, I'm going to omit the @code{style} tag in these examples, but it's fine to leave it in):
@fileblock["template.html.p"
@codeblock[#:keep-lang-line? #f]{
#lang pollen
<html>
<head>
<meta charset="UTF-8">
<title>◊(select 'h1 doc)
◊(->html doc)