6 Second tutorial
In this tutorial, you’ll use Pollen to publish a multiple-page article written in Markdown. You’ll learn about:
Using Markdown files with the preprocessor
X-expressions
Markdown authoring mode
Templates
Pagetrees
If you want the shortest possible introduction to Pollen, try the Quick tour.
6.1 Prerequisites
I’ll assume you’ve completed the First tutorial and you understand how to create source files in DrRacket and view them in the project server. I will not be spelling out those tasks as I did before.
6.2 Prelude: my principled objection to Markdown
I recognize that people like Markdown. I want people to like Pollen too, so that’s why Markdown support exists in Pollen. But just to be clear about my own views —
I’m mystified by the popularity of Markdown among writers. I can agree that it’s a clever and readable way of notating basic HTML. And sure, that makes it great for things like web comments, where speed and simplicity are primary virtues.
In longer-form writing, however, its shortcomings become evident. Like programming languages, the best writing tools maximize expressive possibilities, and minimize constraints. But Markdown is hugely constrained. First and worst, Markdown isn’t semantic. It only knows about formatting, and in that regard, isn’t that much of an improvement on tools like Microsoft Word. Second, even as a formatting-notation tool, it’s limited to a small subset of the already-small set of formatting tags permitted in HTML. Third, it can’t be extended by an author.
An animating principle of Pollen, as explained in the Backstory, is that after 20 years, we ought to move beyond thinking of HTML as an source format. Since Markdown is just well-disguised HTML, a vote for Markdown is really a vote to continue the status quo (albeit with fewer angle brackets). For me, that’s not good enough. I’m ready for the tools to expand to fit my ideas; I don’t want to keep cutting down my ideas to fit the tools.
All that said, if you genuinely prefer Markdown, I’m not looking to pry it from your fingers. Pollen has excellent Markdown support (due entirely to Greg Hendershott’s excellent Markdown parser for Racket). It makes Markdown more useful.
But let’s make a deal, Markdown fans. Having met you more than halfway, will you at least consider that Pollen markup might be a better option for you than Markdown? Because it can notate anything that’s in your brain, not just a subset of HTML? And if The book is a program, the source for that book should look more like your brain, and less like HTML?
That’s all I ask.
6.3 Markdown in Pollen: two options
There are two ways to use Markdown within Pollen: you can either send Markdown files through the preprocessor, or use Markdown authoring mode.
The preprocessor approach is better if you want to end up with a set of Markdown files that can be passed along to a HTML converter (or other Markdown-to-______ converter) elsewhere.
The authoring-mode approach is better if you want to end up with something other than Markdown, e.g., finished HTML files.
6.3.1 Using Markdown with the preprocessor
Because Markdown is a text-based format, you can use the Pollen preprocessor to add programmatic features to existing Markdown files. (See Working with the preprocessor in the First tutorial if you need a refresher.)
Suppose we have a Markdown file called brennan.md that we want to use with the preprocessor. Create this file in DrRacket, save it, and start the project server in that directory.
"brennan.md"
My name is _Brennan_, and I enjoy:
+ boring sauce
+ 24 fish nuggets
You’ll be able to see this file in the project server, but for now, it’s just a static file. Pollen isn’t doing anything to it.
Let’s change that. Consistent with the usual preprocessor practice, we add #lang pollen as the first line, and append the .pp file extension, so our new preprocessor-ready file looks like this:
"brennan.md.pp"
#lang pollen
My name is _Brennan_, and I enjoy:
+ boring sauce
+ 24 fish nuggets
Go back to the project server and you’ll see the new filename. When you click on it, Pollen will render a new markdown.md file, but it will look the same as the one you had before.
Now we’ll change some of the values using Pollen commands:
"brennan.md.pp"
#lang pollen
◊(define sauce-type "fancy")
◊(define nugget-type "chicken")
◊(define nugget-quantity (* 2 2 3))
My name is _Brennan_, and I enjoy:
+ ◊sauce-type sauce
+ ◊nugget-quantity ◊nugget-type nuggets
When you reload this file in the project server, brennan.md will be regenerated, and will now look like this:
My name is _Brennan_, and I enjoy:
+ fancy sauce
+ 12 chicken nuggets
Instead of running Markdown files through the preprocessor, you can also use Markdown authoring mode within Pollen. This is the better choice if you want to end up with rendered HTML files.
But first, let’s pause to clarify the general concept of an authoring mode.
6.3.2 Authoring mode
Though the preprocessor is useful, it limits you to inserting chunks of text at various positions into an existing file.
Pollen’s authoring mode, by contrast, parses the whole source file into a special data structure called an X-expression. You can then process the whole X-expression any way you like, and output to any format you like — or multiple formats — using a template.
Compared to the preprocessor, authoring mode offers more abstraction and flexibility. Of course, it’s also a bit more effort to set up.
Pollen offers two variants of authoring mode: one that uses Markdown syntax (which we’ll cover later in this tutorial) and the other that uses a free-form markup syntax (which we’ll cover in the next tutorial). In both cases, the basic idea is the same: parse the source into an X-expression, and then output it using a template.
6.3.3 X-expressions
Don’t skip this part! It’s not the same old shit. And it describes a concept that’s key to how Pollen works.
I avoid nerdy jargon whenever possible. But in this case, the thing is called an X-expression throughout the Racket documentation, for good reasons. So I use the term too. Better to acclimate you now.
An X-expression is a way of representing markup-based data in code. X-expressions are indigenous to Lisp-based languages like Pollen and Racket. They don’t exist in Python or JavaScript or Ruby.
Let’s start with the part you’re familiar with. By “markup-based data,” I mean things like HTML and XML and SVG. The idea is that you have text-based data surrounded by tags. Each tag can also have its own attributes that are made of keys and values. Tags can contain other tags, thus creating a tree-like structure. Right? You know what I mean:
<body><h1>Hello world</h1><p class="first">Nice to <i>see</i> you.</p></body>
An X-expression is just a simplified, generalized method of notation for these data structures — much like Markdown is a simplified method of notation for HTML. To see the relationship, we’ll convert one into the other.
First, we change the angle brackets to parentheses, and only use them on the outside of tags:
(body (h1 Hello world /h1) (p class="first" Nice to (i see /i) you. /p) /body)
Then we get rid of the closing tags, which are superfluous, since the closing parenthesis marks the end of the tag perfectly well:
(body (h1 Hello world) (p class="first" Nice to (i see) you.))
However, this creates ambiguity between the name of the tag and the content. So we’ll put the content within quote marks:
(body (h1 "Hello world") (p class="first" "Nice to" (i "see") "you."))
As for the class attribute, we need to distinguish it from both the markup tags and the content, so we’ll move it between double parentheses:
(body (h1 "Hello world") (p ((class "first")) "Nice to" (i "see") "you."))
Net of a few boring details, that’s basically all there is to it.
So why is it called an X-expression? Lisp languages are built out of units called S-expressions, which look like this:
(and (txexpr? x) (member (get-tag x) (project-block-tags)) #t))
S-expressions use prefix notation, where each pair of parentheses contains a list. The first element in the list names a function, and the other elements are the arguments to that function. (This is a review of Racket basics (if you’re not familiar).) X-expressions are just a minor adaptation of S-expression notation to represent markup, hence the name (the X is short for XML-like).
For handling markup-based data, X-expressions have some useful advantages compared to other methods:
Readability. X-expressions retain all the semantics of markup-based data while dispensing with the infamous verbosity.
A hybrid between a tree and a string. Most programming languages represent markup-based data either as a string or as an XML tree. Neither is a good choice. The string captures none of the internal structure of the data. An XML tree captures the structure, but conceals the sequential nature of the data elements. The X-expression gets both.
An ideal match for an expression-based programming language. Aside from some notational details, X-expressions are just a subset of S-expressions generally, which are the building block of Racket. Processing X-expressions in Racket maximizes flexibility and minimizes yak-shaving.
Given the close kinship between XML-ish data structures and Lisp-ish programming languages, I have no explanation why, during the Internet era, they have not been paired more often.
In Pollen’s authoring modes, your source file is parsed into an X-expression, which can then be processed further before being injected into a template & converted to output. As a first example, we’ll look at Markdown authoring mode.
6.3.4 Markdown authoring mode
Let’s start putting together our article. For simplicity, I’m going to use unrealistically short sample texts. But you can use whatever Markdown content you want.
We want to use Markdown authoring mode to make a file that will ultimately be HTML. So consistent with Pollen file-naming conventions (see Saving & naming your source file), we’ll start with our desired output filename, article.html, and then append the Markdown authoring suffix, .pmd. So in DrRacket, start a new file called article.html.pmd and put some Markdown in it:
"article.html.pmd"
#lang pollen
Deep Thought
============
I am **so** happy to be writing this.
Before you preview this file in the project server, click the Run button in DrRacket just to see what the file produces. You’ll see something like this:
’(root (h1 ((id "my-article")) "Deep Thought") (p () "I am " +
6 Second tutorial On this page:6.0.1.136 Second tutorial
In this tutorial, you’ll use Pollen to publish a multiple-page article written in Markdown. You’ll learn about:
Using Markdown files with the preprocessor
X-expressions
Markdown authoring mode
Templates
Pagetrees
If you want the shortest possible introduction to Pollen, try the Quick tour.
6.1 Prerequisites
I’ll assume you’ve completed the First tutorial and you understand how to create source files in DrRacket and view them in the project server. I will not be spelling out those tasks as I did before.
6.2 Prelude: my principled objection to Markdown
I recognize that people like Markdown. I want people to like Pollen too, so that’s why Markdown support exists in Pollen. But just to be clear about my own views —
I’m mystified by the popularity of Markdown among writers. I can agree that it’s a clever and readable way of notating basic HTML. And sure, that makes it great for things like web comments, where speed and simplicity are primary virtues.
In longer-form writing, however, its shortcomings become evident. Like programming languages, the best writing tools maximize expressive possibilities, and minimize constraints. But Markdown is hugely constrained. First and worst, Markdown isn’t semantic. It only knows about formatting, and in that regard, isn’t that much of an improvement on tools like Microsoft Word. Second, even as a formatting-notation tool, it’s limited to a small subset of the already-small set of formatting tags permitted in HTML. Third, it can’t be extended by an author.
An animating principle of Pollen, as explained in the Backstory, is that after 20 years, we ought to move beyond thinking of HTML as a source format. Since Markdown is just well-disguised HTML, a vote for Markdown is really a vote to continue the status quo (albeit with fewer angle brackets). For me, that’s not good enough. I’m ready for the tools to expand to fit my ideas; I don’t want to keep cutting down my ideas to fit the tools.
All that said, if you genuinely prefer Markdown, I’m not looking to pry it from your fingers. Pollen has excellent Markdown support (due entirely to Greg Hendershott’s excellent Markdown parser for Racket). It makes Markdown more useful.
But let’s make a deal, Markdown fans. Having met you more than halfway, will you at least consider that Pollen markup might be a better option for you than Markdown? Because it can notate anything that’s in your brain, not just a subset of HTML? And if The book is a program, the source for that book should look more like your brain, and less like HTML?
That’s all I ask.
6.3 Markdown in Pollen: two options
There are two ways to use Markdown within Pollen: you can either send Markdown files through the preprocessor, or use Markdown authoring mode.
The preprocessor approach is better if you want to end up with a set of Markdown files that can be passed along to a HTML converter (or other Markdown-to-______ converter) elsewhere.
The authoring-mode approach is better if you want to end up with something other than Markdown, e.g., finished HTML files.
6.3.1 Using Markdown with the preprocessor
Because Markdown is a text-based format, you can use the Pollen preprocessor to add programmatic features to existing Markdown files. (See Working with the preprocessor in the First tutorial if you need a refresher.)
Suppose we have a Markdown file called brennan.md that we want to use with the preprocessor. Create this file in DrRacket, save it, and start the project server in that directory.
"brennan.md"
My name is _Brennan_, and I enjoy:
+ boring sauce
+ 24 fish nuggets
You’ll be able to see this file in the project server, but for now, it’s just a static file. Pollen isn’t doing anything to it.
Let’s change that. Consistent with the usual preprocessor practice, we add #lang pollen as the first line, and append the .pp file extension, so our new preprocessor-ready file looks like this:
"brennan.md.pp"
#lang pollen
My name is _Brennan_, and I enjoy:
+ boring sauce
+ 24 fish nuggets
Go back to the project server and you’ll see the new filename. When you click on it, Pollen will render a new markdown.md file, but it will look the same as the one you had before.
Now we’ll change some of the values using Pollen commands:
"brennan.md.pp"
#lang pollen
◊(define sauce-type "fancy")
◊(define nugget-type "chicken")
◊(define nugget-quantity (* 2 2 3))
My name is _Brennan_, and I enjoy:
+ ◊sauce-type sauce
+ ◊nugget-quantity ◊nugget-type nuggets
When you reload this file in the project server, brennan.md will be regenerated, and will now look like this:
My name is _Brennan_, and I enjoy:
+ fancy sauce
+ 12 chicken nuggets
Instead of running Markdown files through the preprocessor, you can also use Markdown authoring mode within Pollen. This is the better choice if you want to end up with rendered HTML files.
But first, let’s pause to clarify the general concept of an authoring mode.
6.3.2 Authoring mode
Though the preprocessor is useful, it limits you to inserting chunks of text at various positions into an existing file.
Pollen’s authoring mode, by contrast, parses the whole source file into a special data structure called an X-expression. You can then process the whole X-expression any way you like, and output to any format you like — or multiple formats — using a template.
Compared to the preprocessor, authoring mode offers more abstraction and flexibility. Of course, it’s also a bit more effort to set up.
Pollen offers two variants of authoring mode: one that uses Markdown syntax (which we’ll cover later in this tutorial) and the other that uses a free-form markup syntax (which we’ll cover in the next tutorial). In both cases, the basic idea is the same: parse the source into an X-expression, and then output it using a template.
6.3.3 X-expressions
Don’t skip this part! It’s not the same old shit. And it describes a concept that’s key to how Pollen works.
I avoid nerdy jargon whenever possible. But in this case, the thing is called an X-expression throughout the Racket documentation, for good reasons. So I use the term too. Better to acclimate you now.
An X-expression is a way of representing markup-based data in code. X-expressions are indigenous to Lisp-based languages like Pollen and Racket. They don’t exist in Python or JavaScript or Ruby.
Let’s start with the part you’re familiar with. By “markup-based data,” I mean things like HTML and XML and SVG. The idea is that you have text-based data surrounded by tags. Each tag can also have its own attributes that are made of keys and values. Tags can contain other tags, thus creating a tree-like structure. Right? You know what I mean:
<body><h1>Hello world</h1><p class="first">Nice to <i>see</i> you.</p></body>
An X-expression is just a simplified, generalized method of notation for these data structures — much like Markdown is a simplified method of notation for HTML. To see the relationship, we’ll convert one into the other.
First, we change the angle brackets to parentheses, and only use them on the outside of tags:
(body (h1 Hello world /h1) (p class="first" Nice to (i see /i) you. /p) /body)
Then we get rid of the closing tags, which are superfluous, since the closing parenthesis marks the end of the tag perfectly well:
(body (h1 Hello world) (p class="first" Nice to (i see) you.))
However, this creates ambiguity between the name of the tag and the content. So we’ll put the content within quote marks:
(body (h1 "Hello world") (p class="first" "Nice to" (i "see") "you."))
As for the class attribute, we need to distinguish it from both the markup tags and the content, so we’ll move it between double parentheses:
(body (h1 "Hello world") (p ((class "first")) "Nice to" (i "see") "you."))
Net of a few boring details, that’s basically all there is to it.
So why is it called an X-expression? Lisp languages are built out of units called S-expressions, which look like this:
(and (txexpr? x) (member (get-tag x) (project-block-tags)) #t))
S-expressions use prefix notation, where each pair of parentheses contains a list. The first element in the list names a function, and the other elements are the arguments to that function. (This is a review of Racket basics (if you’re not familiar).) X-expressions are just a minor adaptation of S-expression notation to represent markup, hence the name (the X is short for XML-like).
For handling markup-based data, X-expressions have some useful advantages compared to other methods:
Readability. X-expressions retain all the semantics of markup-based data while dispensing with the infamous verbosity.
A hybrid between a tree and a string. Most programming languages represent markup-based data either as a string or as an XML tree. Neither is a good choice. The string captures none of the internal structure of the data. An XML tree captures the structure, but conceals the sequential nature of the data elements. The X-expression gets both.
An ideal match for an expression-based programming language. Aside from some notational details, X-expressions are just a subset of S-expressions generally, which are the building block of Racket. Processing X-expressions in Racket maximizes flexibility and minimizes yak-shaving.
Given the close kinship between XML-ish data structures and Lisp-ish programming languages, I have no explanation why, during the Internet era, they have not been paired more often.
In Pollen’s authoring modes, your source file is parsed into an X-expression, which can then be processed further before being injected into a template & converted to output. As a first example, we’ll look at Markdown authoring mode.
6.3.4 Markdown authoring mode
Let’s start putting together our article. For simplicity, I’m going to use unrealistically short sample texts. But you can use whatever Markdown content you want.
We want to use Markdown authoring mode to make a file that will ultimately be HTML. So consistent with Pollen file-naming conventions (see Saving & naming your source file), we’ll start with our desired output filename, article.html, and then append the Markdown authoring suffix, .pmd. So in DrRacket, start a new file called article.html.pmd and put some Markdown in it:
"article.html.pmd"
#lang pollen
Deep Thought
============
I am **so** happy to be writing this.
Before you preview this file in the project server, click the Run button in DrRacket just to see what the file produces. You’ll see something like this:
’(root (h1 ((id "my-article")) "Deep Thought") (p () "I am "
(strong () "so") " happy to be writing this."))You should now be able to recognize this as an X-expression. In authoring mode, Pollen parses your Markdown into the corresponding HTML entities, but then provides the data as an X-expression rather than finished HTML.
The empty parentheses () after p and strong signal that the tag’s attributes are empty. When you write an X-expression without attributes, these parentheses are optional — (tag () "text") and (tag "text") are equivalent — but Pollen will always print X-expressions this way.
From what you learned in the last section, it should be evident that this X-expression corresponds to HTML that looks like this:
<root><h1 id="my-article">Deep Thought</h1><p>I am
<strong>so</strong> happy to be writing this.</p></root>“But what’s this root tag? That’s not HTML.” An X-expression must have a root tag, so in the spirit of obviousness, every X-expression produced by a source file in authoring mode will start with root. If you don’t need it, you can discard it. But it also creates a useful hook for further processing, as we’ll see later.
By the way, as review, let’s remind ourselves how this is different from preprocessor mode. Let’s take the same Markdown content, but this time put it into a preprocessor source file called article.md.pp.
"article.md.pp"
#lang pollen
Deep Thought
============
I am **so** happy to be writing this.
When you run this file in DrRacket, you’ll see:
Deep Thought
============
I am **so** happy to be writing this.Hopefully, this result makes sense to you: when you run Markdown source in preprocessor mode, you get Markdown. When you run Markdown source in authoring mode, you get an X-expression.
6.4 Templates
So how do you convert an X-expression into a finished file? You use a Pollen template, which takes data from an X-expression and converts it to the target format.
If you’ve used other web-publishing systems, templates are probably a familiar idea. Templates in Pollen are in some ways similar to the ones you’ve seen before, but in other ways different.
First, the similarities. At its simplest, a template holds boilerplate material that you want to reuse across multiple pages. For instance, in a set of HTML pages, you might have layout and navigation elements that stay the same, while the content changes. In that case, you could put the layout and navigation in the template, and keep the content in your Pollen source files. When you want to add a new page, you can make a new source file and just use it with the existing template. Moreover, if you want to change the layout and navigation globally, you can just change the template, rather than changing the source files.
Pollen templates, like others, can also have conditional features — meaning, you can embed simple code in your templates that allows them to change based on the content in the page. For instance, a template could show or hide “previous page” and “next page” links depending on whether there’s actually a previous or next page.
The major difference with Pollen templates is that there’s no special “template language” you need to learn, with magic delimiters and whatnot. Instead, you can use all the same Pollen commands in a template that you can in authoring mode or preprocessor mode.
To see a template in action, let’s return to the source file we started in the last section:
"article.html.pmd"
#lang pollen
Deep Thought
============
I am **so** happy to be writing this.
Last time, I had you run this file in DrRacket to see the X-expression it produced. This time, load it in the project server. You’ll see something like this:
Deep Thought
I am so happy to be writing this.
Here, you’re seeing the X-expression from your source combined with an HTML template, which adds the necessary boilerplate for the finished HTML:
<html><head><meta charset="UTF-8" /></head><body>
<root><h1 id="my-article">Deep Thought</h1><p>I am
<strong>so</strong> happy to be writing this.</p></root> diff --git a/scribblings/tutorial-second.scrbl b/scribblings/tutorial-second.scrbl index 9eda5c7..195f591 100644 --- a/scribblings/tutorial-second.scrbl +++ b/scribblings/tutorial-second.scrbl @@ -34,7 +34,7 @@ I'm mystified by the popularity of Markdown among writers. I can agree that it's In longer-form writing, however, its shortcomings become evident. Like programming languages, the best writing tools maximize expressive possibilities, and minimize constraints. But Markdown is hugely constrained. First and worst, Markdown isn't semantic. It only knows about formatting, and in that regard, isn't that much of an improvement on tools like Microsoft Word. Second, even as a formatting-notation tool, it's limited to a small subset of the already-small set of formatting tags permitted in HTML. Third, it can't be extended by an author. -An animating principle of Pollen, as explained in the @secref["Backstory" #:doc '(lib "pollen/scribblings/pollen.scrbl")], is that after 20 years, we ought to move beyond thinking of HTML as an source format. Since Markdown is just well-disguised HTML, a vote for Markdown is really a vote to continue the status quo (albeit with fewer angle brackets). For me, that's not good enough. I'm ready for the tools to expand to fit my ideas; I don't want to keep cutting down my ideas to fit the tools. +An animating principle of Pollen, as explained in the @secref["Backstory" #:doc '(lib "pollen/scribblings/pollen.scrbl")], is that after 20 years, we ought to move beyond thinking of HTML as a source format. Since Markdown is just well-disguised HTML, a vote for Markdown is really a vote to continue the status quo (albeit with fewer angle brackets). For me, that's not good enough. I'm ready for the tools to expand to fit my ideas; I don't want to keep cutting down my ideas to fit the tools. All that said, if you genuinely prefer Markdown, I'm not looking to pry it from your fingers. Pollen has excellent Markdown support (due entirely to Greg Hendershott's excellent @link["https://github.com/greghendershott/markdown/"]{Markdown parser} for Racket). It makes Markdown more useful.