Add support for multiple rendering targets #49

Thank you for this project, and in particular the very thorough documentation.

I'm trying to use pollen to generate LaTeX code (specifically ConTeXt flavoured). I'm therefore not terribly interested in using the pollen server; I prefer to render with something like "raco pollen render document.tex.pp".

When doing so, is there a way to use a custom template file, or is that something setup by the server? Can I specify it on the command line (e.g. "raco pollen render --template my-template.tex document.tex.pp")? So far, I don't seem able to persuade the raco tool to use a custom template.

Thank you for this project, and in particular the very thorough documentation. I'm trying to use pollen to generate LaTeX code (specifically ConTeXt flavoured). I'm therefore not terribly interested in using the pollen server; I prefer to render with something like "raco pollen render document.tex.pp". When doing so, is there a way to use a custom template file, or is that something setup by the server? Can I specify it on the command line (e.g. "raco pollen render --template my-template.tex document.tex.pp")? So far, I don't seem able to persuade the raco tool to use a custom template.

When you’re using the .pp file extension, you’re invoking preprocessor mode, which doesn’t use a template. Pollen commands within the file are interpreted, but everything else is left intact.

If you want to use a template, you need to use an authoring mode, meaning either Markdown mode (.pmd extension) or Pollen markup (.pm extension) which I imagine would be the better choice here.

As for specifying a template, you can do that two ways. If your source file is document.tex.pm, then Pollen will automatically look for a template called template.tex. You can also specify a template explicitly within the source file by setting a meta like so: ◊meta['template: "my-template.tex"].

When you’re using the `.pp` file extension, you’re invoking [preprocessor mode](http://pkg-build.racket-lang.org/doc/pollen/quick-tour.html?q=preprocessor#%28part._.Pollen_as_a_preprocessor%29), which doesn’t use a template. Pollen commands within the file are interpreted, but everything else is left intact. If you want to use a template, you need to use an [authoring mode](http://pkg-build.racket-lang.org/doc/pollen/second-tutorial.html?q=preprocessor#%28part._.Authoring_mode%29), meaning either [Markdown mode](http://pkg-build.racket-lang.org/doc/pollen/second-tutorial.html?q=preprocessor#%28part._.Markdown_authoring_mode%29) (`.pmd` extension) or [Pollen markup](http://pkg-build.racket-lang.org/doc/pollen/third-tutorial.html?q=preprocessor#%28part._.Writing_with_.Pollen_markup%29) (`.pm` extension) which I imagine would be the better choice here. As for specifying a template, you can do that two ways. If your source file is `document.tex.pm`, then Pollen will [automatically look for](http://pkg-build.racket-lang.org/doc/pollen/third-tutorial.html?q=preprocessor#%28part._.The_template%29) a template called `template.tex`. You can also specify a template explicitly within the source file [by setting a `meta`](http://pkg-build.racket-lang.org/doc/pollen/Render.html?q=preprocessor#%28def._%28%28lib._pollen%2Frender..rkt%29._get-template-for%29%29) like so: `◊meta['template: "my-template.tex"]`.

PS I wouldn’t be opposed to adding a --template flag as you suggest, but I’d want to have a non-theoretical use case to test it against.

PS I wouldn’t be opposed to adding a `--template` flag as you suggest, but I’d want to have a non-theoretical use case to test it against.

Thank you for your helpful answer. Reading it I realised I didn't understand your project very well, so I've tried to get a better handle on the documentation before replying.

In terms of generating ConTeXt from a pollen markup file, I need somehow to generate strings of ConTeXt from the X-expressions. At first I thought that "decode" in pollen/decode was the way to go, but now I see that this only transforms tagged X-expressions into other X-expressions, rather than strings.

Then I thought that templates was a better approach, but ultimately I need to transform every X-expression into a string--can I do that in a template? Having read the documentation, it looks like finally all the X-expressions go through the function called "->html" which turns them into strings of HTML. Is there a way that I can code a template which will transform all of the X-expressions into strings representing ConTeXt? How can I tell Pollen that '(title "My Title")' means "\title{My Title}"? I don't need a generic converter (e.g. (x "foo") -> "\x{foo}), I'm happy to code each by hand.

I note from your talk on YouTube that you would like to add PDF output to this project. If I do create a template that allows transforming Pollen Markup files into ConTeXt, would you be interested in it? I could use the standard HTML tags to make it likely that pollen markup prepared for HTML publishing would also give reasonable output for ConTeXt, and hence PDF.

In terms of the command line flags, I'd prefer to come back to you with further suggestions once I've got more experience of rendering things. At the moment, I'm not rendering because I haven't worked out how to render into TeX.

Thank you for your helpful answer. Reading it I realised I didn't understand your project very well, so I've tried to get a better handle on the documentation before replying. In terms of generating ConTeXt from a pollen markup file, I need somehow to generate strings of ConTeXt from the X-expressions. At first I thought that "decode" in pollen/decode was the way to go, but now I see that this only transforms tagged X-expressions into other X-expressions, rather than strings. Then I thought that templates was a better approach, but ultimately I need to transform every X-expression into a string--can I do that in a template? Having read the documentation, it looks like finally all the X-expressions go through the function called "->html" which turns them into strings of HTML. Is there a way that I can code a template which will transform all of the X-expressions into strings representing ConTeXt? How can I tell Pollen that '(title "My Title")' means "\title{My Title}"? I don't need a generic converter (e.g. (x "foo") -> "\x{foo}), I'm happy to code each by hand. I note from your talk on YouTube that you would like to add PDF output to this project. If I do create a template that allows transforming Pollen Markup files into ConTeXt, would you be interested in it? I could use the standard HTML tags to make it likely that pollen markup prepared for HTML publishing would also give reasonable output for ConTeXt, and hence PDF. In terms of the command line flags, I'd prefer to come back to you with further suggestions once I've got more experience of rendering things. At the moment, I'm not rendering because I haven't worked out how to render into TeX.

Perhaps what's needed is a "->ConTeXt" function, which I can place in my templates like this: ◊->ConTeXt[doc]

I think what you’re overlooking is that strings are valid X-expressions. For instance, this sample tag function in the docs takes markup of the form (em ...) and converts it to the string "BOOM".

So, it’s no problem to convert Pollen markup into ConTeXt strings using a tag function:

#lang pollen/markup

◊(define (title . xs)
   (format "\\title{~a}" (apply string-append xs)))

◊title{My Title}

This will get you '(root "\\title{My Title}").

You can create these tag functions individually, or use define-syntax-rule to generalize the pattern:

#lang pollen/markup

◊(define-syntax-rule (make-ConTeXt-tag tag)
   (define (tag . xs)
     (format "\\~a{~a}" 'tag (apply string-append xs))))

◊(make-ConTeXt-tag title)
◊(make-ConTeXt-tag foo)

◊title{My Title}
◊foo{My Foo}

It’s conceivable that you could do something similar with decode. But decode works from the top down, whereas tag functions work from the bottom up. So tag functions are more helpful for making markup that composes correctly, e.g. nestable tags —

#lang pollen/markup

◊(define-syntax-rule (make-ConTeXt-tag tag)
   (define (tag . xs)
     (format "\\~a{~a}" 'tag (apply string-append xs))))

◊(make-ConTeXt-tag title)
◊(make-ConTeXt-tag foo)

◊title{My Title ◊foo{My Foo}}

As for whether you should use preprocessor mode (plain text, no template) or Pollen markup mode (with a template), if you’re already converting all your markup to text as you go (using tag functions), then there’s not necessarily a need for Pollen markup. For instance, if we take the example above and change it to preprocessor mode:

#lang pollen/pre

◊(define-syntax-rule (make-ConTeXt-tag tag)
   (define (tag . xs)
     (format "\\~a{~a}" 'tag (apply string-append xs))))

◊(make-ConTeXt-tag title)
◊(make-ConTeXt-tag foo)

◊title{My Title, ◊foo{My Foo}}

The output is just \title{My Title, \foo{My Foo}} rather than '(root "\\title{My Title, \\foo{My Foo}}"), which may suit you.

But if you wanted to use a template, yes you could do that too. You could create template.tex and as you suggest, do something like ◊->ConTeXt[doc]. That function might look like this:

#lang racket/base
(require txexpr)

(define (->ConTeXt x)
  (cond
    [(txexpr? x)
     (define tag (get-tag x))
     (define elements (get-elements x))
     (format "\\~a{~a}" tag (apply string-append (map ->ConTeXt elements)))]
    [else x]))

(->ConTeXt '(root (title "My Title, " (foo "My Foo"))))
> "\\root{\\title{My Title, \\foo{My Foo}}}"

I think what you’re overlooking is that strings are valid X-expressions. For instance, [this sample tag function in the docs](http://pkg-build.racket-lang.org/doc/pollen/third-tutorial.html#%28part._.Multiple_input_values___rest_arguments%29) takes markup of the form `(em ...)` and converts it to the string `"BOOM"`. So, it’s no problem to convert Pollen markup into ConTeXt strings using a tag function: ``` racket #lang pollen/markup ◊(define (title . xs) (format "\\title{~a}" (apply string-append xs))) ◊title{My Title} ``` This will get you `'(root "\\title{My Title}")`. You can create these tag functions individually, or use `define-syntax-rule` to generalize the pattern: ``` racket #lang pollen/markup ◊(define-syntax-rule (make-ConTeXt-tag tag) (define (tag . xs) (format "\\~a{~a}" 'tag (apply string-append xs)))) ◊(make-ConTeXt-tag title) ◊(make-ConTeXt-tag foo) ◊title{My Title} ◊foo{My Foo} ``` It’s conceivable that you could do something similar with `decode`. But `decode` works from the top down, whereas tag functions work from the bottom up. So tag functions are more helpful for making markup that composes correctly, e.g. nestable tags — ``` racket #lang pollen/markup ◊(define-syntax-rule (make-ConTeXt-tag tag) (define (tag . xs) (format "\\~a{~a}" 'tag (apply string-append xs)))) ◊(make-ConTeXt-tag title) ◊(make-ConTeXt-tag foo) ◊title{My Title ◊foo{My Foo}} ``` As for whether you should use preprocessor mode (plain text, no template) or Pollen markup mode (with a template), if you’re already converting all your markup to text as you go (using tag functions), then there’s not necessarily a need for Pollen markup. For instance, if we take the example above and change it to preprocessor mode: ``` racket #lang pollen/pre ◊(define-syntax-rule (make-ConTeXt-tag tag) (define (tag . xs) (format "\\~a{~a}" 'tag (apply string-append xs)))) ◊(make-ConTeXt-tag title) ◊(make-ConTeXt-tag foo) ◊title{My Title, ◊foo{My Foo}} ``` The output is just `\title{My Title, \foo{My Foo}}` rather than `'(root "\\title{My Title, \\foo{My Foo}}")`, which may suit you. But if you wanted to use a template, yes you could do that too. You could create `template.tex` and as you suggest, do something like `◊->ConTeXt[doc]`. That function might look like this: ``` racket #lang racket/base (require txexpr) (define (->ConTeXt x) (cond [(txexpr? x) (define tag (get-tag x)) (define elements (get-elements x)) (format "\\~a{~a}" tag (apply string-append (map ->ConTeXt elements)))] [else x])) (->ConTeXt '(root (title "My Title, " (foo "My Foo")))) > "\\root{\\title{My Title, \\foo{My Foo}}}" ```

Here is a use-case I have in mind: generating multiple outputs from a single input.

Suppose that I have a .pm document and now I want to render it both to HTML and whatever. Currently I have two options: using the Racket API myself or hard-linking the input file twice with different output extensions to make Pollen use two different templates for it. Note that meta is not an option in this case, but adding the ability to specify metas on command line would solve this (and would probably be useful in other situations).

Here is a use-case I have in mind: generating multiple outputs from a single input. Suppose that I have a `.pm` document and now I want to render it _both_ to HTML and whatever. Currently I have two options: using the Racket API myself or hard-linking the input file twice with different output extensions to make Pollen use two different templates for it. Note that `meta` is not an option in this case, but adding the ability to specify metas on command line would solve this (and would probably be useful in other situations).

I agree that there should be an integrated interface for converting one source file to multiple targets. But I also think that interface should be designed to work with all the Pollen tools (e.g., not just with raco pollen render but also the project server, also with DrRacket). So that’s why I was asking for a non-theoretical use case. When you say “HTML and whatever,” what is “whatever”?

I agree that there should be an integrated interface for converting one source file to multiple targets. But I also think that interface should be designed to work with all the Pollen tools (e.g., not just with `raco pollen render` but also the project server, also with DrRacket). So that’s why I was asking for a non-theoretical use case. When you say “HTML and whatever,” what is “whatever”?

The exact use-case I’m thinking about right now is writing my CV in Pollen and converting it to PDF (through LaTeX) and some text format (Markdown, I guess).

I’d like to point out that currently Pollen’s command line interface is very… uhm… unconvential. Probably it’s a result of it being, let’s say, layman-oriented. I’d expect something like pandoc:

raco pollen file.pm -o file.html  # Uses template.html
raco pollen file.pm -o file.foo  # Uses template.foo
raco pollen file.pm -o file.bar -t html  # Uses template.html

I’m not sure how to integrate this into the project server (probably because, to be honest, I don’t find it useful as I’m not targeting html). Here is an idea. When I request file.html the project server finds a file that is likely to be the source for this one by appending extensions, but it could just replace extensions instead, i.e. check file.pm, file.pmd, etc. If I request file.md it will also check the same source files. This can break things if someone uses the same basename for different files. For example, index.html that links index.css in the same folder, but, really, just put CSS into the css subdirectory.

The exact use-case I’m thinking about right now is writing my CV in Pollen and converting it to PDF (through LaTeX) _and_ some text format (Markdown, I guess). I’d like to point out that currently Pollen’s command line interface is very… uhm… unconvential. Probably it’s a result of it being, let’s say, layman-oriented. I’d expect something like [pandoc](http://pandoc.org/): ``` Shell raco pollen file.pm -o file.html # Uses template.html raco pollen file.pm -o file.foo # Uses template.foo raco pollen file.pm -o file.bar -t html # Uses template.html ``` I’m not sure how to integrate this into the project server (probably because, to be honest, I don’t find it useful as I’m not targeting html). Here is an idea. When I request `file.html` the project server finds a file that is likely to be the source for this one by appending extensions, but it could just replace extensions instead, i.e. check `file.pm`, `file.pmd`, etc. If I request `file.md` it will also check the same source files. This can break things if someone uses the same basename for different files. For example, `index.html` that links `index.css` in the same folder, but, really, just put CSS into the `css` subdirectory.

I think the way forward looks like this:

[1] Add support for a generic output extension, e.g., file.any.pm, indicating that the source file can have more than one target.

[2] ~~Add a way of specifying which targets a source file actually can support (though perhaps that would end up being unnecessary housekeeping)~~ Superfluous.

[3] Extend the current template-specifying mechanism (e.g., template key in metas can potentially have multiple templates corresponding to different output formats, but absent that, Pollen will just look for a template.ext file).

[4] Add a way of selecting different functions from pollen.rkt. I think this one is non-negotiable, as tag functions might naturally want to have different meanings for different render formats. Possibilities:

[4.1] ~~Multiple pollen.rkt files corresponding to different formats.~~ Terrible. Well, fine if someone wants to set it up that way, but that shouldn’t be a Pollen-level requirement.

[4.2] A new (current-render-target) parameter. This can be used to branch behavior within a function. Straightforward, but it also makes functions more complex / less concise. (OTOH, it would have the benefit of being generally useful across the Pollen system.)

[4.3] Submodules within pollen.rkt, named by target extension. I was leaning toward this one because it enforces a separation at a structural level, but still allows you to push things together if you want. But on reflection, I don’t think it composes well with the expectation that pollen.rkt provides functions that serve all source files, all the time. Using a (current-render-target) parameter, OTOH, would allow certain functions to branch their behavior and others not (for instance, it’s easy to imagine functions like (insert-author-name) that should always work the same way). And, even supposing you had a large set of functions that needed to branch, it would be easy to put the format-specific definitions in separate rkt files, and then put the branching versions in pollen.rkt (which would simply dispatch to the format-specific ones — a little extra housekeeping, but easily automated). I might sound hand-wavy here but I can foresee a simple way to do this with Racket’s existing require logic.

[5] And to get back to your example, the command line looks like this, with a -t switch for render target and -o switch to change the output name (this would be simple to implement with a (current-render-target) parameter):

raco pollen render file.any.pm -t html  # Uses template.html, outputs file.html
raco pollen render file.any.pm -t foo  # Uses template.foo, outputs file.foo
raco pollen render file.any.pm -o file.bar -t html  # Uses template.html, outputs file.bar

[6] The project server isn’t totally irrelevant: even if you’re not using HTML, it’s a quick way of checking the output of a source file rendered with a certain template. I need to think about how the display would change for a file.any.pm source file — ideally you’d be able to click through to see a preview of the output file, just as you can now for HTML.

I think the answer is that if a file has the generic extension (e.g., file.any.pm), then the project server looks for any template.ext files in the directory, for instance template.html and template.tex. Then it creates links in the dashboard to render file.any.pm as file.html or file.tex. This corresponds to the behavior that you would get with raco pollen render. These template files would just be used to figure out what output formats are supported — if file.any.pm had a template meta, that would still override the default template.ext.

Arguably it would be more precise for the project server to look at this value while making the dashboard, but it would require loading file.any.pm, which would be far too slow. The dashboard relies on being able to figure things out from file names.

I think the way forward looks like this: [1] Add support for a generic output extension, e.g., `file.any.pm`, indicating that the source file can have more than one target. [2] ~~Add a way of specifying which targets a source file actually can support (though perhaps that would end up being unnecessary housekeeping)~~ Superfluous. [3] Extend the current template-specifying mechanism (e.g., `template` key in `metas` can potentially have multiple templates corresponding to different output formats, but absent that, Pollen will just look for a `template.ext` file). [4] Add a way of selecting different functions from `pollen.rkt`. I think this one is non-negotiable, as tag functions might naturally want to have different meanings for different render formats. Possibilities: [4.1] ~~Multiple `pollen.rkt` files corresponding to different formats.~~ Terrible. Well, fine if someone **wants** to set it up that way, but that shouldn’t be a Pollen-level requirement. [4.2] A new `(current-render-target)` parameter. This can be used to branch behavior within a function. Straightforward, but it also makes functions more complex / less concise. (OTOH, it would have the benefit of being generally useful across the Pollen system.) [4.3] Submodules within `pollen.rkt`, named by target extension. I was leaning toward this one because it enforces a separation at a structural level, but still allows you to push things together if you want. But on reflection, I don’t think it composes well with the expectation that `pollen.rkt` provides functions that serve all source files, all the time. Using a `(current-render-target)` parameter, OTOH, would allow certain functions to branch their behavior and others not (for instance, it’s easy to imagine functions like `(insert-author-name)` that should always work the same way). And, even supposing you had a large set of functions that needed to branch, it would be easy to put the format-specific definitions in separate `rkt` files, and then put the branching versions in `pollen.rkt` (which would simply dispatch to the format-specific ones — a little extra housekeeping, but easily automated). I might sound hand-wavy here but I can foresee a simple way to do this with Racket’s existing `require` logic. [5] And to get back to your example, the command line looks like this, with a `-t` switch for render target and `-o` switch to change the output name (this would be simple to implement with a `(current-render-target)` parameter): ``` Shell raco pollen render file.any.pm -t html # Uses template.html, outputs file.html raco pollen render file.any.pm -t foo # Uses template.foo, outputs file.foo raco pollen render file.any.pm -o file.bar -t html # Uses template.html, outputs file.bar ``` [6] The project server isn’t totally irrelevant: even if you’re not using HTML, it’s a quick way of checking the output of a source file rendered with a certain template. I need to think about how the display would change for a `file.any.pm` source file — ideally you’d be able to click through to see a preview of the output file, just as you can now for HTML. I think the answer is that if a file has the generic extension (e.g., `file.any.pm`), then the project server looks for any `template.ext` files in the directory, for instance `template.html` and `template.tex`. Then it creates links in the dashboard to render `file.any.pm` as `file.html` or `file.tex`. This corresponds to the behavior that you would get with `raco pollen render`. These template files would just be used to figure out what output formats are supported — if `file.any.pm` had a `template` meta, that would still override the default `template.ext`. Arguably it would be more precise for the project server to look at this value while making the dashboard, but it would require loading `file.any.pm`, which would be far too slow. The dashboard relies on being able to figure things out from file names.

Here’s the tutorial, showing how to generate HTML, plain text, LaTeX, and PDF from a single Pollen markup source file.

[Here’s the tutorial](http://pkg-build.racket-lang.org/doc/pollen/fourth-tutorial.html), showing how to generate HTML, plain text, LaTeX, and PDF from a single Pollen markup source file.

Labels Milestones

Add support for multiple rendering targets #49