A couple of settables for renderers
Openopened 2 years ago by otherjoel · 21 comments
Reference in New Issue
There is no content yet.
Delete Branch '%!s(<nil>)'
Deleting a branch is permanent. It CANNOT be undone. Continue?
@mbutterick — this is another thought relating to your offer in https://github.com/mbutterick/pollen-users/issues/86#issuecomment-786851964. The thing I’m making is an alternate scheme for making templates and applying them to Pollen sources. As things stand right now, this means either duplicating the functionality of
raco pollen renderand Pollen’s project web server, or giving those things up.
It would be really nice (not crucial) to have a way to provide Pollen with:
I think this would be enough to allow my (or any) alternative template scheme to be used in
raco pollen renderand the web server.
Supposing this were done using overrideable
pollen/setupvalues, here is an example
render-markup-or-markdown-sourcefunctions would not change. Instead, they would become the effective* defaults for these settable values.
(* — If I were writing a PR, I'd probably have these two settables default to
#f, and have
pollen/rendercheck them, rather than trying to pull the default functions into
On second thought it would probably be better for this to be: a function to use internally in place of
Though I agree the
setupmodule is the most obvious place for this, I’m also leery of making these rendering functions dependent on resolution through
setup(because of the performance cost — in general I’ve been trying to keep things out of
More broadly, how is
beeswaxideally invoked? Per source file, per template, per directory, per project?
Can you clarify “invoked”? Do you mean, “how would the user ideally cause documents to be rendered using
beeswaxtemplates”? Because what I’m proposing here is that (after some additional configuration step) they will do that by using
raco pollen render(and the Pollen project web server), in the same way they do now: sometimes per-file, sometimes per batch of files (directory or pagetree). All
beeswax(or another system) would supply is the
#langfor the templates themselves and the two functions described above: one for finding a template and another for applying it to a particular source file (example of applying a beeswax template)
The alternative I envision would be for
beeswaxto be invoked via its own
raco beeswax renderthat behaves in the same way as and does mostly the same things as
raco pollen renderdoes now. It would be kind of redundant but doable. I don’t think I am realistically going to duplicate the local project server functionality in any case though. I guess the workaround in that case would be to develop using normal Pollen templates and just add
#lang beeswax/templateto the top when you don’t need the Pollen web server any more.
I suppose I’m just trying to conceptualize what the narrowest / most general API between
pollencould be. For instance, is it fair to say that you’re rendering against a function (generated by a
beeswaxsource file) rather than a snippet of plain text (generated by a
Yes, that is fair to say. And to be clear, in this framework, that function wouldn't have to be generated by a
beeswaxsource file; it could be provided by any Racket module, as long as it is called
renderand looks like
(any/c hash? pagenode? . -> . bytes?). (And indeed I would think
beeswaxtemplate files would have a
OK, so maybe the way for us to think about it is opening Pollen rendering to some function that returns
bytes?(by whatever method) and then
beeswaxis a client.
BTW why do you need access to
I’m thinking about what method Pollen could possibly use to know which
renderfunction to use (each template provides its own render function). It already has a heuristic for marrying up source files with templates, but that heuristic (specifically step 2) isn’t going to find templates that don’t follow those filename conventions (for example, templates with a
.rktfile extension). Having a mechanism for substituting some other function for
get-default-templateseems like a relatively clean way of replacing just that second step in the template search process. Source files could still specify a template in the metas, but for those that don’t do so, there would then be some other default (but project-specific) way of finding the right template.
Essentially, Pollen could accommodate pretty much any auxiliary templating scheme, without caring at all about the details, if it can be given two things: 1) how to find the right template for a given source file and output extension, and 2) how to combine a given template with a given source file. A hook for replacing
get-default-templatewas my idea for the first, and the hook for replacing
render-markup-or-markdown-sourcewas my idea for the second. It seemed like a smaller ask than proposing that Pollen make more-complicated internal changes.
I’m starting to think we’d be better off creating a separate hook within Pollen for this kind of thing, rather than shoehorning it into the current idea of templates. I foresee that there will be escalating difficulties if two unrelated things are allowed to be called templates, and if two essentially unrelated rendering routines are forced to coexist.
For instance, maybe instead of writing
We would create an idea of a
And then when Pollen sees the
renderermeta, it can go down the new rendering path, getting a
renderfunction out of that source file and applying it.
What do you think?
For instance, one problem with muddling the two ideas: if
render-markup-or-markdown-sourceis given a certain path as a template file, it won’t know whether it a) is meant to be used as a good old string-based template, or b) it contains a
renderfunction. As you say, the distinction won’t necessarily be apparent from the file suffix. So I guess it would have to first dynamically evaluate the file (to see if it exports a
renderfunction) and if that fails, treat it as a string-based template. That is going to impose awful costs on a project that just uses string-based templates. Which is why I think the fork has to happen further upstream.
Following on that idea,
renderercould be used with
define-metaas illustrated above (to set the renderer per-source). We could also make it a
setupvalue with a default value of
#falseor a path string. If it’s
#false, then you get the current behavior of string-based templates. Otherwise, the value of the
rendereris used for all the source files controlled by that
Yes, I concur about this! (Maybe I have already created some of this confusion here.) If this discussion does introduce anything new to Pollen itself, that thing really should be called a
rendererand not “templates”. But also, I would not consider an individual
#lang beeswax/templatefile (or the the
renderfunction such a file provides) to be, itself, a “renderer”! More on this below…
To be clear, I was not proposing that
render-markup-or-markdown-sourcebe changed in the slightest. Rather, I’m suggesting that I be able to say to Pollen (via a
setupvalue or some other way) “here’s a function that takes exactly the same arguments as
render-markup-or-markdown-source, and returns the same kind of result; so, for this project, please use my function in place of that one.”. Note: this function I propose to supply is not the
renderfunction directly provided by the
#lang beeswax/templatefile (which takes
metasand an output path), but a separate function that looks like this:
The above is what I would call a
renderer: not an individual template-thing, but the thing that finds the right template-thing for a source file and applies it. (And Pollen need not know anything about what goes on inside that function. It could simply be
(lambda (x [y #f] [z #f]) #"same thing everywhere")as far as Pollen cares.)
Yes! This is the essential idea behind what I first proposed: a
setupvalue with a default value of
#falsethat can be used to hook in a different renderer on a per-project basis. I proposed this setup value would contain an actual function; I suppose it could just as well be a path string to a module that provides the function, but I still like the idea of using actual functions in there.
OK, realization is dawning. I am apparently converging on the right spot 😉
I was asking how you wanted to invoke these renderers because if you want to possibly do so per-file (say, via
define-meta) then the value of the meta needs to be a string or other serializable value (which means not a function). (Though in a
setupmodule, it could also be a function.)
Also, what data other than the
metaswould you need to choose a template for a given source file? (Recalling that
here-path.) I ask because it seems like Pollen only needs to cooperate with one outside function (that is, taking your example,
render-with-templatecan become a single function like
Yes, let me paint the picture :
beeswaxin my Pollen project
template.html.pand rename it to
#lang beeswax/templateat the top. There, now my template is a beeswax template.
template.html.rktfile at this point.
raco pollen render source.html.pm— or I preview the source file in the Pollen project server
metasof the source file. If it finds a value for
'templatethen it would use whatever file that points to. (Or it could look for some other key like
'beeswax-template.) If it finds nothing there, it could search the filesystem for the
.rktfile to use as the template.
.rktfile to use as its “template” for this particular source file, it
renderfunction from that file, and calls it, giving it
metasand the output filename.
here-pathis in the metas, but that is the source filename path. I need the output filename to use as the value for the magic variable
herein the template, which always points to the output filename.
I just pushed the simplest possible implementation of this:
establishes a setup value called
If this value is not
#false, uses it in place of the usual
render. You suggested using it in place of
render-markup-or-markdown-source. It seems to me that an external renderer would want to be able to override any render operation. Though maybe there should be a mechanism for the external renderer to signal “nah Pollen, you can just handle this one normally”, e.g., by raising a certain exception.
external-renderermust be a function that accepts three values: a source path, a template path (in the Pollen sense of that word), and an output path. You suggested passing
metas. But that incurs cost, and I can imagine external renderers that perform tasks that don’t need
metas. So it feels like this should be an opt-in. If an external renderer (like yours) wants
metas, you can pull them out of the source file in some convenient way (say with
Nothing else changes, so AFAICT the
racocommands, project server, etc. all work as usual.
I have not yet added support for a
define-meta. But let’s sort that out in the next step. For now, I’m interested to hear whether this much is useful.
For instance, here’s a toy
metasand just writes out the three paths it receives as input:
Thank you! This is very promising.
This should work, although it’s a bit suboptimal for Pollen to spend cycles hunting for a
template.html.p(or whatever) that is not going to be used; ideally the external renderer would be the one to find a template (if it needs one). Maybe some other external renderer would make use of it though.
Not true! I suggested this:
Beeswax itself does indeed grab the
metaswhen it needs them; all it wants from Pollen is the two things above. And I can get those things from the three values you are passing me 👍
You shouldn’t have to do anything there. Again, the external renderer can check the metas if it wants to.
I’m persuaded by what you said above that an external renderer would be a “function that takes exactly the same arguments“ as the internal renderer. Beyond that, the template path is used in keys for some of the internal caches. So in that way, the template lookup pays for itself.
Apologies. You’re correct; I misread.
But like I say, this is the simplest version — I consider that a virtue because it keeps the interaction with the external renderer simple — see how far you get & we can improve based on your experience.
Preliminary results, it is working very well! Rendering via
raco pollen renderis working (including parallel rendering) and is much faster than when using string-based templates. Pollen project server works seamlessly also.
Here’s the external renderer I’m been testing with. For now it just has a single
beeswaxtemplate file (
"bw-template.html.rkt") hardcoded in:
One limitation of the current approach is that things go sideways if the external renderer tries to
requireany function from Pollen, because of the loading loop that happens with
pollen/setup. So I now see the virtue of your original notion of supplying the renderer as a module path rather than directly as a function. I now think doing it that way would be a big improvement. For now, I can get around this by using
dynamic-requireto get the specific functions I need.
I like that I’m also able to use Pollen’s own
renderfunction as a fallback. This is another advantage of your implementation over what I originally proposed.
When I have time I want to do some more testing about how the Pollen cache behaves with an external renderer in the presence or absence of the normal Pollen template files (like
Hmm — maybe it will make more sense, ultimately, to make
external-renderera parameter rather than a
setupvalue. Though that would probably mean you need to set it imperatively in a
pollen.rkt, which is venturing close to the icky world of global variables.
Are you saying that would be preferable to passing
external-rendereras a module-path (suitable for
It‘s a little fuzzy to me how setting a Pollen-define parameter in
pollen.rkt(outside of the
setupsubmodule) would have any effect during this stage of the rendering phase.
No surprises here of course. But being able to supply a template-finder in addition to
external-rendererwould be nice, so that touching things the external renderer uses for rendering a particular file will naturally invalidate the cache.