Provide get-doc/get-meta from template.rkt #57

Following up from issue 55 I'm using a similar tactic to export categories under which a blog post will come.

What I'm now attempting to achieve is building a pagetree programmatically by reading all *.pm files in a directory posts/and then rendering it. As part of this process I want to generate category pages (i.e. generate programming.html that lists all posts tagged with programming. As I am storing the categories in the doc structure I need to generate the doc txexpr programmatically, pull out the categories and collate them across all posts.

In order to do this I've tracked down the get-doc function which I believe will work, unfortunately it isn't currently exported by template.rkt. Unless I'm missing something (another way to do this etc.) it would be useful for template.rkt to export get-doc and, indeed, get-metas. I can imagine there would be other cases where these functions would be required.

The code will look something like the following:

#lang racket

(require pollen/template) ;; Need get-doc from here

(define (list-pms directory)
  (filter (λ (file) 
         (regexp-match #rx"\\.pm" (path->string file))) 
      (directory-list directory)))

(define (pm->html files)
  (map (λ (file)
      (string->symbol (string-append directory
                     (string-trim (path->string file) ".pm" #:left? #f #:repeat? #f))))
       files))

(define (generate-categories files)
  ...
  (map (λ (file)
         (select-from-doc 'categories (get-doc files))))
  ...)

(define (make-pagetree)
  (let ([post-files (list-pms "posts/")])
    `(pagetree-root
      (index.html ,@(pm->html post-files))
      ,@(generate-categories post-files))

Following up from [issue 55](https://github.com/mbutterick/pollen/issues/55) I'm using a similar tactic to export categories under which a blog post will come. What I'm now attempting to achieve is building a pagetree programmatically by reading all `*.pm` files in a directory `posts/`and then rendering it. As part of this process I want to generate category pages (i.e. generate `programming.html` that lists all posts tagged with `programming`. As I am storing the categories in the `doc` structure I need to generate the `doc` txexpr programmatically, pull out the categories and collate them across all posts. In order to do this I've tracked down the `get-doc` function which I believe will work, unfortunately it isn't currently exported by `template.rkt`. Unless I'm missing something (another way to do this etc.) it would be useful for `template.rkt` to export `get-doc` and, indeed, `get-metas`. I can imagine there would be other cases where these functions would be required. The code will look something like the following: ``` #lang racket (require pollen/template) ;; Need get-doc from here (define (list-pms directory) (filter (λ (file) (regexp-match #rx"\\.pm" (path->string file))) (directory-list directory))) (define (pm->html files) (map (λ (file) (string->symbol (string-append directory (string-trim (path->string file) ".pm" #:left? #f #:repeat? #f)))) files)) (define (generate-categories files) ... (map (λ (file) (select-from-doc 'categories (get-doc files)))) ...) (define (make-pagetree) (let ([post-files (list-pms "posts/")]) `(pagetree-root (index.html ,@(pm->html post-files)) ,@(generate-categories post-files)) ```

What I'm now attempting to achieve is building a pagetree programmatically by reading all *.pm files in a directory posts/and then rendering it.

If you need to do this, it's not wrong. But don't forget that if Pollen doesn't find an index.ptree, it will do exactly what you describe — generate a pagetree from all renderable files. IOW, if directory posts contains one.html.pm, two.html.pm ..., and you do raco pollen render posts, then all these pm files will be rendered. (See also “Using the automatic pagetree”.)

As I am storing the categories in the doc structure I need to generate the doc txexpr programmatically, pull out the categories and collate them across all posts.

I will assume you have your reasons for storing them in doc, but IMO categories seem like an excellent thing to store in metas, because they are in fact metadata.

Unless I'm missing something (another way to do this etc.) it would be useful for template.rkt to export get-doc and, indeed, get-metas.

get-doc and get-metas aren’t exposed in the public interface because they’re really just helper functions for select-from-doc and select-from-metas (which are part of the interface). How are they falling short? (BTW in case of emergency, you can break the glass with require/expose.)

> What I'm now attempting to achieve is building a pagetree programmatically by reading all `*.pm` files in a directory `posts/`and then rendering it. If you need to do this, it's not wrong. But don't forget that if Pollen doesn't find an `index.ptree`, it will do exactly what you describe — generate a pagetree from all renderable files. IOW, if directory `posts` contains `one.html.pm, two.html.pm ...`, and you do `raco pollen render posts`, then all these `pm` files will be rendered. (See also “[Using the automatic pagetree](http://pkg-build.racket-lang.org/doc/pollen/second-tutorial.html#%28part._tutorial-2._.Using_the_automatic_pagetree%29)”.) > As I am storing the categories in the `doc` structure I need to generate the `doc` txexpr programmatically, pull out the categories and collate them across all posts. I will assume you have your reasons for storing them in `doc`, but IMO categories seem like an excellent thing to store in `metas`, because they are in fact metadata. > Unless I'm missing something (another way to do this etc.) it would be useful for `template.rkt` to export `get-doc` and, indeed, `get-metas`. `get-doc` and `get-metas` aren’t exposed in the public interface because they’re really just helper functions for `select-from-doc` and `select-from-metas` (which are part of the interface). How are they falling short? (BTW in case of emergency, you can break the glass with [`require/expose`](http://pkg-build.racket-lang.org/doc/rackunit/api.html?q=rackunit#%28form._%28%28lib._rackunit%2Fmain..rkt%29._require%2Fexpose%29%29).)

As part of this process I want to generate category pages (i.e. generate programming.html that lists all posts tagged with programming.

BTW it sounds like you’re conceptualizing the generation of your category pages as a side effect of rendering the posts directory. You might find it fruitful to flip it around: have separate source files in your project that represent the categories (e.g., programming.html.pm, pizza.html.pm, disco.html.pm ...) and put the page-gathering logic in those. Then you can develop & preview them independent of rendering the other pages. (That assumes, however, that you have a fixed list of categories.)

> As part of this process I want to generate category pages (i.e. generate `programming.html` that lists all posts tagged with `programming`. BTW it sounds like you’re conceptualizing the generation of your category pages as a side effect of rendering the `posts` directory. You might find it fruitful to flip it around: have separate source files in your project that represent the categories (e.g., `programming.html.pm, pizza.html.pm, disco.html.pm ...`) and put the page-gathering logic in those. Then you can develop & preview them independent of rendering the other pages. (That assumes, however, that you have a fixed list of categories.)

What I'm now attempting to achieve is building a pagetree programmatically by reading all *.pm files in a directory posts/and then rendering it.

If you need to do this, it's not wrong. But don't forget that if Pollen doesn't find an index.ptree, it will do exactly what you describe — generate a pagetree from all renderable files. IOW, if directory posts contains one.html.pm, two.html.pm ..., and you do raco pollen render posts, then all these pm files will be rendered. (See also “Using the automatic pagetree”.)

The reason I'm trying to do it like this, which I didn't mention above, is that I want to order posts by date. I'd need some logic in pm->html to read the date I'm storing in meta (again I think I'd need get-doc or get-meta to achieve this). Obviously I could manually generate the pagetree but ideally this should be automated.

I will assume you have your reasons for storing them in doc, but IMO categories seem like an excellent thing to store in metas, because they are in fact metadata.

I agree this is metadata and should probably be stored in metas. My reservation is that meta can only hold strings (right?). Seems to me that it would be nice to store arbitrary data, for example store categories as a list of strings under a single key categories. Obviously I could store a single space-separated string of categories and then split on space.

get-doc and get-metas aren’t exposed in the public interface because they’re really just helper functions for select-from-doc and select-from-metas (which are part of the interface). How are they falling short? (BTW in case of emergency, you can break the glass with require/expose.)

Again, I want to order the posts by date so I need data from all the posts at a single time. The select-froms are great inside of the template that renders a particular post but I don't see how I can use them without get-doc/get-meta when needing to collate information across more than one markup file. I didn't know about require/expose (I've just started learning Racket), cool!

BTW it sounds like you’re conceptualizing the generation of your category pages as a side effect of rendering the posts directory. You might find it fruitful to flip it around: have separate source files in your project that represent the categories (e.g., programming.html.pm, pizza.html.pm, disco.html.pm ...) and put the page-gathering logic in those. Then you can develop & preview them independent of rendering the other pages. (That assumes, however, that you have a fixed list of categories.)

My hope would be to not have such a fixed list and it would generate category pages for whatever I throw into the posts.

You've stated elsewhere that you're not looking to personally push Pollen in the direction of blogging, and maybe I'm abusing rather than using it, but I think it's a perfect application. I've messed around with a bunch of other static site generators and they never quite do everything I want them to but Pollen's programmability makes it really simple to basically do whatever I want it to. It's awesome!

> > What I'm now attempting to achieve is building a pagetree programmatically by reading all *.pm files in a directory posts/and then rendering it. > > If you need to do this, it's not wrong. But don't forget that if Pollen doesn't find an index.ptree, it will do exactly what you describe — generate a pagetree from all renderable files. IOW, if directory posts contains one.html.pm, two.html.pm ..., and you do raco pollen render posts, then all these pm files will be rendered. (See also “Using the automatic pagetree”.) The reason I'm trying to do it like this, which I didn't mention above, is that I want to order posts by date. I'd need some logic in `pm->html` to read the date I'm storing in `meta` (again I think I'd need `get-doc` or `get-meta` to achieve this). Obviously I could manually generate the pagetree but ideally this should be automated. > I will assume you have your reasons for storing them in doc, but IMO categories seem like an excellent thing to store in metas, because they are in fact metadata. I agree this is metadata and should probably be stored in metas. My reservation is that meta can only hold strings (right?). Seems to me that it would be nice to store arbitrary data, for example store categories as a list of strings under a single key `categories`. Obviously I could store a single space-separated string of categories and then split on space. > get-doc and get-metas aren’t exposed in the public interface because they’re really just helper functions for select-from-doc and select-from-metas (which are part of the interface). How are they falling short? (BTW in case of emergency, you can break the glass with require/expose.) Again, I want to order the posts by date so I need data from all the posts at a single time. The `select-from`s are great inside of the template that renders a particular post but I don't see how I can use them without `get-doc`/`get-meta` when needing to collate information across more than one markup file. I didn't know about `require/expose` (I've just started learning Racket), cool! > BTW it sounds like you’re conceptualizing the generation of your category pages as a side effect of rendering the posts directory. You might find it fruitful to flip it around: have separate source files in your project that represent the categories (e.g., programming.html.pm, pizza.html.pm, disco.html.pm ...) and put the page-gathering logic in those. Then you can develop & preview them independent of rendering the other pages. (That assumes, however, that you have a fixed list of categories.) My hope would be to not have such a fixed list and it would generate category pages for whatever I throw into the posts. You've stated [elsewhere](https://github.com/mbutterick/pollen/issues/43) that you're not looking to personally push Pollen in the direction of blogging, and maybe I'm abusing rather than using it, but I think it's a perfect application. I've messed around with a bunch of other static site generators and they never quite do everything I want them to but Pollen's programmability makes it really simple to basically do whatever I want it to. It's awesome!

Seems to me that it would be nice to store arbitrary data, for example store categories as a list of strings under a single key categories. Obviously I could store a single space-separated string of categories and then split on space.

I’ve considered the issue before. I agree that because metas ultimately becomes a hashtable, it would be nice to be able to write any type of value. The problem is that it would mean losing compatibility with X-expressions, which would cost a lot of simplicity. To justify that loss, I’d need to come up with a compelling example of a datatype that wasn’t tolerably encoded as a string. I've never been able to. (Maybe you can.) For instance, numbers can be converted with string->number. And as you say, a list of categories can be string-split.

The select-froms are great inside of the template that renders a particular post but I don't see how I can use them without get-doc/get-meta when needing to collate information across more than one markup file.

I would intuitively think of doing something like

(map (λ(sf) (select-from-metas 'date-key sf)) list-of-pm-sourcefiles)

Would that not work for you?

My hope would be to not have such a fixed list and it would generate category pages for whatever I throw into the posts.

Fair enough. Carry on.

you're not looking to personally push Pollen in the direction of blogging, and maybe I'm abusing rather than using it, but I think it's a perfect application

I appreciate the compliment. I endorse Pollen being used as a blogging tool (though also consider frog). I’m sure that bloggers will have good suggestions for improving it. But to call something a “blogging system” sets expectations of certain features, which have been implemented by others, before, better. As you say, Pollen’s comparative advantage is its programmability, which I’d like to use to investigate the territory that lies beyond blogs.

> Seems to me that it would be nice to store arbitrary data, for example store categories as a list of strings under a single key `categories`. Obviously I could store a single space-separated string of categories and then split on space. I’ve considered the issue before. I agree that because `metas` ultimately becomes a hashtable, it would be nice to be able to write any type of value. The problem is that it would mean losing compatibility with X-expressions, which would cost a lot of simplicity. To justify that loss, I’d need to come up with a compelling example of a datatype that wasn’t tolerably encoded as a string. I've never been able to. (Maybe you can.) For instance, numbers can be converted with `string->number`. And as you say, a list of categories can be `string-split`. > The `select-from`s are great inside of the template that renders a particular post but I don't see how I can use them without `get-doc`/`get-meta` when needing to collate information across more than one markup file. I would intuitively think of doing something like ``` racket (map (λ(sf) (select-from-metas 'date-key sf)) list-of-pm-sourcefiles) ``` Would that not work for you? > My hope would be to not have such a fixed list and it would generate category pages for whatever I throw into the posts. Fair enough. Carry on. > you're not looking to personally push Pollen in the direction of blogging, and maybe I'm abusing rather than using it, but I think it's a perfect application I appreciate the compliment. I endorse Pollen being used as a blogging tool (though also consider [`frog`](https://github.com/greghendershott/frog)). I’m sure that bloggers will have good suggestions for improving it. But to call something a “blogging system” sets expectations of certain features, which have been implemented by others, before, better. As you say, Pollen’s comparative advantage is its programmability, which I’d like to use to investigate the territory that lies beyond blogs.

I’ve considered the issue before. I agree that because metas ultimately becomes a hashtable, it would be nice to be able to write any type of value. The problem is that it would mean losing compatibility with X-expressions, which would cost a lot of simplicity.

Can X-expressions be stored in meta?

To justify that loss, I’d need to come up with a compelling example of a datatype that wasn’t tolerably encoded as a string. I've never been able to. (Maybe you can.) For instance, numbers can be converted with string->number. And as you say, a list of categories can be string-split.

Nothing I can think of off the top of my head!

The select-froms are great inside of the template that renders a particular post but I don't see how I can use them without get-doc/get-meta when needing to collate information across more than one markup file.

I would intuitively think of doing something like

(map (λ(sf) (select-from-metas 'date-key sf)) list-of-pm-sourcefiles)
Would that not work for you?

<high_embarrassment_factor_mode>
Woops! The selects work on paths; I was under the impression that they operated only on txexprs. No need for get-doc and get-metas to be exposed then.
</high_embarrassment_factor_mode>

> I’ve considered the issue before. I agree that because metas ultimately becomes a hashtable, it would be nice to be able to write any type of value. The problem is that it would mean losing compatibility with X-expressions, which would cost a lot of simplicity. Can X-expressions be stored in meta? > To justify that loss, I’d need to come up with a compelling example of a datatype that wasn’t tolerably encoded as a string. I've never been able to. (Maybe you can.) For instance, numbers can be converted with string->number. And as you say, a list of categories can be string-split. Nothing I can think of off the top of my head! > > The select-froms are great inside of the template that renders a particular post but I don't see how I can use them without get-doc/get-meta when needing to collate information across more than one markup file. > > I would intuitively think of doing something like > > `(map (λ(sf) (select-from-metas 'date-key sf)) list-of-pm-sourcefiles)` > Would that not work for you? <high_embarrassment_factor_mode> Woops! The `select`s work on paths; I was under the impression that they operated only on txexprs. No need for `get-doc` and `get-metas` to be exposed then. </high_embarrassment_factor_mode>

Can X-expressions be stored in meta?

That’s a good idea. I’ve pushed a fix to permit that. You can still write your key-value pair as an attribute:

◊meta['key: "value"]

But you can also write it as an element, which allows you to use any X-expression as the value:

◊meta{◊key{◊tx{value}}}

You can also store lists of values, like categories:

◊meta{◊categories['brindles 'boxers 'working-dogs]}

The selects work on paths; I was under the impression that they operated only on txexprs.

I’ve pushed a clarification for the docs.

> Can X-expressions be stored in meta? That’s a good idea. I’ve pushed a fix to permit that. You can still write your key-value pair as an attribute: ``` racket ◊meta['key: "value"] ``` But you can also write it as an element, which allows you to use any X-expression as the value: ``` racket ◊meta{◊key{◊tx{value}}} ``` You can also store lists of values, like categories: ``` racket ◊meta{◊categories['brindles 'boxers 'working-dogs]} ``` > The `select`s work on paths; I was under the impression that they operated only on txexprs. I’ve pushed a clarification for the docs.

Labels Milestones

Provide get-doc/get-meta from template.rkt #57