faster renders (I think) #49

I’ve been burned before by overconfidence (see: parallel processing). But today I had (what I hope will turn out to be) a genuinely good idea: to create a fast path for raco pollen setup and raco pollen render that skips certain expensive luxuries that one really needs only during interactive development sessions with the project server (which will be as sluggish as always). For instance, rebuilding Beautiful Racket is now 3x faster. Try it and let me know if it improves your life (or the opposite).

I’ve been burned before by overconfidence (see: parallel processing). But today I had (what I hope will turn out to be) a genuinely good idea: to create a fast path for `raco pollen setup` and `raco pollen render` that skips certain expensive luxuries that one really needs only during interactive development sessions with the project server (which will be as sluggish as always). For instance, rebuilding _Beautiful Racket_ is now 3x faster. Try it and let me know if it improves your life (or the opposite).

I spent some time testing this last night.

I didn't happen to see any improvement when doing a full from-scratch rebuild on either 7.6 BC or 7.7 BC. In the case where I do a full build and then did touch pollen.rkt and I did see an improvement of between 6–9 seconds. (By the way, my raco make pollen.rkt step still pays pretty hefty performance dividends)

But. Each of my pages is doing one or more INSERTs on a SQLite file during the template rendering phase. I now wonder if for that reason my projects are not a fair comparison to use for this improvement? I want to delve into what happens behind the scenes when multiple Racket threads attempt to do this. I always assumed that SQLite's file locking would have a minimal effect, but now I am not so sure.

I think there is no denying a 3x improvement in any one project!

I spent some time testing this last night. I didn't happen to see any improvement when doing a full from-scratch rebuild on either 7.6 BC or 7.7 BC. In the case where I do a full build and then did `touch pollen.rkt` and I did see an improvement of between 6–9 seconds. (By the way, my `raco make pollen.rkt` step still pays pretty hefty performance dividends) **But**. Each of my pages is doing one or more `INSERT`s on a SQLite file during the template rendering phase. I now wonder if for that reason my projects are not a fair comparison to use for this improvement? I want to delve into what happens behind the scenes when multiple Racket threads attempt to do this. I always assumed that SQLite's file locking would have a minimal effect, but now I am not so sure. I think there is no denying a 3x improvement in any one project!

I think the fact that there's more than one more INSERT is due to the reuse of namespace, as I detailed in #43 :(

I think the fact that there's more than one more `INSERT` is due to the reuse of namespace, as I detailed in #43 :(

Yes, I made the change you suggested in #43: reuse a namespace (looking back I see that you also got nearly 3x performance from that change). I realized I could introduce an extra parameter to signal when it was safe to reuse the namespace. Good idea — thanks!

Each of my pages is doing one or more INSERTs on a SQLite file during the template rendering phase

In my admittedly minimal experiments with SQLite in Pollen, I found that the per-transaction costs of a SQL query were very high compared to the average Pollen computation. Unless each INSERT creates state that affects later renders, you might consider sending all the INSERTs to be handled in a separate thread, so they don’t block the main render threads (or better — batch them into one large query at the end). Again, this is the opinion of a SQLite idiot.

> Each of my pages is doing one or more INSERTs on a SQLite file during the template rendering phase In my admittedly minimal experiments with SQLite in Pollen, I found that the per-transaction costs of a SQL query were very high compared to the average Pollen computation. Unless each `INSERT` creates state that affects later renders, you might consider sending all the `INSERT`s to be handled in a separate thread, so they don’t block the main render threads (or better — batch them into one large query at the end). Again, this is the opinion of a SQLite idiot.

I think the fact that there's more than one more INSERT is due to the reuse of namespace

Actually, the multiple INSERTs are something I’m doing on purpose. Each page does at least one, but if certain content is present that will be saved to the database as well, currently in a separate query.

I will do some experimenting with threads and batching for sure. Will start another thread about bespoke caching in general when I have any results to share.

> I think the fact that there's more than one more INSERT is due to the reuse of namespace Actually, the multiple `INSERT`s are something I’m doing on purpose. Each page does at least one, but if certain content is present that will be saved to the database as well, currently in a separate query. I will do some experimenting with threads and batching for sure. Will start another thread about bespoke caching in general when I have any results to share.

Could you please point to a specific commit and/or summarize the changes that allowed for faster renders?

149f2cc389 (diff-87016fbc1ea08a94b64815d70faa9309R83)

https://github.com/mbutterick/pollen/commit/149f2cc389b54af621f6a40dcb206dbac6b6965a#diff-87016fbc1ea08a94b64815d70faa9309R83

Labels Milestones

faster renders (I think) #49