Some (long) thoughts about the long term #101

Beautiful Racket, Pollen, Quad - oh my, where to start ...

As software (and embedded-hardware) developer who has worked with dozens of programming languages over a period of more than half a century, one aspect always stands out for me: How long will idea X last? -- Writing documents and code using a plain-text editor has stood the test of time, and while LOP makes tremendously much sense on so many levels, it also creates a major dependency: will Racket be around ten years from now? will it be usable on what is then commodity hardware? or will it be on life-support as a "legacy" system? -- No one knows obviously, and it's really not LOP's fault as any project with some form of "tooling" has this same issue.

In this context, I find it very hard to come up with a strategy for writing about the projects I work on, either in the form of weblog posts or as various kinds of documentation. Like so many people, I've gone from WordPress, to home-baked, to database-backed, to static HTML generated from Markdown. Recent ones, such as Hugo, MkDocs, Docsify … been there, done that.

But none of them make sense, long term (let's say a decade). If the information is permanent and essentially frozen, then there's no reason to hold on to the tooling that produced the web pages - just generate the HTML one last time and place it wherever. If the information is evolving, such as a discussion forum, then good luck - pick whatever you want, it's always going to be trouble down the road (again: long term). I have several old forums lying around, saved as web-spidered HTML dumps before flipping the switch one last time and leaving an un-maintained (or unmaintainable) discussion forum software behind (BB-Press and Redmine, amongst them).

For prose, the concept of books still makes a lot of sense to me (and that includes the PDF electronic form). Sure, you get a copy, it's static, they're not interactive, but that's also a benefit: they stay around. Unlike websites, i.e. anything with a URL: who says the website will be around in a few years? And that includes https://beautifulracket.com and https://racket-lang.org. Who says the pages will stay in the same place forever?

My thinking at the moment is that my best bet is to 1) stick to plain text for document generation (LOP could be wonderful, but I have no experience with it yet) and 2) to generate both HTML and PDF formats (Quad + Pollen? definitely worth a try).

While I've greatly enjoyed going through many of Edward Tufte's books, I do not place that much emphasis on using the best typography in the world. If information is presented clearly with absolutely minimal clutter in browser windows of varying sizes as well as on the (paper- or PDF-) page, and does so with consistency and room for precise graphic diagrams, I'm happy.

My reason for posting here is that I want to stress the long-term aspects of writing & publishing, because in the end that's really where all the time and energy is spent: you have to pick your tools and formatting conventions up front, and then you spend weeks / months / years ... typing. Each character typed-in will solidify your early choices, in the sense that change becomes increasingly tedious (and therefore unlikely). And when you pick the wrong tools or conventions - these efforts can end up being wasted.

This is not a hypothetical issue - I've written over 1,500 weblog posts in the past 15 years or so, and the results are stuck as "legacy texts" in various formats and systems. Not to mention the many thousands of posts and comments people have collectively placed in a variety of now-extinct forums (fora?). There's not really much joy in that (Marie Kondo would no doubt advise to just delete it all).

Personally, I've almost reached a point where I just can't get myself to write anything down anymore (FWIW, all my work is on OSS projects nowadays, there's no "business side" to it). Because ... what's the point, if it's all going to become unmaintainable a few years from now?

Back to Beautiful Racket, Pollen, and Quad: maybe they're the miracle cure which will help address these longevity issues. Then again, maybe it'll be just another iteration with the same outcome as before. I've not yet committed to investing serious time into BR/P/Q - still just exploring / evaluating. But it's definitely a tempting proposition.

Adopting a new approach for writing is a bit like XKCD's 14 competing standards :)

_Beautiful Racket, Pollen, Quad - oh my, where to start ..._ As software (and embedded-hardware) developer who has worked with dozens of programming languages over a period of more than half a century, one aspect always stands out for me: _How long will idea X last?_ -- Writing documents and code using a plain-text editor has stood the test of time, and while LOP makes tremendously much sense on so many levels, it also creates a major dependency: will Racket be around ten years from now? will it be usable on what is _then_ commodity hardware? or will it be on life-support as a "legacy" system? -- No one knows obviously, and it's really not LOP's fault as _any_ project with some form of "tooling" has this same issue. In this context, I find it very hard to come up with a strategy for writing about the projects I work on, either in the form of weblog posts or as various kinds of documentation. Like so many people, I've gone from WordPress, to home-baked, to database-backed, to static HTML generated from Markdown. Recent ones, such as Hugo, MkDocs, Docsify … been there, done that. But none of them make sense, long term (let's say a decade). If the information is permanent and essentially frozen, then there's no reason to hold on to the tooling that produced the web pages - just generate the HTML one last time and place it wherever. If the information is evolving, such as a discussion forum, then good luck - pick whatever you want, it's _always_ going to be trouble down the road (again: long term). I have several old forums lying around, saved as web-spidered HTML dumps before flipping the switch one last time and leaving an un-maintained (or unmaintainable) discussion forum software behind (BB-Press and Redmine, amongst them). For prose, the concept of books still makes a lot of sense to me (and that includes the PDF electronic form). Sure, you get a copy, it's static, they're not interactive, but that's also a benefit: _they stay around_. Unlike websites, i.e. anything with a URL: who says the website will be around in a few years? And that includes <https://beautifulracket.com> and <https://racket-lang.org>. Who says the pages will stay in the same place forever? My thinking at the moment is that my best bet is to 1) stick to plain text for document generation (LOP could be wonderful, but I have no experience with it yet) and 2) to generate _both_ HTML and PDF formats (Quad + Pollen? definitely worth a try). While I've greatly enjoyed going through many of Edward Tufte's books, I do not place _that_ much emphasis on using the best typography in the world. If information is presented clearly with absolutely minimal clutter in browser windows of varying sizes as well as on the (paper- or PDF-) page, and does so with consistency and room for precise graphic diagrams, I'm happy. My reason for posting here is that I want to stress the _long-term_ aspects of writing & publishing, because in the end that's really where all the time and energy is spent: you have to pick your tools and formatting conventions up front, and then you spend weeks / months / years ... typing. Each character typed-in will _solidify_ your early choices, in the sense that change becomes increasingly tedious (and therefore unlikely). And when you pick the wrong tools or conventions - these efforts can end up being wasted. This is not a hypothetical issue - I've written over 1,500 weblog posts in the past 15 years or so, and the results are _stuck_ as "legacy texts" in various formats and systems. Not to mention the many thousands of posts and comments people have collectively placed in a variety of now-extinct forums (fora?). There's not really much joy in that (Marie Kondo would no doubt advise to just delete it all). Personally, I've almost reached a point where I just can't get myself to write anything down anymore (FWIW, all my work is on OSS projects nowadays, there's no "business side" to it). Because ... _what's the point, if it's all going to become unmaintainable a few years from now?_ Back to Beautiful Racket, Pollen, and Quad: maybe they're the miracle cure which will help address these longevity issues. Then again, maybe it'll be just another iteration with the same outcome as before. I've not yet _committed_ to investing serious time into BR/P/Q - still just exploring / evaluating. But it's definitely a tempting proposition. Adopting a new approach for writing is a bit like XKCD's [14 competing standards](https://xkcd.com/927/) :)

Thinking about all of that was what got me into Pollen in the first place.

These days I find myself thinking of ephemerality as a feature, not a bug. The real artifact isn’t the thing you wrote, it’s your influence on (maybe connections with) the people who read it. (yes, the friends we made along the way, haha.)

Personally, I’m still having a lot of fun with Pollen (and more generally Racket), and I still think it strikes the best balance between advanced functionality and longevity. But I’m no longer imagining that longevity is a priority of mine, or that it’s even achievable.

Thinking about all of that was what [got me into Pollen in the first place](https://thelocalyarn.com/excursus/secretary/posts/web-books.html). These days I find myself thinking of [ephemerality as a feature, not a bug](https://joeldueck.com/wiki?name=Approaches+to+publishing). The real artifact isn’t the thing you wrote, it’s your influence on (maybe connections with) the people who read it. (yes, the friends we made along the way, haha.) Personally, I’m still having a lot of fun with Pollen (and more generally Racket), and I still think it strikes the best balance between advanced functionality and longevity. But I’m no longer imagining that longevity is a priority of mine, or that it’s even achievable.

Touché – what a good point you make, as I now realise. You're right: I've been trying to juggle and keep "everything" (whatever that means) intact over the years, but maybe the journey is indeed what should matter most in the long term, and not the deliverables.

Thanks for the nudge. I can now look into Racket and Pollen more as a way to something new instead of a tool to try and rescue the old. Could be fun.

Touché – what a good point you make, as I now realise. You're right: I've been trying to juggle and keep "everything" (whatever that means) intact over the years, but maybe the journey is indeed what should matter most in the long term, and _not_ the deliverables. Thanks for the nudge. I can now look into Racket and Pollen more as a way to something new instead of a tool to try and rescue the old. Could be fun.

The original impetus for Pollen was not longevity so much as possibility: I didn’t see how I could accomplish certain things with any existing tools. (See also.)

That said, arguably the oldest and most enduring data structure for hierarchical data is the Lisp S-expression. Which is the underlying medium of exchange for everything in Pollen. So I would quibble with the “14 competing standards” critique — Lisp came first.

Longevity in programming is an interesting topic. IMO it doesn’t correlate to funding or popularity. If I had to guess, I think it has something to do with API stability, because that induces more people to rely on the project over time, and makes code more valuable. (Racket’s repo has code going back to 1995. Now that it’s transitioned to Chez Scheme, I assume it runs on code dating back to the mid-1980s.)

Still, as the tech industry matures, APIs seem to become less stable. I suppose it’s because earlier in computing history, an API was a way of ensuring consistent behavior in different places and times. Growing the installed base of devices was in everyone’s interest. Today, APIs are used increasingly as an assertion of power, because the market is already saturated with devices, so companies have to compete by ruining existing devices — which can be done by breaking their APIs.

For instance, I own MIDI-equipped synthesizers from the 1980s that, crazily, are getting more capable because the stability of the API enables new generations of hacking. Meanwhile, I expect my 2016 iPhone SE to be bricked pretty soon by a corporate behemoth who, like some tropical lava god, is angry that I refuse to buy a new one.

OTOH we have TeX, whose dedication to API stability is both its greatest feature and greatest hobble, because after TeX came the most consequential developments in digital publishing — PostScript, OpenType, PDF. I don’t know if these will be around forever. Nothing will, right? But the fact that large organizations, especially governments, have now committed to this stack of technologies for long-term archiving, IMO hardens it to the kind of API f*ckery that might otherwise occur. (Indeed, I think part of the reason Adobe has become largely a digital marketing & surveillance company is that it forfeited control of its publishing APIs.)

With Quad, one of the ideas that’s been important to me is creating an API that’s not Racket-specific. You can feed Quad a text-based S-expression, generated with any tool you like — maybe Racket, maybe not — and get a PDF.

I’ve wondered whether I should write Quad in, say, ANSI C, so its dependencies are minimal and it can be compiled & used basically anywhere. That’s rather outside my skill set, though. I can get more done by developing Quad in Racket. But I do think in terms of having it become more of a reference implementation of a certain set of functions that can be ported to other, more popular languages, in which I am unlikely to ever have any inherent interest.

I do agree with Knuth’s point that art is just the category of things we haven’t yet taught a computer to do. I consider the reproduction of the written word to be the greatest problem that has existed in the technological age. Before computers existed, the most amazing machines in the world were those that were built for typesetting and printing. After computers, some of the most important early programs were for typesetting. (e.g., troff, which Brian Kernighan himself ported to C.) I’m used to having my work shorthanded as “ooh, you make things pretty”. But that’s not at all how I think of it. My goal is to create a program that does what I do, better than I do it. Then it will brick me.

The original impetus for Pollen was not longevity so much as *possibility*: I didn’t see how I could accomplish certain things with any existing tools. ([See also.](https://docs.racket-lang.org/pollen/Backstory.html)) That said, arguably the oldest and most enduring data structure for hierarchical data is the Lisp S-expression. Which is the underlying medium of exchange for everything in Pollen. So I would quibble with the “14 competing standards” critique — Lisp came first. Longevity in programming is an interesting topic. IMO it doesn’t correlate to funding or popularity. If I had to guess, I think it has something to do with API stability, because that induces more people to rely on the project over time, and makes code more valuable. (Racket’s repo has code going [back to 1995](https://github.com/racket/racket/blob/ddf6985020d7dd11d17cd42b1746bd853667fc2b/racket/collects/racket/HISTORY.txt). Now that it’s transitioned to Chez Scheme, I assume it runs on code dating back to the mid-1980s.) Still, as the tech industry matures, APIs seem to become *less* stable. I suppose it’s because earlier in computing history, an API was a way of ensuring consistent behavior in different places and times. Growing the installed base of devices was in everyone’s interest. Today, APIs are used increasingly as an assertion of power, because the market is already saturated with devices, so companies have to compete by ruining existing devices — which can be done by breaking their APIs. For instance, I own MIDI-equipped synthesizers from the 1980s that, crazily, are getting **more** capable because the stability of the API enables [new generations of hacking](http://www.vecoven.com/superjx/superjx.html). Meanwhile, I expect my 2016 iPhone SE to be bricked pretty soon by a corporate behemoth who, like some tropical lava god, is angry that I refuse to buy a new one. OTOH we have TeX, whose dedication to API stability is both its greatest feature and greatest hobble, because after TeX came the most consequential developments in digital publishing — PostScript, OpenType, PDF. I don’t know if these will be around forever. Nothing will, right? But the fact that large organizations, especially governments, have now committed to this stack of technologies for long-term archiving, IMO hardens it to the kind of API f*ckery that might otherwise occur. (Indeed, I think part of the reason Adobe has become largely a digital marketing & surveillance company is that it forfeited control of its publishing APIs.) With Quad, one of the ideas that’s been important to me is creating an API that’s **not** Racket-specific. You can feed Quad a text-based S-expression, generated with any tool you like — maybe Racket, maybe not — and get a PDF. I’ve wondered whether I should write Quad in, say, ANSI C, so its dependencies are minimal and it can be compiled & used basically anywhere. That’s rather outside my skill set, though. I can get more done by developing Quad in Racket. But I do think in terms of having it become more of a reference implementation of a certain set of functions that can be ported to other, more popular languages, in which I am unlikely to ever have any inherent interest. I do agree with Knuth’s point that art is just the category of things we haven’t yet taught a computer to do. I consider the reproduction of the written word to be the greatest problem that has existed in the technological age. Before computers existed, the most amazing machines in the world were those that were built for typesetting and printing. After computers, some of the most important early programs were for typesetting. (e.g., `troff`, which Brian Kernighan himself ported to C.) I’m used to having my work shorthanded as “ooh, you make things pretty”. But that’s not at all how I think of it. My goal is to create a program that does what I do, better than I do it. Then it will brick me.

Allow me to clarify: I didn't mean to suggest that Racket et al would be the 15th language - just that for me it'll be the 15th variant to learn and carry around (for some interpretation of "15th" ...).

Indeed, I would think that S-expressions pre-date just about every other structured text format, but they're not totally trivial to parse outside the Lisp / Scheme / Clojure / etc ecosystems. Long ago, I used to think that keeping everything in Tcl's {}-formatting style would make it timeless (it's a simple and fairly regular syntax), but that choice didn't really help interoperability with C/C++ or Python in the end. Then came the early wiki markup languages, followed by Markdown. That then turns out to be too restrictive, at which point Leanpub invented "Markua". Etc, etc, etc. So even when ignoring all semantics, the syntax choice is still a balancing act: make the notation too crude, and it won't address real-world needs. Make it too complex, and it creates a tool (or even language) dependency.

One could argue that JSON is now one of the most "interoperable" formats, but it has major shortcomings for text-oriented use, e.g. no comments, only Unicode via escapes, and too much line noise for "structured" prose, at least for a lazy human like me.

Just a thought: if you're aiming for a reference implementation / widespread use, perhaps the hesitance to adopting this notation could be reduced with a simple trick: define & provide a precise transformation to and from JSON?

Anyway. I personally very much like the idea of using S-expressions and the lozenge as intentionally-unconventional escape marker, and look forward to trying it out on more than just introductory examples.

Allow me to clarify: I didn't mean to suggest that Racket et al would be the 15th language - just that _for me_ it'll be the 15th variant to learn and carry around (for some interpretation of "15th" ...). Indeed, I would think that S-expressions pre-date just about every other structured text format, but they're not totally trivial to parse outside the Lisp / Scheme / Clojure / etc ecosystems. Long ago, I used to think that keeping everything in Tcl's `{}`-formatting style would make it timeless (it's a simple and fairly regular syntax), but that choice didn't really help interoperability with C/C++ or Python in the end. Then came the early wiki markup languages, followed by Markdown. That then turns out to be too restrictive, at which point Leanpub invented "Markua". Etc, etc, etc. So even when ignoring all semantics, the _syntax_ choice is still a balancing act: make the notation too crude, and it won't address real-world needs. Make it too complex, and it creates a tool (or even language) dependency. One could argue that JSON is now one of the most "interoperable" formats, but it has major shortcomings for text-oriented use, e.g. no comments, only Unicode via escapes, and too much line noise for "structured" prose, at least for a lazy human like me. Just a thought: if you're aiming for a reference implementation / widespread use, perhaps the hesitance to adopting this notation could be reduced with a simple trick: define & provide a _precise_ transformation _to_ and _from_ JSON? Anyway. I personally very much _like_ the idea of using S-expressions and the lozenge as intentionally-unconventional escape marker, and look forward to trying it out on more than just introductory examples.

if you're aiming for a reference implementation / widespread use, perhaps the hesitance to adopting this notation could be reduced

These are two separate things. “Reference implementation” just means the ideas are laid out in such a way that someone can reconstruct the algorithm elsewhere. “Widespread use” is not something I care about or design for. On the contrary, the learning curve of Pollen is deliberately constructed to deter casual users, and thereby avoid wasting their time (and mine).

> if you're aiming for a reference implementation / widespread use, perhaps the hesitance to adopting this notation could be reduced These are two separate things. “Reference implementation” just means the ideas are laid out in such a way that someone can reconstruct the algorithm elsewhere. “Widespread use” is not something I care about or design for. On the contrary, the learning curve of Pollen is deliberately constructed to deter casual users, and thereby avoid wasting their time (and mine).

not something I care about or design for

Understand (so your world is not about "clicks", "likes", and "views"? how refreshing, for a change!).

I'll close this issue. Very much appreciate the thoughtful feedback. Many thanks.

> not something I care about or design for Understand (so your world is not about "clicks", "likes", and "views"? how refreshing, for a change!). I'll close this issue. Very much appreciate the thoughtful feedback. Many thanks.

Labels Milestones

Some (long) thoughts about the long term #101