You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
pollen/doc/Backstory.html

2 lines
22 KiB
HTML

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>3&nbsp;Backstory</title><link rel="stylesheet" type="text/css" href="scribble.css" title="default"/><link rel="stylesheet" type="text/css" href="racket.css" title="default"/><link rel="stylesheet" type="text/css" href="manual-style.css" title="default"/><link rel="stylesheet" type="text/css" href="manual-racket.css" title="default"/><script type="text/javascript" src="scribble-common.js"></script><script type="text/javascript" src="manual-racket.js"></script><!--[if IE 6]><style type="text/css">.SIEHidden { overflow: hidden; }</style><![endif]--></head><body id="scribble-racket-lang-org"><div class="tocset"><div class="tocview"><div class="tocviewlist tocviewlisttopspace"><div class="tocviewtitle"><table cellspacing="0" cellpadding="0"><tr><td style="width: 1em;"><a href="javascript:void(0);" title="Expand/Collapse" class="tocviewtoggle" onclick="TocviewToggle(this,&quot;tocview_0&quot;);">&#9660;</a></td><td></td><td><a href="index.html" class="tocviewlink" data-pltdoc="x">Pollen:<span class="mywbr"> &nbsp;</span> the book is a program</a></td></tr></table></div><div class="tocviewsublisttop" style="display: block;" id="tocview_0"><table cellspacing="0" cellpadding="0"><tr><td align="right">1&nbsp;</td><td><a href="Installation.html" class="tocviewlink" data-pltdoc="x">Installation</a></td></tr><tr><td align="right">2&nbsp;</td><td><a href="quick-tour.html" class="tocviewlink" data-pltdoc="x">Quick tour</a></td></tr><tr><td align="right">3&nbsp;</td><td><a href="Backstory.html" class="tocviewselflink" data-pltdoc="x">Backstory</a></td></tr><tr><td align="right">4&nbsp;</td><td><a href="big-picture.html" class="tocviewlink" data-pltdoc="x">The big picture</a></td></tr><tr><td align="right">5&nbsp;</td><td><a href="first-tutorial.html" class="tocviewlink" data-pltdoc="x">First tutorial</a></td></tr><tr><td align="right">6&nbsp;</td><td><a href="second-tutorial.html" class="tocviewlink" data-pltdoc="x">Second tutorial</a></td></tr><tr><td align="right">7&nbsp;</td><td><a href="third-tutorial.html" class="tocviewlink" data-pltdoc="x">Third tutorial</a></td></tr><tr><td align="right">8&nbsp;</td><td><a href="raco-pollen.html" class="tocviewlink" data-pltdoc="x">Using <span class="stt">raco pollen</span></a></td></tr><tr><td align="right">9&nbsp;</td><td><a href="File_formats.html" class="tocviewlink" data-pltdoc="x">File formats</a></td></tr><tr><td align="right">10&nbsp;</td><td><a href="reader.html" class="tocviewlink" data-pltdoc="x">&#9674; command overview</a></td></tr><tr><td align="right">11&nbsp;</td><td><a href="Module_reference.html" class="tocviewlink" data-pltdoc="x">Module reference</a></td></tr><tr><td align="right">12&nbsp;</td><td><a href="Acknowledgments.html" class="tocviewlink" data-pltdoc="x">Acknowledgments</a></td></tr><tr><td align="right">13&nbsp;</td><td><a href="License___source_code.html" class="tocviewlink" data-pltdoc="x">License &amp; source code</a></td></tr><tr><td align="right"></td><td><a href="doc-index.html" class="tocviewlink" data-pltdoc="x">Index</a></td></tr></table></div></div><div class="tocviewlist"><table cellspacing="0" cellpadding="0"><tr><td style="width: 1em;"><a href="javascript:void(0);" title="Expand/Collapse" class="tocviewtoggle" onclick="TocviewToggle(this,&quot;tocview_1&quot;);">&#9658;</a></td><td>3&nbsp;</td><td><a href="Backstory.html" class="tocviewselflink" data-pltdoc="x">Backstory</a></td></tr></table><div class="tocviewsublistbottom" style="display: none;" id="tocview_1"><table cellspacing="0" cellpadding="0"><tr><td align="right">3.1&nbsp;</td><td><a href="Backstory.html#%28part._.Web_development_and_its_discontents%29" class="tocviewlink" data-pltdoc="x">Web development and its discontents</a></td></tr><tr><td align="right">3.2&nbsp;</td><td><a href="Backstory.html#%28part._.The_better_idea__a_programming_model%29" class="tocviewlink" data-pltdoc="x">The better idea:<span class="mywbr"> &nbsp;</span> a programming model</a></td></tr><tr><td align="right">3.3&nbsp;</td><td><a href="Backstory.html#%28part.__.Now_you_have_two_problems_%29" class="tocviewlink" data-pltdoc="x">&ldquo;Now you have two problems&rdquo;</a></td></tr><tr><td align="right">3.4&nbsp;</td><td><a href="Backstory.html#%28part._.Rethinking_the_solution_for_digital_books%29" class="tocviewlink" data-pltdoc="x">Rethinking the solution for digital books</a></td></tr><tr><td align="right">3.5&nbsp;</td><td><a href="Backstory.html#%28part._.Enter_.Racket%29" class="tocviewlink" data-pltdoc="x">Enter Racket</a></td></tr><tr><td align="right">3.6&nbsp;</td><td><a href="Backstory.html#%28part._.What_is_.Pollen_%29" class="tocviewlink" data-pltdoc="x">What is Pollen?</a></td></tr><tr><td align="right">3.7&nbsp;</td><td><a href="Backstory.html#%28part._why-racket-why-lisp%29" class="tocviewlink" data-pltdoc="x">Further reading</a></td></tr></table></div></div></div><div class="tocsub"><div class="tocsubtitle">On this page:</div><table class="tocsublist" cellspacing="0"><tr><td><span class="tocsublinknumber">3.1<tt>&nbsp;</tt></span><a href="#%28part._.Web_development_and_its_discontents%29" class="tocsubseclink" data-pltdoc="x">Web development and its discontents</a></td></tr><tr><td><span class="tocsublinknumber">3.2<tt>&nbsp;</tt></span><a href="#%28part._.The_better_idea__a_programming_model%29" class="tocsubseclink" data-pltdoc="x">The better idea:<span class="mywbr"> &nbsp;</span> a programming model</a></td></tr><tr><td><span class="tocsublinknumber">3.3<tt>&nbsp;</tt></span><a href="#%28part.__.Now_you_have_two_problems_%29" class="tocsubseclink" data-pltdoc="x">&ldquo;Now you have two problems&rdquo;</a></td></tr><tr><td><span class="tocsublinknumber">3.4<tt>&nbsp;</tt></span><a href="#%28part._.Rethinking_the_solution_for_digital_books%29" class="tocsubseclink" data-pltdoc="x">Rethinking the solution for digital books</a></td></tr><tr><td><span class="tocsublinknumber">3.5<tt>&nbsp;</tt></span><a href="#%28part._.Enter_.Racket%29" class="tocsubseclink" data-pltdoc="x">Enter Racket</a></td></tr><tr><td><span class="tocsublinknumber">3.6<tt>&nbsp;</tt></span><a href="#%28part._.What_is_.Pollen_%29" class="tocsubseclink" data-pltdoc="x">What is Pollen?</a></td></tr><tr><td><span class="tocsublinknumber">3.7<tt>&nbsp;</tt></span><a href="#%28part._why-racket-why-lisp%29" class="tocsubseclink" data-pltdoc="x">Further reading</a></td></tr></table></div></div><div class="maincolumn"><div class="main"><div class="versionbox"><span class="version">6.1.0.5</span></div><div class="navsettop"><span class="navleft"><div class="nosearchform"></div>&nbsp;&nbsp;</span><span class="navright">&nbsp;&nbsp;<a href="quick-tour.html" title="backward to &quot;2 Quick tour&quot;" data-pltdoc="x">&larr; prev</a>&nbsp;&nbsp;<a href="index.html" title="up to &quot;Pollen: the book is a program&quot;" data-pltdoc="x">up</a>&nbsp;&nbsp;<a href="big-picture.html" title="forward to &quot;4 The big picture&quot;" data-pltdoc="x">next &rarr;</a></span>&nbsp;</div><h3>3<tt>&nbsp;</tt><a name="(part._.Backstory)"></a>Backstory</h3><p>I created Pollen to overcome limitations &amp; frustrations I repeatedly encountered with existing web-publishing tools.</p><p>If you agree with my characterization of those problems, then you&rsquo;ll probably like the solution that Pollen offers. If not, you probably won&rsquo;t.</p><h4>3.1<tt>&nbsp;</tt><a name="(part._.Web_development_and_its_discontents)"></a>Web development and its discontents</h4><p>I made my first web page in 1994, shortly after the web was invented. I opened my text editor (at the time, <a href="http://www.barebones.com/products/bbedit/">BBEdit</a>), pecked out <span class="RktSym">&lt;html&gt;&lt;body&gt;Hello</span><span class="RktMeta"></span><span class="hspace">&nbsp;</span><span class="RktMeta"></span><span class="RktSym">world&lt;/body&gt;&lt;/html&gt;</span><span class="RktMeta"></span>, then loaded it in <a href="http://en.wikipedia.org/wiki/Mosaic_(web_browser)">Mosaic</a>. So did a million other nerds.</p><p>If you weren&rsquo;t around then, you didn&rsquo;t miss much. Everything about the web was horrible: the web browsers, the computers running the browsers, the dial-up connections feeding the browsers, and of course HTML itself. At that point, the desktop-software experience was already slick and refined. By comparison, using the web felt like banging rocks together.</p><p>That&rsquo;s no longer true. The web is now more than 20 years old. During that time, most parts of the web have improved dramatically &#8212; for instance, the connections are faster, the browsers are more sophisticated, the screens have more pixels.</p><p>But one part has not improved: the way we make web pages. Over the years, tools promising to simplify HTML development have come and mostly gone &#8212;&#160;from <a href="http://www.macobserver.com/reviews/pagemill2.shtml">PageMill</a> to <a href="http://www.adobe.com/products/dreamweaver.html">Dreamweaver</a> to <a href="http://wordpress.org">WordPress</a> to <a href="http://jekyllrb.com">Jekyll</a>. Meanwhile, true web jocks have remained loyal to the original HTML power tool: the humble text editor.</p><p>In one way, this makes sense. Web pages are mostly made of text-based data &#8212;&#160;HTML, CSS, JavaScript, and so on &#8212; and the simplest way to mainpulate this data is with a text editor. While HTML and CSS are <a href="http://programmers.stackexchange.com/questions/28098/why-does-it-matter-that-html-and-css-are-not-programming-languages">not</a> programming languages, they lend themselves to semantic and logical structure that&rsquo;s most easily expressed by editing them as text. Furthermore, text-based editing makes debugging and performance improvements easier.</p><p>But text-based editing is also limited. Though the underlying description of a web page is notionally human-readable, it&rsquo;s optimized to be readable by other software &#8212; namely, web browsers. HTML markup in particular is verbose and easily mistyped. And isn&rsquo;t it fatally dull to manage all the boilerplate, like surrounding every paragraph with <span class="RktSym">&lt;p&gt;...&lt;/p&gt;</span><span class="RktMeta"></span>? Yes, it is.</p><p>For these reasons, much of web development should lend itself to <span style="font-style: italic">abstraction</span> &amp; <span style="font-style: italic">automation</span>. Abstraction means consolidating repetitve, complex patterns into simpler, parameterized forms. Automation means avoiding the manual drudgery of generating the output files. But in practice, tools that enable this abstraction &amp; automation have been slow to arrive, and most have come hobbled with unacceptable deficiencies.</p><h4>3.2<tt>&nbsp;</tt><a name="(part._.The_better_idea__a_programming_model)"></a>The better idea: a programming model</h4><p>Parallel with my HTML education, I also goofed around with various programming languages &#8212;&#160;C, C++, Perl, Java, PHP, JavaScript, Python. Unlike HTML, programming languages excel at abstraction and automation. This seemed like the obvious direction for web development to go.</p><p>What distinguishes the text-editing model from the programming model? It&rsquo;s a matter of direct vs. indirect manipulation of output. The text-editing model treats HTML as something to be written directly with a text editor. Whereas the programming model treats HTML &#8212;&#160;or whatever the output is &#8212;&#160;as the result of compiling a set of source files, which are written in a programming language. The costs of working indirectly via the programming language are offset by the benefits of abstraction &amp; automation.</p><p>On the early web, the text-editing model was appealingly precise and quick. On small projects, it worked well enough. But as projects grew, the text-editing model was going to lose steam. I wasn&rsquo;t the only one to notice. Shortly after those million nerds made their first web page by hand, many of them set about devising ways to apply a programming model to web development.</p><h4>3.3<tt>&nbsp;</tt><a name="(part.__.Now_you_have_two_problems_)"></a>&ldquo;Now you have two problems&rdquo;</h4><p>What followed was a steady stream of products, frameworks, tools, and content management systems that claimed to bring a programming model to web development. Some were better than others. But none of them displaced the text editor as the preferred tool of web developers.</p><p>Why not? All these tools promised a great leap forward in solving the web-development problem. In practice, they simply redistributed the pain. I needn&rsquo;t bore you with enumerating the deficiencies of specific tools, because they&rsquo;ve tended to fail in the same thematic ways:</p><ul><li><p><span style="font-weight: bold">No native data structure for HTML.</span> Core to any programming model is data structures. Good data structures make processing easy; bad ones make it hard. Even though HTML has a <a href="http://www.w3.org/TR/html401/struct/global.html">well documented</a> format, rarely has it been handled within a programming system with a native, intuitive data structure. Instead, it&rsquo;s either been treated as a string (wrong), a tree (also wrong), or some magical parsed object. This has made working with HTML in programming environments needlessly difficult.</p></li><li><p><span style="font-weight: bold">Mandatory separation of code, presentation, and content.</span> This principle has often been <a href="http://alistapart.com/article/separationdilemma/">held out</a> as an ideal in web development. But it&rsquo;s also counterintuitive, because an HTML page naturally contains all three. If you want to separate them, your tools should let you. But if you don&rsquo;t, your tools shouldn&rsquo;t make you.</p></li><li><p><span style="font-weight: bold">Compromised template languages.</span> It seems like every programming language has at least 10 templating systems for HTML, all of which require you to learn a new &ldquo;template language&rdquo; that offers the worst of both worlds: fewer features and different syntax than the underlying language.</p></li><li><p><span style="font-weight: bold">Steep learning curves.</span> Web programmers have often chided designers for not knowing <a href="http://elliotjaystocks.com/blog/web-designers-who-cant-code/">how to code</a>. But programming-based web-development tools have often had a high initial learning curve that requires you to throw out your existing workflow. Programmers built these tools &#8212; no surprise that programmers have been more comfortable with them.</p></li></ul><p>I&rsquo;ve tried a lot of these tools over the years. Some I liked. Some I didn&rsquo;t. Invariably, however, whenever I could still make do with hand-editing an HTML project, I would. After trying to cajole the web framework du jour into doing my bidding, it was relaxing to trade off some efficiency for control.</p><h4>3.4<tt>&nbsp;</tt><a name="(part._.Rethinking_the_solution_for_digital_books)"></a>Rethinking the solution for digital books</h4><p>In 2008, I launched a website called <a href="http://typographyforlawyers.com"><span style="font-style: italic">Typography for Lawyers</span></a>. Initially, I&rsquo;d conceived of it as a book. Then I thought &ldquo;no one&rsquo;s going to publish that.&rdquo; So it became a website, that I aimed to make as book-like as possible. But hand-editing wasn&rsquo;t going to be enough.</p><p>So I used <a href="http://wordpress.org">WordPress</a>. The major chore became scraping out all the crap that typically lives in blog templates. Largely because of this, people <a href="http://ma.tt/2010/04/typography-for-lawyers/">liked the site</a>, because it was simpler &amp; cleaner than the usual WordPress website.</p><p>Eventually, a publisher offered to release it as <a href="http://typo.la/amzn">a paperback</a>, which came out in 2010.</p><p>Later came the inevitable request to make it into a Kindle book. As a fan of typography, I hate the Kindle. The layout controls are coarse, and so is the reading experience. But I didn&rsquo;t run and hide. Basically a Kindle book is a little website made with 1995-era HTML. So I coded up some tools in Perl to convert my book to Kindle format while preserving the formatting and images as well as possible.</p><p>At that point, I noticed I had converted <span style="font-style: italic">Typography for Lawyers</span> into web format twice, using two different sets of tools. Before someone asked me to do it a third time, I started thinking about how I might create source code for the book that allowed me to render it into different formats.</p><p>That was the beginning of the Pollen project.</p><p>I wrote the initial version of Pollen in Python. I devised a simplified markup-notation language for the source files. This language was compiled into XML-ish data structures using <a href="http://www.dabeaz.com/ply/">ply</a> (Python lex/yacc). These structures were parsed into trees using <a href="http://lxml.de/">LXML</a>. The trees were combined with templates made in <a href="http://chameleon.readthedocs.org/en/latest/">Chameleon</a>. These templates were rendered and previewed with the <a href="http://bottlepy.org/">Bottle</a> web server.</p><p>Did it work? Sort of. Source code went in; web pages came out. But it was also complicated and fragile. Moreover, though the automation was there, there wasn&rsquo;t yet enough abstraction at the source layer. I started thinking about how I could add a source preprocessor.</p><h4>3.5<tt>&nbsp;</tt><a name="(part._.Enter_.Racket)"></a>Enter Racket</h4><p>I had come across Racket while researching languages suitable for HTML/XML processing. I had unexpectedly learned about the <a href="http://www.defmacro.org/ramblings/lisp.html">secret kinship</a> of XML and Lisp: though XML is not a programming language, it uses a variant of Lisp syntax. Thus Lisp languages are particularly adept at handling XMLish structures. That was interesting.</p><p>After comparing some of the Lisp &amp; Scheme variants, Racket stood out because it had a text-based dialect called Scribble. Scribble could be used to embed code within textual content. That was interesting too. Among other things, this meant Scribble could be used as a general-purpose preprocessor. So I thought I&rsquo;d see if I could add it to Pollen.</p><p>It worked. So well, in fact, that I started thinking about whether I could reimplement other parts of Pollen in Racket. Then I started thinking about reimplementing all of it in Racket.</p><p>So I did. And here we are.</p><h4>3.6<tt>&nbsp;</tt><a name="(part._.What_is_.Pollen_)"></a>What is Pollen?</h4><p>Pollen is a publishing system built on top of Scribble and Racket. So far, I&rsquo;ve optimized Pollen for digital books, because that&rsquo;s mainly what I use it for. But it can be used for small projects too.</p><p>As a publishing system, Pollen includes:</p><ul><li><p><span style="font-weight: bold">A programming language.</span> The Pollen language is a variant of Scribble, with specific dialects tailored to different kinds of source files. You don&rsquo;t need to use the programming features to do useful work, but they&rsquo;re available when you need them.</p></li><li><p><span style="font-weight: bold">A set of tools &amp; libraries.</span> Pollen targets HTML output. So it includes a variety of tools that cure common HTML annoyances, including a CSS preprocessor.</p></li><li><p><span style="font-weight: bold">A development environment.</span> Pollen works with the DrRacket IDE. It also includes a project web server so you can dynamically preview and revise your publication.</p></li></ul><p>Pollen addresses the deficiencies I experienced with other tools:</p><ul><li><p><span style="font-weight: bold">Yes, we have a native data structure for HTML.</span> Racket represents HTML structures as <span style="font-style: italic">X-expressions</span>, which are a variant of the standard Racket data structure, called <span style="font-style: italic">S-expressions</span>. In other words, not only is there a native representation for HTML, everything in the language is represented this way.</p></li><li><p><span style="font-weight: bold">Flexible blending of code, presentation, and content.</span> Pollen is a text-based language. So a Pollen source file might have no code at all. But as a dialect of Scribble &amp; Racket, if you want to mix code with content, you can.</p></li><li><p><span style="font-weight: bold">No template language.</span> It&rsquo;s not necessary, because you can use the whole Racket language, and all the usual Racket syntax, in every Pollen file.</p></li><li><p><span style="font-weight: bold">Shallow learning curve.</span> You don&rsquo;t need to do a lot of setup and configuration to start doing useful work with Pollen. Programmers and non-programmers can easily collaborate. Yes, I concede that if you plan to get serious, you&rsquo;ll need to learn some Racket. I don&rsquo;t think you&rsquo;ll regret it.</p></li></ul><h4>3.7<tt>&nbsp;</tt><a name="(part._why-racket-why-lisp)"></a>Further reading</h4><p>In <a href="http://practicaltypography.com/why-racket-why-lisp.html"><span style="font-style: italic">Why Racket? Why Lisp?</span></a>, I explain why Racket was the right tool for this job.</p><div class="navsetbottom"><span class="navleft"><div class="nosearchform"></div>&nbsp;&nbsp;</span><span class="navright">&nbsp;&nbsp;<a href="quick-tour.html" title="backward to &quot;2 Quick tour&quot;" data-pltdoc="x">&larr; prev</a>&nbsp;&nbsp;<a href="index.html" title="up to &quot;Pollen: the book is a program&quot;" data-pltdoc="x">up</a>&nbsp;&nbsp;<a href="big-picture.html" title="forward to &quot;4 The big picture&quot;" data-pltdoc="x">next &rarr;</a></span>&nbsp;</div></div></div><div id="contextindicator">&nbsp;</div></body></html>