Problem with syntax highlighter replacing &s #39

Hi,

I was trying the example of syntax highlight provided in the source code comments:

◊(require pollen/pygments)
◊(make-highlight-css)


◊highlight['python]{ 
for x in range(10):
    print x
if zero is False:
    print "Hello world"}

but I get this exception:

The application raised an exception with the message:
regexp-replace*: contract violation
  expected: (or/c string? bytes?)
  given: '()
Stack trace:
core83 at:
  line 362, column 11, in file /Applications/Racket v6.1.1/collects/racket/private/string.rkt
decode-ampersands-in-attributes at:
  line 43, column 0, in file /[...]/Racket/6.1.1/pkgs/pollen/highlight.rkt
read-html-as-xexprs at:
  line 54, column 0, in file /[...]/Racket/6.1.1/pkgs/pollen/highlight.rkt
highlight at:
  line 111, column 0, in file /[...]/Racket/6.1.1/pkgs/pollen/highlight.rkt
[running body] at:
  line #f, column #f, in file (submod /[...]/erlang_doc/erlang_tut.html.pm pollen-lang-module inner)
[traversing imports] at:
  line #f, column #f, in file (submod /[...]/erlang_doc/erlang_tut.html.pm pollen-lang-module)
[traversing imports] at:

I have tried to fix it, but, frankly, my racket skills are not enough. I wonder if the format of the data provided to decode-ampersands-in-attributes has changed.

Hi, I was trying the example of syntax highlight provided in the source code comments: ``` ◊(require pollen/pygments) ◊(make-highlight-css) ◊highlight['python]{ for x in range(10): print x if zero is False: print "Hello world"} ``` but I get this exception: ``` The application raised an exception with the message: regexp-replace*: contract violation expected: (or/c string? bytes?) given: '() Stack trace: core83 at: line 362, column 11, in file /Applications/Racket v6.1.1/collects/racket/private/string.rkt decode-ampersands-in-attributes at: line 43, column 0, in file /[...]/Racket/6.1.1/pkgs/pollen/highlight.rkt read-html-as-xexprs at: line 54, column 0, in file /[...]/Racket/6.1.1/pkgs/pollen/highlight.rkt highlight at: line 111, column 0, in file /[...]/Racket/6.1.1/pkgs/pollen/highlight.rkt [running body] at: line #f, column #f, in file (submod /[...]/erlang_doc/erlang_tut.html.pm pollen-lang-module inner) [traversing imports] at: line #f, column #f, in file (submod /[...]/erlang_doc/erlang_tut.html.pm pollen-lang-module) [traversing imports] at: ``` I have tried to fix it, but, frankly, my racket skills are not enough. I wonder if the format of the data provided to `decode-ampersands-in-attributes` has changed.

Yes, there was an error in that function. Now fixed. Thanks for the report.

BTW, the syntax highlighting that’s in Pollen right now is of the quick-and-dirty variety (which is why I don’t mention it in the documentation). If it works for you, great. I just want to keep expectations appropriately low.

Yes, there was an error in that function. Now fixed. Thanks for the report. BTW, the syntax highlighting that’s in Pollen right now is of the quick-and-dirty variety (which is why I don’t mention it in the documentation). If it works for you, great. I just want to keep expectations appropriately low.

Absolutely, I saw that it was experimental. I am just dabbling with pollen (which I find great, btw), and the kind of expectations I have are the in the level of "I think I will comment the decode-ampersands-in-attributescall and fix manually the ampersands if there are one or two in the code", so no worries :).

Thanks.

Absolutely, I saw that it was experimental. I am just dabbling with pollen (which I find great, btw), and the kind of expectations I have are the in the level of "I think I will comment the `decode-ampersands-in-attributes`call and fix manually the ampersands if there are one or two in the code", so no worries :). Thanks.

I should add that I think syntax highlighting is a good and worthwhile thing to have in Pollen. I would welcome suggestions about how it could / should work. I haven’t done any projects yet that require syntax highlighting, so I haven’t formed opinions.

One conundrum I’ve faced is that Racket doesn’t have its own syntax-highlighting library. I don’t mind relying on an external library. But I also want to create the most benefit with the fewest dependencies.
For now, I’ve been relying on Pygments because @GregHendershott already integrated it with Racket. But for those who don’t use Python, dealing with Pygments is probably a drag. That suggests that using a JavaScript syntax highlighter (e.g., highlight.js) might be a better option.

I should add that I think syntax highlighting is a good and worthwhile thing to have in Pollen. I would welcome suggestions about how it could / should work. I haven’t done any projects yet that require syntax highlighting, so I haven’t formed opinions. One conundrum I’ve faced is that Racket doesn’t have its own syntax-highlighting library. I don’t mind relying on an external library. But I also want to create the most benefit with the fewest dependencies. For now, I’ve been relying on Pygments because @GregHendershott already integrated it with Racket. But for those who don’t use Python, dealing with Pygments is probably a drag. That suggests that using a JavaScript syntax highlighter (e.g., [highlight.js](https://github.com/isagalaev/highlight.js)) might be a better option.

I don't think anything comes close to Pygments in terms of the number and quality of lexers. Not any of the JS highlighting libs. (And certainly not the awful stuff GitHub has been doing lately.)

Although Pygments requires Python, AFAIK it works fine with the older Python that's probably already installed on OS X or most Linux distros, if not Windows. Using it as a command-line utility from Racket via system is easy. (In Frog I start it once and pipe multiple things through it -- but that's to eke out more speed, not to make it work well at all.)

The output is plain HTML so it works in browsers where JS is slow or disabled by the user or absent.

When you're using a tool like Pollen or Frog to prepare the HTML, I think it makes sense also to prepare the syntax highlighting then, vs. leaving it to (hopefully) happen JIT on the user's device. If you're using that kind of tool, you probably care about the appearance of the syntax highlighting as much as everything else you're preparing.

Of course if the tool chain is all JS, or the whole point is to interactively change highlighting or whatever, then doing it in JS is the way to go.

I don't think anything comes close to Pygments in terms of the number and quality of lexers. Not any of the JS highlighting libs. (And certainly not the awful stuff GitHub has been doing lately.) Although Pygments requires Python, AFAIK it works fine with the older Python that's probably already installed on OS X or most Linux distros, if not Windows. Using it as a command-line utility from Racket via `system` is easy. (In Frog I start it once and pipe multiple things through it -- but that's to eke out more speed, not to make it work well at all.) The output is plain HTML so it works in browsers where JS is slow or disabled by the user or absent. When you're using a tool like Pollen or Frog to prepare the HTML, I think it makes sense also to prepare the syntax highlighting then, vs. leaving it to (hopefully) happen JIT on the user's device. If you're using that kind of tool, you probably care about the appearance of the syntax highlighting as much as everything else you're preparing. Of course if the tool chain is all JS, or the whole point is to interactively change highlighting or whatever, then doing it in JS is the way to go.

All true. Other things being equal, I’d rather have syntax highlighting at compile time, because a) I don’t like delegating work to web browsers, because that work is often done poorly, or not at all; b) a JS syntax highlighter requires a browser — it can’t be used with a system that targets, say, PDFs or ebooks.

But since I’m avoiding the work of making or maintaining a syntax highlighter, it would probably be simple enough to support both methods (Pygments & highlight.js, for instance).

All true. Other things being equal, I’d rather have syntax highlighting at compile time, because a) I don’t like delegating work to web browsers, because that work is often done poorly, or not at all; b) a JS syntax highlighter requires a browser — it can’t be used with a system that targets, say, PDFs or ebooks. But since I’m avoiding the work of making or maintaining a syntax highlighter, it would probably be simple enough to support both methods (Pygments & highlight.js, for instance).

Any more info on this? Syntax highlighting is of key importance to me when considering Pollen, so any recent thinking or guidance in hownto use it as is would be great.

I haven’t added it to the docs yet, but there is now a pollen/pygments module. I used it for syntax highlighting in this recent article (source listings available at the bottom).

How to use pollen/pygments

Make sure you have pygments already installed.
Add (require pollen/pygments) to your Pollen source file, or …
Add (require pollen/pygments) and (provide highlight) to your directory-require.rkt file for your project.
The Pollen syntax to invoke highlighting is ◊highlight[lang-name]{code to highlight ...}.
This will create the correct markup, but you still need to add styles to your CSS to change the appearance of the code. If you look at the CSS source here, I just pasted a Pygments theme at the bottom (that I picked up from https://github.com/richleland/pygments-css, and then edited with some variable values.)

I concede that step 5 isn’t super convenient. But I haven’t yet decided how it to make it easier. Pasting a bunch of prefab Pygments themes into the Pollen distribution doesn’t make sense. Moreover, even if convenient, they wouldn’t be editable / programmable, which is sort of the point of the whole exercise.

Anyhow, I’m open to suggestions on that last point.

I haven’t added it to the docs yet, but there is now a `pollen/pygments` module. I used it for syntax highlighting in [this recent article](http://unitscale.com/mb/technique/dual-typed-untyped-library.html) (source listings available at the bottom). **How to use `pollen/pygments`** 1. Make sure you have `pygments` already installed. 2. Add `(require pollen/pygments)` to your Pollen source file, or … 3. Add `(require pollen/pygments)` and `(provide highlight)` to your `directory-require.rkt` file for your project. 4. The Pollen syntax to invoke highlighting is `◊highlight[lang-name]{code to highlight ...}`. 5. This will create the correct markup, but you still need to add styles to your CSS to change the appearance of the code. If you look at the [CSS source here](http://unitscale.com/mb/technique/styles.css.pp.html), I just pasted a Pygments theme at the bottom (that I picked up from https://github.com/richleland/pygments-css, and then edited with some variable values.) I concede that step 5 isn’t super convenient. But I haven’t yet decided how it to make it easier. Pasting a bunch of prefab Pygments themes into the Pollen distribution doesn’t make sense. Moreover, even if convenient, they wouldn’t be editable / programmable, which is sort of the point of the whole exercise. Anyhow, I’m open to suggestions on that last point.

PS. As for highlight.js, since it’s all browser-based, it doesn’t require any special cooperation from Pollen.

Add these lines to the <head> section of your template.html (or other template):

<link rel="stylesheet" href="/path/to/styles/default.css">
<script src="/path/to/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>

With these resources loaded, highlight.js will automatically highlight any HTML of the form <pre><code class="html">...</code></pre>. So in Pollen markup, you could write that directly like so:

#lang pollen/markup

◊pre{◊code['class: "html"]{code to highlight ...}}

Or if you wanted to match the notation for pollen/pygments, you could write a highlight function that expands to the same markup:

#lang pollen/markup

◊(define (highlight lang . xs)
   `(pre (code ((class ,(format "~a" lang))) ,@xs))) 

◊highlight['html]{code to highlight ...}

Again, I concede that it would be nicer to have Pollen automatically drop the necessary incantations into the <head> of your HTML. But as a matter of policy, I prefer to avoid magic behavior.

PS. As for `highlight.js`, since it’s all browser-based, it doesn’t require any special cooperation from Pollen. 1) Add these lines to the `<head>` section of your `template.html` (or other template): ``` html <link rel="stylesheet" href="/path/to/styles/default.css"> <script src="/path/to/highlight.pack.js"></script> <script>hljs.initHighlightingOnLoad();</script> ``` 2) With these resources loaded, `highlight.js` will automatically highlight any HTML of the form `<pre><code class="html">...</code></pre>`. So in Pollen markup, you could write that directly like so: ``` racket #lang pollen/markup ◊pre{◊code['class: "html"]{code to highlight ...}} ``` 3) Or if you wanted to match the notation for `pollen/pygments`, you could write a `highlight` function that expands to the same markup: ``` racket #lang pollen/markup ◊(define (highlight lang . xs) `(pre (code ((class ,(format "~a" lang))) ,@xs))) ◊highlight['html]{code to highlight ...} ``` Again, I concede that it would be nicer to have Pollen automatically drop the necessary incantations into the `<head>` of your HTML. But as a matter of policy, I prefer to avoid magic behavior.

Labels Milestones

Problem with syntax highlighter replacing &s #39