stumped by this bytecode-caching problem #48

If Pollen generates bytecode for #lang pollen files, builds are about 20% faster. But there is a bytecode-caching bug I can’t figure out, illustrated by the sample code below.

This sample simulates a render of a Pollen source "doc.rkt" that relies on "pollen.rkt". After "pollen.rkt" is changed and the bytecode regenerated for both files, only "pollen.rkt" can be re-evaluated correctly. When "doc.rkt" is re-evaluated, it shows the older value.

Why is this so?

At first, I thought the dependency manager was failing to notice that "doc.rkt" relies on "pollen.rkt" (and thus should be recompiled when "pollen.rkt" changes). But the tests using module-recorded-dependencies seem to disprove this theory.

Then I thought that the bytecode for "doc.rkt" wasn’t being refreshed on the second call to managed-compile-zo. But even if I erase the first set of bytecode files, the problem persists.

My only remaining theory is that managed-compile-zo has some deeper cache mechanism that needs to be overcome, but if so how?

#lang at-exp debug racket
(require compiler/cm compiler/depend rackunit)

(display-to-file @string-append{
 #lang racket/base
 (provide id)
 (define id "first")} "pollen.rkt" #:exists 'replace)

(display-to-file @string-append{#lang pollen
 ◊id doc} "doc.rkt" #:exists 'replace)

(managed-compile-zo "pollen.rkt")
(managed-compile-zo "doc.rkt")

;; proves that dependency manager notices that "doc.rkt" relies on "pollen.rkt"
(check-true (and (member (path->complete-path (string->path "pollen.rkt")) (module-recorded-dependencies "doc.rkt")) #true))

(parameterize ([current-namespace (make-base-namespace)])
  (check-equal? (dynamic-require "pollen.rkt" 'id) "first"))

(parameterize ([current-namespace (make-base-namespace)])
  (check-equal? (dynamic-require "doc.rkt" 'doc) "first doc"))

(display-to-file @string-append{
 #lang racket/base
 (provide id)
 (define id "second")} "pollen.rkt" #:exists 'replace)

;; we erase the bytecode!
;; we erase the bytecode!
(for ([bc '("compiled/pollen_rkt.zo" "compiled/doc_rkt.zo")])
     (when (file-exists? bc)
       (delete-file bc)))

(managed-compile-zo "pollen.rkt")
(managed-compile-zo "doc.rkt")

(check-true (and (member (path->complete-path (string->path "pollen.rkt")) (module-recorded-dependencies "doc.rkt")) #true))

(parameterize ([current-namespace (make-base-namespace)])
  (check-equal? (dynamic-require "pollen.rkt" 'id) "second"))

(parameterize ([current-namespace (make-base-namespace)])
  (check-equal? (dynamic-require "doc.rkt" 'doc) "second doc")) ; huh?!?!

If Pollen generates bytecode for `#lang pollen` files, builds are about 20% faster. But there is a bytecode-caching bug I can’t figure out, illustrated by the sample code below. This sample simulates a render of a Pollen source `"doc.rkt"` that relies on `"pollen.rkt"`. After `"pollen.rkt"` is changed and the bytecode regenerated for both files, only `"pollen.rkt"` can be re-evaluated correctly. When `"doc.rkt"` is re-evaluated, it shows the older value. **Why is this so?** At first, I thought the dependency manager was failing to notice that `"doc.rkt"` relies on `"pollen.rkt"` (and thus should be recompiled when `"pollen.rkt"` changes). But the tests using `module-recorded-dependencies` seem to disprove this theory. Then I thought that the bytecode for `"doc.rkt"` wasn’t being refreshed on the second call to `managed-compile-zo`. But even if I erase the first set of bytecode files, the problem persists. My only remaining theory is that `managed-compile-zo` has some deeper cache mechanism that needs to be overcome, but if so how? ```racket #lang at-exp debug racket (require compiler/cm compiler/depend rackunit) (display-to-file @string-append{ #lang racket/base (provide id) (define id "first")} "pollen.rkt" #:exists 'replace) (display-to-file @string-append{#lang pollen ◊id doc} "doc.rkt" #:exists 'replace) (managed-compile-zo "pollen.rkt") (managed-compile-zo "doc.rkt") ;; proves that dependency manager notices that "doc.rkt" relies on "pollen.rkt" (check-true (and (member (path->complete-path (string->path "pollen.rkt")) (module-recorded-dependencies "doc.rkt")) #true)) (parameterize ([current-namespace (make-base-namespace)]) (check-equal? (dynamic-require "pollen.rkt" 'id) "first")) (parameterize ([current-namespace (make-base-namespace)]) (check-equal? (dynamic-require "doc.rkt" 'doc) "first doc")) (display-to-file @string-append{ #lang racket/base (provide id) (define id "second")} "pollen.rkt" #:exists 'replace) ;; we erase the bytecode! ;; we erase the bytecode! (for ([bc '("compiled/pollen_rkt.zo" "compiled/doc_rkt.zo")]) (when (file-exists? bc) (delete-file bc))) (managed-compile-zo "pollen.rkt") (managed-compile-zo "doc.rkt") (check-true (and (member (path->complete-path (string->path "pollen.rkt")) (module-recorded-dependencies "doc.rkt")) #true)) (parameterize ([current-namespace (make-base-namespace)]) (check-equal? (dynamic-require "pollen.rkt" 'id) "second")) (parameterize ([current-namespace (make-base-namespace)]) (check-equal? (dynamic-require "doc.rkt" 'doc) "second doc")) ; huh?!?! ```

This is indeed weird. I'm not sure what's going on here, but it's worth noting that v7.7 [cs] passes these tests, while v7.7 [non-cs] doesn't.

This is indeed weird. I'm not sure what's going on here, but it's worth noting that `v7.7 [cs]` passes these tests, while `v7.7 [non-cs]` doesn't.

Ah yes thanks, I should’ve tried that. Well, that tells me it’s a Racket bug (perhaps even longstanding).

For now, I put in the extra bytecode patch for Racket CS (and will extend it to BC later, if someone smarter figures out a patch). Still, given the comparatively slower performance of CS, the performance improvement is more valuable there.

Turns out it is not a bug in Racket, but rather a side effect of undefined behavior.

Turns out it is not a bug in Racket, but rather a [side effect of undefined behavior.](https://github.com/racket/racket/issues/3158)

Labels Milestones

stumped by this bytecode-caching problem #48