Q3 2026 Funding Survey

Greetings Clojurists!

Please take a moment to complete the Q3 2026 Funding Survey which helps inform our Q3 project awards. It is not a heavy lift - maybe 5 minutes of your time. Your input is invaluable! A link to the survey was sent to your email in the last few weeks - and just in case it made its way to spam, you can look for “We Need Your Input - Q3 2026 Funding”. The survey closes midnight PST on July 15, 2026.

Thanks as always for your support of Clojurists Together and for being a part of this awesome community.

Any questions, please email me at kdavis@clojuriststogether.org

Kathy Davis Program Manager Clojurists Together Foundation

Permalink

Destructuring with computed keys

In Clojure you can destructure a map using an arbitrary expression as the key. For example, here kw is a local binding.

(let [kw :key
      {a kw} {:key 1}]
  a)
;=> 1

Usually this syntax is demonstrated as {sym0 :kw0 sym1 :kw1 ...}, which doesn’t reveal that the keywords are actually in expression position, or an evaluation context. The reason why this more recognizable syntax works is because keyword literals are self-evaluating.

(let [{a :key} {:key 1}]
  a)
;=> 1

The basic rule for expanding these expressions is:

(let [{binding expression} map])
=>
(let [binding (get map expression)])

So that code is equivalent to:

(let [a (get {:key 1} :key)]
  a)

Symbols are not self-evaluating syntax in Clojure, so they must be quoted:

(let [{a 'key} {'key 1}]
  a)
;=> 1

Applying the rule makes the need more obvious:

(let [a (get {'key 1} 'key)]
  a)

Destructuring is pleasingly compositional. This ability to drop down to computed keys makes destructuring available in many more situations than if all keys were required to be statically declared. I found a few examples in my own code and other libraries where this flexibility has been useful.

An example that dereferences the var u/expr-type to compute the key:

(let [{cargs :args
       res u/expr-type} (-> expr-noinline
                            ana2/unmark-top-level
                            ana2/unmark-eval-top-level
                            (check-expr expected opts))]

Another example that uses three class literals as computed keys:

(let [r (reflect-u/reflect cls)
      {methods clojure.reflect.Method
       fields clojure.reflect.Field
       ctors clojure.reflect.Constructor
       :as members}
      (group-by
        class
        (filter (fn [{:keys [name] :as m}] 
                  (if constructor-call
                    (instance? clojure.reflect.Constructor m)
                    (= m-or-f name)))
                (:members r)))]

A snippet of code that destructures nested maps using a mix of keywords, quoted symbols and computed vectors-of-locals as keys. Notice that the vectors are in binding position sometimes to introduce names, then in expression position to perform lookups.


(let [...
      {{[x1 x2] 'x} :fv
       {[y1] 'y [z1 z2] 'z} :idx} remap
      {{{[y1_x1 y1_x2] 'x
         [y1_y1 y1_y2 y1_y3 y1_y4] 'y} [y1]
        {[z1_x1] 'x
         [z1_y1] 'y} [y1 z1]
        {[z2_x1] 'x
         [z2_y1] 'y} [y1 z2]} :idx-context} remap]
  (is (= {:fv {'x [x1 x2]}
          :idx {'y [y1]
                'z [z1 z2]}
          :idx-context {[y1] {'x [y1_x1 y1_x2]
                              'y [y1_y1 y1_y2 y1_y3 y1_y4]}
                        [y1 z1] {'x [z1_x1]
                                 'y [z1_y1]}
                        [y1 z2] {'x [z2_x1]
                                 'y [z2_y1]}}}
         remap)))

You’ve probably seen code like this that destructures booleans from a group-by:

(let [...
      {anns false inits true} (group-by list? normalised-bindings)]

This Malli snippet nests :keys destructuring under a local binding key, method:


(-value-transformer [_ schema method options]
  (reduce
   (fn [acc {{:keys [name qname default transformers]} method}]

And this example elegantly destructures a nested map using keywords and locals, supporting the common pattern of updating a nested value in an atom then destructuring the swapped-in value’s relevant parts.


(defn remove-stale-cache-entries
  [nsym ns-form-str sforms slurped opts]
  {:pre [(simple-symbol? nsym)]}
  (when ns-form-str
    (let [{{{forms-cache ns-form-str} nsym} ::check-form-cache}
          (env/swap-checker!
            (env/checker opts)
            update-in
            [::check-form-cache nsym]
            (fn [m]
              (some-> m
                      (select-keys [ns-form-str])
                      not-empty
                      (update ns-form-str select-keys sforms))))]

Permalink

biff.graph: structure your data model as a queryable graph

The next Biff 2 library is ready to go: biff.graph

biff.graph makes your codebase more understandable/maintainable by helping you split up your code for reading/deriving data into small independent chunks. "Data-oriented dependency injection" is a term I've used to describe the approach.

biff.graph is a lightweight clone/variant of Pathom. I'm a huge fan of Pathom and wanted to include it in Biff by default, however it is a bit of a heavy abstraction and I was concerned if the benefits would be worth the learning effort for people working on small projects.

So biff.graph is an attempt to provide something similar to Pathom in as few lines of code as possible (~600 to be specific, with ~200 for the query engine). You still have to learn the same conceptual model, but learning what exactly it's doing under the hood should be easier. The tradeoff is that biff.graph has less functionality--most significantly there's no query planner, so biff.graph isn't as intelligent about how it executes your queries as Pathom is.

Permalink

libl.in, & Injee, Thanks To Clojure Community. Updates in Clojure Book.

Notes

Permalink

Clojure 1.13.0-alpha3

Clojure 1.13.0-alpha3 is now available! Find download and usage information on the Downloads page.

:select directive in map destructuring

The :select directive binds a name to a subset of the map being destructured containing only the keys mentioned (anywhere) in the binding form.

  • CLJ-2964 :select directive in map destructuring

  • CLJ-2963 Update specs for :select in destructuring

Other changes since Clojure 1.13.0-alpha2

  • RT.map, and thus reader, tracks new PAM thresholds

  • CLJ-1789 select-keys - improve performance (transients, etc)

  • CLJ-2958 ILookup on sets

  • CLJ-2902 pprint - prints arbitary objects in unreadable form

  • CLJ-2801 TaggedLiteral - doesn’t define print-dup

  • CLJ-2269 definterface - does not resolve parameter type hints

  • CLJ-2781 clojure.test/report - docstring has broken references

  • CLJ-2929 zipper - docstring typo

  • CLJ-2901 bytes, shorts, chars - docstring typos

  • CLJ-2811 scalb - docstring links to the documentation for nextDown

  • CLJ-2809 clojure.math/floor - docstring has line that should be on ceil docstring

Try it out

Update your deps.edn :deps with:

org.clojure/clojure {:mvn/version "1.13.0-alpha3"}

Start a REPL with the Clojure CLI (any version) with:

clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.13.0-alpha3"}}}

Permalink

OSS updates May and June 2026

In this post I&aposll give updates about open source I worked on during May and June 2026.

To see previous OSS updates, go here.

Sponsors

I&aposd like to thank all the sponsors and contributors that make this work possible. Without you, the below projects would not be as mature or wouldn&apost exist or be maintained at all! So a sincere thank you to everyone who contributes to the sustainability of these projects.

gratitude

Current top tier sponsors:

Open the details section for more info about sponsoring.

Sponsor info

If you want to ensure that the projects I work on are sustainably maintained, you can sponsor this work in the following ways. If you work for a company that uses my OSS, please ask your employer, that would be even better. Thank you!

Updates

A lot happened in the past two months! Not just coding but also...

Babashka Conf 2026 and Dutch Clojure Days

Three years after the initial installment, Babashka Conf 2026 happened on May 8th at the OBA Oosterdok library in Amsterdam, with David Nolen, primary maintainer of ClojureScript, as our keynote speaker. Thanks to our sponsors Nubank, Exoscale, Bob, Flexiana and Itonomi, to Wendy Randolph for hosting, and to all the speakers, volunteers and attendees who made it such an inspiring day. You can watch all the videos here. Thanks to Ray for recording! The day after, Dutch Clojure Days 2026 rounded out a full weekend of Clojure in Amsterdam, where I did a presentation about ClojureScript and async/await. The video of that is hopefully coming soon.

Babashka Conf 2026 speakers and organizers

Babashka Conf 2026. From left to right: David Nolen, Jen Myers, Adrian Smith, Josh Glover, Rahul Dé, Arne Brasseur, Christoph Neumann, Timo Kramer, Jynn Nelson, Wendy Randolph.

Upcoming: babashka workshop at the Clojure/conj

I&aposm pleased to announce that Rahul Dé and I will be hosting a babashka workshop at the Clojure Conj 2026. The workshop will showcase various use cases of babashka. This hands-on workshop covers the whole lifecycle of a babashka tool, from a quick script to a published, installable CLI app. We assume you know the basics of Clojure and won&apost explain the language itself. Topics include:

  • Setting up your dev environment
  • Managing projects with babashka tasks (bb.edn)
  • A tour of built-in libraries (fs, process, http-client, and more)
  • Writing and running tests
  • Building a CLI with subcommands and automatic help
  • Programming a terminal UI (TUI)
  • Producing a small web app
  • Publishing via GitHub or as an installable tool with bbin

Every concept comes with an exercise, building toward one culminating CLI app. There will be lots of interaction and fun!

Blog posts

Besides this update I published two blog posts in the past two months:

and a ClojureScript reference on async functions:

Projects

Babashka CLI got the most attention this cycle. I added automatic --help generation for dispatch-based CLIs and shell tab completion for bash, zsh, fish, PowerShell and Nushell. There&aposs a dedicated post with a "build your own git" walkthrough linked above. I also made Babashka CLI Squint compatible, so CLIs built with it run on Node.js and in the browser, published as the @babashka/cli npm package. Also ClojureDart support for Babashka CLI got added.

Squint saw a large amount of work that kept going right into early July: a browser nREPL, dynamic vars and binding that survive across separately-compiled ESM modules, an EDN reader, cached lazy seqs, defrecord and a wide set of core protocols, and a big compatibility push to make it pass jank&aposs clojure-test-suite. Replicant now runs on Squint too. I added key diffing to Reagami and did some benchmarks, showing that Reagami on squint performs in the ballpark of React. The benchmark also shows that Replicant on Squint performs even a tad better than on ClojureScript. Not that this makes a huge difference in practice, but it&aposs nice to validate the idea that Squint, for typical apps, can be a valid CLJS replacement while not giving up that much in terms of Clojure features.

A security issue in SCI deserves a callout. A string type-hint could bypass the :classes allowlist and statically initialize any class on the classpath at analysis time. If you sandbox untrusted code with SCI, upgrade to 0.13.53. ClojureDart support and fine-grained interop control (which was needed for cljd support since it has no reflection) also got added. You can now make REPLs for your mobile apps!

Since porting was a theme these past months, I&aposll mention another one: babashka.fs now runs on Node.js via ClojureScript and squint, published as the @babashka/fs npm package.

Here are some highlights per project. See each project&aposs CHANGELOG.md for the full list.

  • babashka CLI: Turn Clojure functions into CLIs!

    • Automatic --help generation for dispatch CLIs, plus shell completions for bash, zsh, fish, PowerShell and Nushell (#112, #24, #95). I wrote a full post on it with a "write your own git" walkthrough: babashka CLI: automatic --help and shell completions
    • Exposed the underlying building blocks so you can roll your own custom CLI parsing: parse-opts*, coerce-opts, validate-opts, apply-defaults, table->tree
    • dispatch now accepts a tree directly (as returned by table->tree), and subcommand order is preserved in printed help and completions
    • Squint support and a new @babashka/cli npm package
    • ClojureDart support (#182)
    • opts->table accepts :columns to override the auto-detected columns (#148, thanks Jan Seeger)
    • Better error messages: negation errors now name the base option, --no-foo on a non-boolean option errors instead of silently coercing, and :edn :coerce now requires an explicit value (#166, #174)
    • Thanks to @lread for a lot of documentation review and general maintenance during this cycle
    • Full changelog

  • Squint: CLJS syntax to JS compiler

    • Browser nREPL support landed, followed by a number of REPL/nREPL fixes: #815 (str wrapping tripping esbuild), #819 (macro changes not picked up in watch mode), #820 (:macros option ignored from JS callers) and #832 (nREPL server hanging on advertised-but-unimplemented ops)
    • The CLI now gets its --help, usage and error handling from babashka.cli&aposs dispatch, plus shell tab completion
    • Dynamic vars and binding now work via a mutable box, safe across separately-compiled ESM modules; syntax-quote resolves symbols through the current namespace and aliases like Clojure. defprotocol got :extend-via-metadata support.
    • reify added
    • clojure.walk added
    • Added squint.edn/clojure.edn with a ~300-line EDN reader
    • Printing is now done through *print-fn*, print, pr and with-out-str, like CLJS
    • Lazy seqs are now cached instead of recomputed on every consumption, matching CLJS&aposs chunked-seq behavior
    • A big push for compatibility with jank&aposs clojure-test-suite: dozens of core functions (sorted-map, hash-map, subvec, pop, merge, keys/vals, peek, transducers, = on dates/regexes/lazy seqs, and more) now throw or behave exactly like CLJS instead of the old loose JS semantics, alongside full built-in cljs.test support
    • #771: dead-code elimination for varargs/multi-arity functions, now emitted via ... spread
    • Replicant support landed, with an example
    • Added defrecord, record? and the IRecord marker protocol. Records store their fields as own string-keyed properties and implement the map-facing protocols, so keyword lookup, keys, seq, assoc, conj and = all work through the regular core functions; the generated implementations are shared runtime functions imported only by files that use defrecord
    • Added a large set of core protocols so custom types participate in the standard functions: ILookup, IAssociative, IMap, ICounted, ICollection, IEquiv, ISet, the transient protocols, and IAtom/IDeref/IReset/ISwap/IWatchable (so a reagent-style reactive atom can be a plain deftype)
    • Compile-time namespace resolution: cljs.analyzer.api/resolve now sees vars of built-in library namespaces like clojure.string, plus :squint/compile-time forms and fixes for macro self-use
    • Full changelog

  • clj-kondo: static analyzer and linter for Clojure code that sparks joy.

    • NEW: macros from source. A defmacro (plus any supporting defn/defn-/def) tagged with {:clj-kondo/macroexpand-hook true} is automatically extracted into .clj-kondo/ and registered as a :macroexpand hook on the next run. See doc/hooks.md
    • Support for async/await in ClojureScript: bumped built-in CLJS analysis to 1.12.145 and added the :await-without-async-fn and :misplaced-async-metadata linters
    • #2822: NEW linter :alias-same-as-ns, warns when an alias equals the namespace it aliases (default :off) (@tomdl89)
    • #2807: NEW linter :conditional-build-up, warns on successive (if pred (assoc m ...) m) rebinding and suggests cond-> (default :off) (@walber-araujo)
    • #2062: NEW linter :if-x-x-y, suggests (or x y) instead of (if x x y) (default :off) (@jramosg)
    • #2818: fix :redefined-var false positive across files declaring the same namespace
    • #2814: fix :protocol-method-arity-mismatch false positive for definterface declaring the same method with multiple arities (@jramosg)
    • #2817: warn on recur inside a vector, map or set literal, since recur is never in tail position there
    • #2854: fix :invalid-arity false positive when an inner binding or fn param shadows a local function name (@yuhan0)
    • Performance work on the rewrite-clj parser and analysis internals: efficient get-in/select-keys, faster sexpr, leaner node allocation (@alexander-yakushev)
    • Deprecation notice: 2026.05.25 is the last release to include the clj-kondo LSP server and VS Code extension; use clojure-lsp instead, which embeds clj-kondo
    • Queued for the next release: early support for the Clojure 1.13 map destructuring keys (:keys!/:syms!/:strs!), including inferring required keys and reporting them at call sites (#2870)
    • Full changelog

  • SCI: Configurable Clojure/Script interpreter suitable for scripting

    • ClojureDart support, with a Flutter REPL example
    • Instance/static method and field overrides plus a :closed allowlist for :classes, giving fine-grained control over host interop; see the interop control docs. Also 1.6x faster instance-method interop on babashka
    • Security fix (sandbox escape): a string type-hint (e.g. ^"some.Class" x) bypassed the :classes allowlist, loading and static-initializing any class on the classpath at analysis time. Only affects sandboxing of untrusted code via :classes; upgrade to 0.13.53
    • Add an :interrupt-fn option: a zero-arg function called on every interpreted fn entry, so host code can interrupt or cancel a running SCI eval (thanks @whilo)
    • Add sci.interrupt/interrupt! to throw an interrupt that sandboxed try/catch cannot catch, and gate finally and the regex functions (re-matches/re-find/re-seq, JVM) through :interrupt-fn too, closing off ways to mask an interrupt and escape the sandbox #1044
    • Fix copy-var incorrectly marking a function as inlined when its unqualified name collided with a clojure.core/cljs.core inlined var (e.g. a custom get), silently breaking with-redefs (@verberktstan)
    • Fix cross-namespace defrecord/deftype type symbol resolution via alias (e.g. (instance? r/Foo x)), fixing nbb#410
    • Fix a self-require (a namespace requiring itself) being reported as a cyclic load dependency
    • Full changelog

  • fs: file system utility library for Clojure

    • Released 0.5.34 with Node.js support (#265): fs now runs on Node.js via ClojureScript and Squint / JavaScript, published as the @babashka/fs npm package. Most functions are supported. The JVM behavior is the reference implementation so all operations are synchronous, and the glob syntax is reimplemented from scratch to match the JVM. File times are BigInt nanoseconds to preserve sub-millisecond precision. zip is left out since Node.js has no native support for it
    • Added spit and slurp on both the JVM and Node.js
    • exec-paths returns [] when PATH is unset or blank instead of throwing
    • @lread did a thorough review pass making the return values of copy, copy-tree, delete-tree, zip/unzip, gunzip and the setters explicit and documented/tested (#197)

  • Babashka: native, fast starting Clojure interpreter for scripting.

    • Working towards a new release integrating all the newest updates in Babashka CLI and babashka.fs. Most importantly I&aposm working on autocompletions added for tasks defined in bb.edn.
    • #1979: fix with-redefs on copied vars (e.g. org.httpkit.client/get) incorrectly treated as inlined
    • Add org.jline.keymap.BindingReader for reading key bindings in terminal applications, completing the input side of the bundled JLine API
    • #1982: add clojure.lang.ChunkedCons, clojure.lang.APersistentVector$SubVector, clojure.lang.ArraySeq, clojure.lang.PersistentVector$ChunkedSeq, java.util.AbstractCollection and java.util.Queue to :instance-checks (@paintparty)
    • Added a terminal tetris example (examples/tetris.clj) built on JLine&aposs Display and AttributedString, showing off the new terminal APIs
    • Full changelog

  • Reagami: A minimal zero-deps Reagent-like for Squint and CLJS

    • Added keyed reconciliation (#40): support :key on children for stable node identity, so diffing reuses nodes instead of recreating them
    • Fixed CLJS :lite-mode compatibility and added it to CI (#41)
    • Added a benchmarks page comparing reagami against CLJS React wrappers and React-free solutions, with mermaid charts to visualize the results (#42, #43)
    • Expanded the README with an ADR on the unkeyed reconciliation algorithm

  • Cream: Clojure + GraalVM Crema native binary

    • I was finally able to reproduce an issue with core.async and filed this upstream
    • Once this is fixed I&aposm going to consider crema more seriously and play with the thought that this could be a substrate for "Babashka next".

  • html: Html generation library inspired by squint&aposs html tag

    • Fixed inline style maps emitting a literal \n between declarations via pr-str, which produced invalid CSS and dropped every declaration after the first (@cycl1st)
    • Only render a map attribute value as CSS when the key is style; other map-like values (e.g. records) now render via str (@telekid)
    • Fixed a symbol-valued attribute resolving to its runtime value instead of its literal name

  • Edamame: configurable EDN and Clojure parser with location metadata and more

    • Added ClojureDart support (non-indexing plain readers matching tools.reader, zero-literal parsing fix, and more)
    • With :auto-resolve-ns, bare syntax-quoted symbols now resolve to the current namespace, matching Clojure&aposs behavior

  • Neil: A CLI to add common aliases and features to deps.edn-based projects

    • #261: neil dep upgrade now upgrades unstable deps (e.g. release candidates) to a newer unstable version when no newer stable version exists
    • Added a README note on brew trust for users who installed neil before Homebrew introduced tap trust
    • The next neil version will make use of the new Babashka CLI features which is already prepared in a PR

  • Nbb: Scripting in Clojure on Node.js using SCI

    • #410: fixed a regression, introduced by the async/await work in #408, where a defrecord/deftype type symbol referenced through a namespace alias (e.g. (instance? r/Foo x)) failed to resolve

  • deps.clj: a faithful port of the clojure CLI bash script to Clojure

    • As always, catching up with the most recent Clojure CLI versions

  • Pod-babashka-gozxing: a babashka pod for QR code and barcode decoding/encoding, backed by gozxing

    • Initial release 0.0.1, installable via the pod registry

  • Graal-build-time: initialize Clojure classes at build time for GraalVM native-image

    • #55: munge package names for namespaces with special characters

Contributions to third party projects:

  • ClojureScript: documented the async/await support from last cycle on the ClojureScript site, including an enhanced reference (#423, #424)
  • Nexus: a data-driven state management library by Christian Johansen. I ported the core engine and test suite to run under squint and added a cljs test runner alongside the existing kaocha setup, so both babashka and squint stay covered in CI (#15, #16, merged)
  • Replicant: a data-driven DOM rendering library by Christian Johansen. I made Replicant itself run under Squint (converting dom.cljs to .cljc, adjusting core.cljc for portability), added babashka/squint test runners and wired them into CI, and fixed a multi-root render bug under squint by switching DOM state tracking to a node-map (#71, #72, merged)

Other projects

These are (some of the) other projects I&aposm involved with but little to no activity happened in the past two months.

Click for more details

Permalink

Clojurists Together Update: May and June 2026

Some of you might know that Clojurists Together are supporting my work on nREPL, CIDER and friends this year. Normally I send them a bi-monthly progress report, but I saw some other people who got funding for their OSS work publish those reports as blog posts for the broader public and I thought to try this for a change.

The past two months were super productive. I had a lot of inspiration during this period and I managed to tackle a lot of long-standing ideas and issues across the entire nREPL/CIDER ecosystem. Funnily enough, I also managed to grow the ecosystem with a couple of brand new projects, but more about those later.

The big highlights from my perspective:

  • CIDER 1.22 is out
  • CIDER 2.0 is essentially ready and needs more user testing
  • Sayid is reborn
  • Two brand new projects saw the light of day: port and neat
  • Piggieback 0.7.0 is out (and Weasel got modernized while I was in the area)
  • clj-refactor and refactor-nrepl got some love as well

Below you’ll find more details about the work I did, project by project.

CIDER

CIDER 1.22 (“São Miguel”) landed in mid-June, wrapping up the 1.x series. Its main features:

  • a registry for jack-in tools, so third parties can plug new build tools and Clojure dialects into cider-jack-in
  • a “default session” escape hatch from sesman’s project-based dispatch
  • keyword-argument versions of the low-level request APIs, alongside a proper decoupling of the nREPL client layer from CIDER’s UI

It also fixed a long list of small annoyances: severe editor lag in unlinked buffers, several TRAMP and SSH tunnel problems, request id leaks, and a bunch of broken menu entries.

Right after that I switched the development version to 2.0 and most of the planned work is already done. The headline items so far:

That last one deserves a special mention: evaluation results that are images now render inline out of the box, and file/URL results offer their content on demand, six years after the feature had to be disabled over its safety problems. There was also a big cleanup pass: consolidated configuration options, the REPL history browser renamed to cider-history to end a long-standing naming clash, theme-aware faces instead of hardcoded colors, refreshed docs and a regenerated refcard. CIDER 2.0 is available from MELPA snapshots and I’d love for more people to take it for a spin before the final release.

cider-nrepl

Lots of cider-nrepl releases, driving the CIDER work above:

  • cider-nrepl 0.60.0 added the ops backing the new protocol exploration commands (cider/who-implements, cider/type-protocols, cider/protocols-with-method).
  • cider-nrepl 0.61.0 brought ClojureScript test support, a ClojureScript macroexpansion fix, formatting that honors the project’s cljfmt configuration, and a pprint backed by orchard.pp.
  • cider-nrepl 0.62.0-alpha1 and 0.62.0-alpha2 hardened the content-type and slurp middleware (URL scheme allowlist, size caps, graceful fetch errors) and cleaned up the response protocol, which is what made it safe to turn rich content on by default in CIDER 2.0.

Along the way the project’s build was migrated from Leiningen to tools.deps, which required a new MrAnderson release (see the blog posts below).

Orchard

Orchard, the library that powers much of cider-nrepl’s functionality, kept pace:

  • Orchard 0.42.0 and Orchard 0.43.0 continued the inspector polish, added symbol classification to orchard.meta, a programmatic listener API for the tracer, and protocol/multimethod introspection in orchard.xref. The project also moved to tools.deps and its CI now covers JDK 26.

Sayid

Sayid, the omniscient Clojure debugger, had been dormant for years and I finally gave it the revival it deserved:

  • Sayid 0.2.0 was the big modernization pass: new mx.cider/sayid coordinates, a documented nREPL middleware API, a consolidated op surface (37 ops down to 26) and fixes for the most annoying Emacs client breakages.
  • Sayid 0.3.0 followed with usability work: no more frozen Emacs during the reload workflow, simpler query commands and help buffers generated from the keymaps.

port

port is a brand new project I started in May: a minimalist Clojure interactive programming environment for Emacs, built on prepl instead of nREPL. It went from nothing to three releases in the course of the month:

  • port 0.1.0
  • port 0.2.0
  • port 0.3.0, which added eldoc with active argument highlighting, a wire-level message log for debugging and a roughly 10x speedup in handling large prepl responses.

I don’t have any particular plans for the future of this project - it was just something that I wanted to experiment with for a while. I see it as an interesting option for people looking for some middle ground between inf-clojure and CIDER.

neat

neat is the other new arrival: a small, language-agnostic nREPL client for Emacs. neat 0.1.0 has the essentials in place: a pure-elisp bencode codec, a comint-based REPL, and a source-buffer minor mode with eval, completion, eldoc, xref and doc lookup, tested against Clojure, Babashka and Basilisp. It’s early days, but it’s a nice testbed for exercising the nREPL protocol outside CIDER.

This project also means I’ve dropped any plans to try to make CIDER a language-agnostic development environment. Going forward CIDER will focus only on Clojure-like languages, and everything else will be covered by neat.

Piggieback and Weasel

The nREPL org saw some ClojureScript-flavored action:

  • Piggieback 0.6.2 and Piggieback 0.7.0. The 0.7.0 release makes load-file evaluate the editor’s buffer contents instead of re-reading from disk, tears down ClojureScript REPLs when their sessions close (no more leaked Node processes) and surfaces ClojureScript status in the describe response.
  • Weasel 0.8.0 modernized the WebSocket REPL: the client now uses the platform’s native WebSocket, so it runs in any modern JavaScript runtime (browsers, Node 22+, Deno, Bun, workers), and the minimum requirements moved to Clojure/ClojureScript 1.12.

I also backfilled proper GitHub releases for the historic tags of both projects, so their release history is finally browsable.

Improving the ClojureScript support in CIDER has long been a major objective for me, and these small changes were some initial steps in that direction.

refactor-nrepl and clj-refactor

refactor-nrepl got three releases: 3.12.0, 3.13.0 and 3.14.0, the last one making the AST-based indexing much faster and more reliable. clj-refactor.el received a round of maintenance on master as well, and will get a new release after I wrap up the work on CIDER 2.0.

I’m still pondering the future of both projects, as I plan to move the most useful refactor-nrepl features (those that don’t carry a lot of complexity) to CIDER and cider-nrepl eventually, and I’m not sure that the flagship AST-powered refactorings are very competitive these days (compared to clojure-lsp and static project-wide analysis a la clj-kondo in general).

I’ll write a bit more about this and I’d certainly appreciate more feedback from the users of clj-refactor on the subject. It’s funny that I’ve been maintaining the project for ages, but I’ve never really used it (mostly due to its brittleness in the past). I think I managed to address some of the biggest problems recently, but perhaps this happened too late and the project has lost its relevance by now.

Blog posts

I wrote a few articles related to the work above:

Wrapping up

Big thanks to Clojurists Together, Nubank and the other organizations and people supporting my Clojure OSS work! I love you and none of this would have happened without you. Sadly, the amount of financial support my projects receive has eroded massively over the past 4 years and I’ve kind of lost hope that this negative trend will eventually be reversed. It was never easy to maintain many popular OSS projects, but the job certainly hasn’t got any easier or more rewarding in recent years…

Overall, a super productive two months. Hopefully the next two are going to be just as productive, although I have to admit I’ve plucked most of the low-hanging fruit already. Then again, I’ve said this many times in the past, so one never knows…

Permalink

Clojure 1.13.0-alpha2

Clojure 1.13.0-alpha2 is now available! Find download and usage information on the Downloads page.

  • Fixes a build problem from new Maven plugins by reverting the change

Try it out

Update your deps.edn :deps with:

org.clojure/clojure {:mvn/version "1.13.0-alpha2"}

Start a REPL with the Clojure CLI (any version) with:

clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.13.0-alpha2"}}}

Permalink

Sayid Redux

I have a soft spot for Sayid - it’s one of the most ingenious Clojure tools ever built, and also one of the most neglected. It’s an omniscient debugger: instead of stopping your program at a breakpoint, it quietly records every call to the functions you’ve traced and lets you rummage through the recording afterwards. It’s the kind of thing you demo to people and watch their jaw drop. And it had been sitting practically unmaintained for the better part of a decade.

Here’s the awkward part: that neglect is largely on me. Bill Piel, Sayid’s original author, handed me the keys ages ago, and I’ve been… let’s say a less than exemplary steward. I’d merge the occasional patch to keep the lights on, but until very recently I’d done precious little to actually move the project forward.

The best time to maintain your open-source project was six years ago. The second best time is now.

– Ancient proverb, lightly adapted

So what finally lit a fire under me? Blame CIDER 2.0. I’ve been reworking CIDER’s debugging and tracing story, and at some point I sat down to make the built-in tracer smarter. A few hours in it hit me: nothing I could realistically bolt onto the built-in tracing would come anywhere close to what Sayid already does. So instead of building a worse Sayid, I dusted off the real one, gave it a good scrub, and here we are - Sayid 0.4.

What is Sayid, anyway?

For the uninitiated: Sayid records the arguments, return value, timing and full call tree of the functions you trace, so you can go back and inspect exactly what happened - no breakpoints, no println, no re-running the thing five times.

While we’re on the subject of embarrassing confessions: I’ve been the maintainer of this thing for years and I still have no idea what “Sayid” actually means or what it’s a reference to. If you happen to know, please, put me out of my misery - I’d love to finally get the joke.

Trace a namespace, run your code, and pop open the workspace with C-c s w:

▾ (demo.coins/can-afford? [:quarter :dime :nickel :penny] 45) => true
    (demo.coins/total-cents [:quarter :dime :nickel :penny]) => 45

can-afford? says true, but four coins worth 41 cents shouldn’t cover a 45-cent tab. Something’s off, and the bug lives inside total-cents. This is where Sayid really shines - flip on an inner trace and it records every expression inside the function:

▾ (demo.coins/can-afford? [:quarter :dime :nickel :penny] 45) => true
  ▾ (demo.coins/total-cents [:quarter :dime :nickel :penny]) => 45
    ▾ (apply + (map coin-values coins)) => 45
        (map coin-values coins) => (25 10 5 5)

There it is, staring right at us: (25 10 5 5), when the last value should be 1. A penny is worth five cents in our coin map. We never wrote a single println, and we never had to guess where to put a breakpoint - we just looked at what actually happened.

That’s the pitch, and it’s a great one. So why was such a cool tool gathering dust?

Why an omniscient debugger, in 2026?

Honestly, part of it is that Clojure folks are spoiled. We have the REPL, so the reflex for most of us is to just re-run a form with some tap> or println sprinkled in and eyeball the output. That works right up until it doesn’t - until the bug is three layers deep in a map over a lazy seq, or only shows up on the 400th call, or lives in code you didn’t write and don’t feel like instrumenting by hand.

A traditional stepping debugger stops the world and makes you drive. Sayid does the opposite: it lets the program run to completion and then hands you the entire execution history as something you can navigate at your own pace. tools.trace is in the same spirit, but it dumps text to stdout - Sayid keeps structured data and gives you a query DSL to slice it. That’s a much better fit for how we actually work in Clojure: run it, capture everything, explore the data.

And here’s the thing that made me want to revive it rather than reinvent it - “capture everything as data and explore it later” is exactly the workflow that’s becoming more relevant, not less. Structured execution traces are gold, whether the thing doing the exploring is you, a data-inspection tool like Portal, or an AI assistant trying to understand why your code misbehaved.

Out with the old coordinates

First order of business was dragging the project into the present.

Sayid used to live under com.billpiel/sayid on Clojars, with namespaces like com.billpiel.sayid.core. The new home is clojure-emacs, so the artifact is now published as mx.cider/sayid:

{:user {:plugins [[mx.cider/sayid "0.4.0"]]}}

I also dropped the personal-domain prefix from every namespace - it’s plain sayid.core, sayid.trace, sayid.nrepl-middleware and so on now. The old com.billpiel/sayid coordinates still get the same releases for the time being, so nobody’s dependency breaks overnight, but the future is mx.cider.

Data first, strings later

Here’s the change I’m most excited about. Sayid’s nREPL middleware used to hand the Emacs client a pre-rendered blob of text plus a pile of text properties for colouring. In other words, the server did all the rendering and the client was a dumb terminal. That single decision is a big part of why there was exactly one Sayid client.

So the middleware now speaks data. There’s a family of new ops - sayid-get-workspace-data, sayid-query-data and friends - that return the recorded call tree as honest, navigable data instead of a wall of text:

{"id"       "4793"
 "name"     "demo.coins/can-afford?"
 "args"     ["[:quarter :dime :nickel :penny]" "45"]
 "return"   "true"
 "file"     "demo/coins.clj"
 "line"     12
 "children" [...]}

The structural bits (ids, names, timings, source location, the tree shape) come across as real nested maps and lists you can walk by key. The captured values are pr-str‘d, since an arbitrary Clojure value can’t always round-trip over the wire - but that’s the only place strings sneak in, and it’s exactly where you’d expect them.

The upshot: any editor or tool that speaks nREPL can now fetch a workspace and render it however it likes, and a REPL one-liner or a Portal tap gets you the same data with zero Sayid-specific machinery. The whole thing is written up in doc/nrepl-api.md, so you don’t have to reverse-engineer the wire format from the Emacs client the way you would have before.

A UI that doesn’t feel like 2015

With the data ops in place, I rebuilt the Emacs UI on top of them. The workspace and the “what’s traced” views are now proper foldable trees, built on CIDER’s new cider-tree-view. You get real folding (TAB), navigation (n/p), jump-to-source (RET), and - my favourite - c i hands the actual captured value straight to CIDER’s inspector, so you can drill into an argument or a return value as a live, navigable object rather than squinting at its printed form.

There’s also a query layer wired into the tree: f narrows to every recorded call of the function at point, i focuses a single call and its subtree. On a big trace that’s the difference between “wall of text” and “actually finding the thing”.

The best part is that the client is now smaller, not bigger - all the tree rendering, folding and value inspection are handled by mature components instead of bespoke code painting text properties by hand. That’s the payoff of moving rendering to the client: the server ships data, and the client is free to be as fancy or as plain as it wants.

Wait, you broke everything?

Backwards compatibility is a promise you make to the users you have. Sayid had exactly one, and reader, I am that user.

– Me, rationalising

Yeah, I did. I went pretty wild with the breaking changes this time around - new artifact coordinates, new namespaces, a reworked nREPL API, a bumped minimum CIDER version. Normally I’d bend over backwards to keep old clients working, but here I made a deliberate call: as far as I can tell, the bundled Emacs client was the only client Sayid ever had. Dancing around imaginary third parties to preserve compatibility nobody was relying on would have just made the project harder to adopt and harder to maintain.

So I opted for sweeping changes that leave Sayid in a much better place to build on, rather than a museum of backwards-compatible cruft. If it turns out I was wrong and you were quietly depending on the old coordinates - I’m sorry, and do let me know, because that’s genuinely useful information.

Take it for a spin

That’s the gist of it. Sayid is alive again, it’s leaner, it speaks data, and it has a UI I’m not embarrassed to demo. What I’d love now is for more people to actually use it and tell me whether the new direction resonates.

So please - [mx.cider/sayid "0.4.0"], trace something gnarly, and pop open the tree. Then head over to the issue tracker and tell me what you think: what feels great, what feels rough, what’s missing. I have ideas for where to take it next (bounding the recording so you can safely trace a whole namespace under a test suite is high on the list), but I’d rather steer by what people actually want out of it.

Big thanks to Bill Piel for building such a wonderful tool in the first place - I’m merely standing on the shoulders of a giant here. And thanks in advance to everyone willing to kick the tyres on the revival!

That’s all from me for now. Keep hacking!

P.S. One more thing…

Well, that didn’t take long. Remember that “high on the list” bit a few paragraphs up - bounding the recording so you can safely trace a whole namespace under a test suite? Turns out I couldn’t leave it alone. A short burst of small improvements after this post went up, and it’s done: Sayid 0.5.

That was the thing that made Sayid feel like a toy - point it at a real workload and it would cheerfully eat your whole heap and fall over. The recording now has a set of tunable bounds instead: a cap on how many top-level calls it keeps, a depth limit, 1-in-N sampling for hot paths, a per-function cap, and a keep-only-the-last-N mode for when what you care about is whatever happened right before things went sideways. Fat and infinite values no longer hang the data ops either, and the traced-functions view got its enable/disable/remove actions back.

The upshot for you: you can finally point Sayid at real code under real load without babysitting it. Same ask as before - give it a spin and tell me how it feels.

Permalink

Compilers, Editors, Typefaces and More From the Ground Up

In the four years since my previous progress-update article, I have been working on a series of experimental projects and learning about the low-level fundamentals of software—down to the finicky particulars of things like text rendering and x86 machine code generation.

Here is a brief, incomplete listing of these projects:

  • (Late 2022) Structural editor for Clojure, with code stored non-hierarchically in a database. (Early prototyping phase.)

  • (First half of 2023) Compiler, structural editor and debugger for a custom language with a bytecode interpreter and x64 machine code backend. The language is imperative, strongly typed and assumes manual memory management, but has a Clojure-like syntax.

  • (Mid 2023) ButteryTaskbar2: a reimplementation of my original utility for Windows, improved to be simpler and more efficient.

  • Basic GPU-powered 2D rendering via Direct3D 12.

  • OLKCH colour picker.

  • (Second half of 2023) Two different attempts at a structural editor for the Jai programming language.

  • A simple alternative language to Markdown.

  • A tool for viewing and editing notes using a graph-based structure similar to Roam Research. Notes are written in my Markdown alternative.

  • Static website generator using my Markdown alternative for webpage content.

  • A programmatic generator for my six-segment logo, using TinyVG for rendering.

  • XML parser.

  • Implementation of Myer’s and Dijkstra’s algorithms for diffing a source code AST, together with a fun (but vitally useful) visualisation of the diffing algorithm.

  • (First half of 2024) A text editor for code, featuring syntax highlighting, Tree-sitter integration, adjustable panels, Regular Expression text search, modal editing, macros and undo history.

  • libgrapheme ported to Jai.

  • (Second half of 2024) Compiler and bytecode interpreter for a custom modern C-like language.

  • (Late 2024) A simple bespoke tool for tracking deadlines and reminders. Uses SQLite and a minimal browser-based UI. I continue to use this tool to this day.

  • Win32 API bindings generator, which converts the JSON data from win32json into a serialised binary form, which is then selectively converted into code declarations.

  • (2025 onwards) A blank-slate code repository with a bespoke build system and custom implementations of core libraries for purposes like memory management, file system access, hash tables, etc. This is the basis of all projects that follow.

  • (Early 2025) A utility to provide various comforts on Windows, including navigation key bindings, synchronised display brightness control, and sticky window edges (like macOS).

  • A more ergonomic and flexible alternative to JSON.

  • Basic SIMD-accelerated software renderer written in assembly for AVX2.

  • (Mid 2025) Initial prototyping work on a new design of a structural code editor, inspired by the Kyra language and editor.

  • Glyph rasteriser (CPU-based), capable of directly rasterising both linear and quadratic edges. Inspired by the approach in stb_truetype, but designed to be SIMD-friendly.

  • A new programmatically-defined quasi-proportionally-spaced typeface designed to be both legible and fast to render, by minimising vertices and preferring quadratic Bézier curves over cubics. A simple bespoke text layout system replaces the dependence on Harfbuzz.

  • (Late 2025) Another attempt at a UI toolkit, which includes a more powerful layout system, better support for incremental updates, and dependency analysis to optimise GPU draw calls.

  • Vulkan bindings generator, which sources data from the official XML-based specification.

  • (2026) A revised text editor, bringing together my latest work on UI systems and text rendering. It is powerful enough that I now use it as my primary code editor for my Jai-based projects. Like 4coder, it implements virtual whitespace, which greatly improves the experience of working with code indentation.

All of these projects constitute an extensive volume of research and time spent in deep thought. This article does not aim to give exhaustive coverage of all the ideas and implementation details across these four years, but merely to serve as an overview of some of the most interesting points.

Reinventing Software Development Tooling

Recurring themes in my work include compilers, innovative code editors, and novel programming language design. I feel like there is opportunity to bring substantial improvements to the development experience by reducing superfluous sources of friction, cutting out bureaucratic busywork, and tightening feedback loops. My main points of focus have been:

  • Structural code editors to give a more refined editing experience with more powerful and robust code intelligence and transformations.

  • Rethinking code organisation to make it easier to scale, manage, and adapt complex codebases. This means storing code declarations in some kind of graph database instead of a tree of files.

  • A more interactive debugging experience leveraging tightly integrated code introspection along with dynamic code compilation and execution.

My research in this area is still early and highly experimental, though I now have a better idea of what probably doesn’t work, and what could work.

Whilst searching for prior art, I encountered this video which is the best demonstration I’ve found of a structural editor and its benefits.

2023H1 Programming System

I started off by implementing a programming language with a syntax very much like Clojure, because I was most comfortable with that at the time. The simplicity of Clojure’s syntax makes it very powerful, and makes it a particularly good fit for a structural code editor.

In addition to a structural editor, this project implements a primitive compiler that emits both bytecode and (to a limited extent) x64 machine code. Initially, I implemented a tree-walking interpreter, but that was awfully slow. The bytecode interpreter was a significant improvement, but still orders of magnitude slower than machine code. The x64 backend is capable of directly generating a valid Windows executable binary without needing to depend on an assembler. The most sophisticated program I tested is a program that counts the total number of lines of all text files in a directory.

I also implemented a primitive debugger that integrates with the bytecode interpreter. Unfortunately, I do not have any images of this and now I cannot easily compile the project. This debugger had a GUI that displayed a listing of the bytecode instructions as well as the current contents of virtual registers and stack memory. You could set breakpoints, step through the program, and watch the program state change in real time.

I was initially using Skia for rendering, but later switched to doing basic software rendering with FreeType and Harfbuzz handling text rasterisation and layout. In the bytecode interpreter, I used Dyncall for dynamically calling procedures in dynamically-loaded libraries.

2023H2 Programming System

My subsequent experiments involved building structural editors for the Jai programming language. The problem with a Lisp-like syntax is that its simplicity means it can be verbose when expressing the same amount of information that a statically-typed language needs to express. In such a statically-typed language like Jai, it pays off to have more complex syntax in return for brevity.

At first, I tried an editor with four types of node: token, string, block and newline. A block corresponds to a pair of brackets—(), [] or {}—and contains a list of nodes. However, this did not feel powerful enough, so I proceeded to implement an alternative design with a more heterogeneous AST that more closely reflects what the compiler is working with. For example, declarations, binary operations, if statements and procedures are each their own distinct type of node in the editor.

The result was a fun, visually unique demonstration, but it revealed a challenge in the design space of structural editors: increasing the heterogeneity of the editor AST greatly increases complexity and makes editing operations similarly heterogeneous. In order to create nodes of a certain type, my solution was to use special key bindings, text commands, automatically converting a typed-out keyword into its corresponding type of node. This does not feel as fluid as the experience of editing a uniform array of characters in a traditional text file. Manipulating the tree becomes more difficult since it requires a larger set of more specialised operations, which imposes a greater cognitive burden.

There are also questions like:

  • How specialised should the editor’s AST be?

  • Should you be able to insert AST nodes of a certain type into a slot that is invalid for its type?

  • Should you be allowed to partially select the contents of a node in the same selection that contains its siblings? How should selection ranges on a tree structure work?

Having done this experiment, I am now leaning towards a structure that is more uniform, but that needs further research.

Graphics Rendering

When implementing GUIs from scratch, there is a question of how to render graphical elements to the screen (after everything is laid out). A few options:

  • Use a library (like Skia or Blend2D) that abstracts away all the difficult parts.

  • Build a CPU-based software renderer.

  • Use the GPU by directly interfacing with an API like OpenGL, Direct3D or Vulkan.

Initially, I was using Skia because it is very capable, cross-platform, and was familiar to me. If I recall correctly, I was having performance issues rendering text in Skia. Almost certainly, this was because I was using it wrong and probably there was a lack of glyph caching, but I took this as a good excuse to write an extremely simple software renderer and handle text myself using FreeType and Harfbuzz.

My original software renderer was slow and I needed more performance, so I looked around. For a brief moment, I used Blend2D, a 2D CPU-based graphics rasterisation library, but performance was still not satisfactory. Furthermore, Blend2D is such a large, complex project that I would rather not depend on.

In 2023, I eventually gave in to GPU rendering and spend over two weeks learning how to use Direct3D 12 to produce the most basic 2D graphics. This was some serious drudgery, but in the end I had working shaders for drawing rectangles, images and glyph runs of text. OffsetAllocator was used for managing GPU heap memory. The speed gain from GPU rendering was especially apparent in the OKLCH colour picker I made.

Later, in 2025, I explored writing a CPU-based software renderer written in x86 assembly language. Because this used AVX2 instructions and could operate on eight pixels at a time, it was much faster than my prior software rendering attempt. I experimented with an implementation that operates on 8-byte integer colour channels, and another that operates on 16-byte floating-point colour channels for better blending accuracy (using special FP16 x86 SIMD instructions). My aim with this effort was to win some simplicity and avoid having to interface with a GPU graphics API which all have significant start-up latency.

This software renderer was initially inspired by the design in Handmade Hero, which approximates gamma correction by the square and square root operations. Unfortunately, this approximation results in a very noticeable loss of accuracy and attempting to do anything more accurate is prohibitively expensive. Even without the accurate gamma correction, the software render is still much slower than the GPU. Therefore, I had to conclude that GPU was the only feasible option for performant and accurate 2D graphics, which sadly meant enduring more of the misery of working with complex graphics APIs.

Approaching the end of 2025, my faith in the Windows operating system was nearing zero. I did not want to tie myself to this rapidly decaying platform if I could avoid it. Thus, I started learning Vulkan and used it to implement a basic 2D render. Vulkan Guide helped me get set up and I used the VulkanMemoryAllocator library because apparently it’s silly not to. HowToVulkan is a newer guide that I haven’t looked at but seems promising.

Vulkan has the benefit of giving you access to the latest features in GPU technology, whereas OpenGL is antiquated and better avoided nowadays. Vulkan is not fun to work with, but that’s where the future is headed, and that’s not a decision I get to make. The bindless APIs in the newer Vulkan versions makes things easier and enables substantial simplifications such as eliminating the need for a glyph atlas.

Designing a Typeface and Text Layout System

In designing UI systems, it has become clear to me that text rendering is by far the most complex and computationally-expensive part of rendering simple 2D UIs. This is because glyphs first have to be rasterised from a vector format into a bitmap image, then a potentially very complex set of rules need to be applied to compute the positions of each glyph in a string of text.

Glyph caching helps improves performance by saving the bitmap of a glyph, but the bitmap is only valid for a specific sub-pixel offset, and glyphs can be positioned at any non-integer coordinate. If you round the glyphs to the nearest pixel boundary, the text becomes noticeably unevenly spaced.

In my prior text rendering solution, I instead rounded each glyph to the nearest 1/3-pixel boundary which reduces the visual error, but requires up to three cached bitmaps per glyph.

Another solution I tried was to instead use a word cache: split up the text into short segments (words) and rasterise each word as a single bitmap image. This means that the spacing between glyphs in a single word is perfect, at the expense of more memory spent on storing bitmaps.

But I wanted something better. Following good engineering principles, I considered my specific situation and considered—from first principles—what an optimal text rendering design would look like. Here are some of the constraints and assumptions I came with:

  • Support only for left-to-right scripts like Latin, Cyrillic and Greek.

  • Glyphs must always be pixel-aligned (so there is one bitmap per glyph with no need for sub-pixel variants).

  • Support for characters of various widths (not just monospaced typefaces).

  • Kerning is applied to each pair of graphemes, with no further context allowed.

  • Font size is determined by cap height which must be an integer.

  • No sub-pixel anti-aliasing like ClearType; it complicates colour blending, only works well in limited cases, and results in colour fringing. I would prefer using high-DPI displays so that sub-pixel anti-aliasing can be a forgotten artefact of the past.

By constraining the problem I am attempting to solve, I can search for opportunities to make the targeted solution significantly superior to general solutions like TrueType and Harfbuzz. And indeed, I concluded that I would be better off throwing out TrueType/OpenType, Harfbuzz (or equivalent) and FreeType (or equivalent).

Firstly, I needed a glyph rasteriser to replace FreeType. The goal of this is to have a simple piece of code that does exactly what I need in a way I control. I learned about the mathematics of glyph rasterisation, giving careful consideration to floating-point precision limitations, and proceeded to implement a rasteriser in less than 800 lines. This rasteriser directly supports quadratic Bézier curves, unlike stb_truetype, which has to approximate such curves by a series of straight line segments.

Secondly, I needed a typeface. Since I had chosen to reject the design of TrueType, I had shut myself out of the ecosystem of pre-existing typefaces—I had to now create my own. This process took much longer than I initially anticipated, and so I spent months swimming a laborious mathematical soup of twists, turns, and dead ends.

TrueType fonts work by representing each glyph as a collection of contours, where each contour consists of segments of either quadratic Bézier curves or straight lines (which is just a degenerate case of a quadratic Bézier curve). It gets further complicated by font hinting, which requires the use of an interpreter to read TrueType bytecode instructions which make adjustments to the vertices for better alignment with the pixel boundaries at a certain font size.

I also wanted to support hinting (where the glyph shape is optimised for the pixel boundaries at a certain size) but I wanted to implement it in a simpler way. Instead of representing my typeface in a bespoke data format (something like a ttf or otf file), I decided that my typeface would be defined in executable code for maximum flexibility. With this design, the process of obtaining a set of glyph contours involves calling a function with three key parameters: the typeface, the font size, and the glyph number. The code then directly computes the optimal glyph vectors depending on the font size, without needing a complicated bytecode interpreter. It’s all just regular imperative code.

Two main principles informed the design of my typeface:

  • Legibility: characters should be reasonably distinct and readable, even at rather small font sizes.

  • Performance: there should be minimal computational cost involved in generating the glyph vectors and rasterising them. This meant that I aimed to use as few curve segments as possible, including minimising the use of cubic Bézier curves (which must be broken up into quadratics prior to rasterisation).

To make the glyphs look as good as possible, I studied quite a bit of mathematical theory as I needed to be able to generate curves that ideally maintain G2 continuity, but these curves need to be cheap to generate. For weeks, I chased after mathematical derivations in SageMath trying to find formulae for computing curves that satisfied various invariants. This turned out to be much less fruitful that I had hoped; often, the sets of equations would be too complex to solve, or no solution would exist at all. I had also taken plenty of time to check the prior research, including the work done by Raph Levien, but I wanted something as simple as possible and much of the research involved mathematical expressions that lack an analytical solution and thus are not trivial to compute (which would violate my performance constraints).

Eventually, I completed the designs for the ASCII range of characters. You may think, as I did, that the small range of ASCII characters wouldn’t take that much work to design. But I was waiting for the day that it would end, and I felt like it would never come.

Putting this all together into a text rendering system, I was definitely surprised by how well the result turned out. The text rendering system is now much simpler, and I can now comfortably use my own proportionally-spaced font in my own code editor.

From Clojure to Jai

This section presents some of my reasoning as to why my style of programming dramatically shifted away from the dynamic, functional, GC-powered land of Clojure and towards embracing a more imperative, statically-typed, systems-level kind of programming.

Clojure Is Too Slow, and Performance Matters

Having become reasonably experienced using the Clojure programming language for over two years, I repeatedly found myself running into the same insurmountable obstacle: performance. Put simply, Clojure is too inefficient for high-performance software, and the program start-up time is unacceptable. I have spent a lot of time optimising performance in Clojure programs but nothing is ever enough.

In my previous article, I discussed my attempt at building a compiler for a custom JVM-based language which aimed to be more performant, but eventually I had to face reality: the Java Virtual Machine is too slow and limiting.

There is a good reason the JVM has not taken over the world of performance-critical software (operating systems, codecs, game engines). Even in cases where performance is less critical, choosing to build your software with inefficient technology like the JVM still does a disservice to your users, since the result is often:

  • Less responsive and more sluggish (even for simple UIs, the slightest difference matters).

  • Less power efficient (especially important for laptops).

  • More complex (larger binary sizes, possibly requiring a separate JVM installation).

To illustrate the importance of performance, if a compiler takes two seconds to compile instead of one, the added delay further breaks the flow of development and decreases dampens one’s spirit. Billions of people use software every day, so each minor delay quickly adds up to an unfathomably great waste of human hours spent waiting for software to respond.

Performance also affects the user’s pattern of behaviour. For example, the slow file search in Windows File Explorer means that I never use that feature. If the performance of the search were much better, then superior workflows could be enabled that take advantage of that search feature. This is not merely hypothetical: nowadays, with FilePilot, I use text search to rapidly navigate the file system all the time.

A common dogma in computer programming is that you should not be optimising for performance until you know your performance requirements and bottlenecks. To a large extent that is true, but it is not an excuse for things that are slow for no good reason; these things, polluted with junk, can be made faster without sacrificing a good programming experience, and in fact the programmer’s experience will often be improved by prioritising performance and simplicity.

As soon as you choose to use something like the JVM, you have already accepted a significant performance and memory penalty. If people don’t think about performance from the beginning, we end up with a proliferation of slow software (the world we live in now). You may take steps to optimise the program (which naturally makes it more brittle), but you will always have the foundational issue of running on a suboptimal, managed VM.

Programs should be fast by default—and the language should push the programmer in the direction of something that is not horribly inefficient. At the very least, there should be a pathway to transform the program into something that makes reasonably efficient use of the hardware.

Also, if you are developing a library, you should assume highly strict performance criteria, since you do not know in which contexts your library may be used. Pouring a huge volume of effort into a fundamentally slow technology may waste the time of your users. But it will also cut off those users for whom the library is too slow, thus their time is wasted by not being able to do what they want to do without reimplementing the library features.

In the case of a desktop applications, it is imperative that they be fast and responsive, out of respect of the user. It should be normal that it be fast and responsive, without any special effort. Computers are incredibly fast—fast enough to drive interactive 3D worlds with complicated graphics—yet even simple programs struggle to meet reasonable performance expectations. Even if you do make the program feel responsive, its inefficiencies can still impact the user experience. For instance, you may consume an absurd amount of memory that reduces the user’s multitasking potential. Additionally, through excessive CPU usage, you could bring real discomfort to the user due to increased fan noise.

Performance aside, the JVM’s class loader system was difficult to work with, and it was impossible to do the things I wanted in a non-clunky way. The JVM is drowned in unnecessary complexity, which I found apparent when working with its bytecode and flawed object model.

I think good engineering is important, and that means valuing efficient and simple solutions. The JVM is not that, and nor is it something that can be fixed; its problems stretch down to the roots.

Finding a Good Programming Language

At the dawn of 2023, I departed from the sweet comforts of JVM-enforced memory safety and escaped into the wilderness of systems-level languages with manual memory management.

I had briefly tried Rust in late 2021, but never again. The misguided design of this language creates tremendous friction and imposes artificial puzzles that get in the way of solving the actual problem at hand. The costs outweigh the benefits, at least in most cases, and the compile times are torturous.

So, I began to learn Zig as I set out to create a new compiler. However, it didn’t take long before my frustration with Zig’s language design reached a critical threshold. For instance, the Zig compiler treats any unused variable as an error, and that is just not the way I want to program.

I promptly adopted Odin, which has a far superior design philosophy. There is still a lot I don’t like about it—like the restrictive namespacing system—but it worked well enough for my compiler, interpreter and debugger project of the first half of 2023. Eventually, the growing project size (albeit still modest) meant that the LLVM compile times were unfortunately becoming a significant source of friction.

In the middle of 2023, I began using the Jai programming language. It is, by an order of magnitude, more well-designed and powerful than any other language in its class, thanks to a unique combination of fast compile times, powerful polymorphism, meta-programming powered by a bytecode interpreter, helpful error messages and ergonomic syntax.

Despite being rather complex, Jai actually feels very simple, uniform, and almost Lisp-like in its design. Unlike other languages, Jai spends it complexity budget wisely—in the places where the added complexity pays off the most. I am using Jai to this day and that is not changing any time soon.

But the Libraries! The Ecosystem!

One of the benefits of being tapped into the Clojure and JVM ecosystem is the access to a vast range of software libraries. However, I value that much less now than I used to in the past.

Building upon layers of abstraction can often be harmful due to the additional constraints of that abstraction. The API of a library may push you towards a design that is not optimal for the problem you are trying to solve. Also, libraries are usually designed to support a variety of use-cases, which probably means that there is more abstraction (and more features) than you truly need.

Often, I have run into trouble when my specific use-case does not fit the exact mould of the library. As a result, when going to do something specific, the library’s design may outright prevent me from doing what I want without having to fork and modify the source code—an undesirable solution that partially defeats the point of using a library in the first place.

Furthermore, I don’t tend to use a large number of third-party libraries, and many of the ones I do use could be feasibly reimplemented myself. I’d say writing that extra code is worth it in many cases, especially where the library in question is simple or only a small subset of it is used. As a bonus, the code you write is tailored to your specific needs.

In any case, I’m of the philosophy that you shouldn’t be afraid to throw away code often. An implication of this is that code should be easy to write and refactor, and a reason for this practice is that the more you work on a project, the better of an idea you get about what your program should ultimately look like. Rather than being held back by old, uninformed code, you should have the freedom to rebuild substantial pieces from a simpler foundation backed by experience. A library is analogous to the old informed code you wrote before you fully understood your needs.

Remember: Source code you don’t own or understand is a liability; it’s good to keep things minimal.

Dynamic Code Execution

For me, the most compelling feature of the JVM was the ability to dynamically modify and inspect the live program. That said, with some work, simple hot code loading can be implemented in an unmanaged language. By not having to deal with the class loader system, it is possible to design a programming language with a much better system for dynamically loading code, tailored and fine-tuned according to how I want it. In fact, it may turn out that the simpler solution overall is to implement this stuff from scratch and bypass the complexity put forth by the JVM.

Another consideration: how dynamic do you need to be? By using self-describing objects everywhere, the JVM is highly dynamic, but at the great expense of performance. Is that trade-off worth it? I don’t think so, after having gained experience in modern unmanaged languages.

I believe it is preferable to opt-in to these dynamic runtime features in the places and times you need them. Ideally, no changes to the source code should be necessary to compile a more dynamic and introspective version of the program compared to a completely static version. Additionally, in the spirit of Mouldable Development, it should be cheap to create ad-hoc tools for live visualisation, debugging, and modification of the program. The video games industry has some good examples of bespoke in-house tooling.

Conclusion

At a very bird’s-eye view, I have presented some hopefully interesting aspects of my projects. This skips over many details like the mathematics involved in typeface design as well as entire topics including the compiler implementations. Also, I haven’t touched on foundational systems such as the memory allocators and the arena allocator system that I use throughout my codebase. There is also a lot yet to talk about relating to UI event handling, caching, and layout. Maybe another day.

To wrap up, I would like to emphasise a piece of truth that I repeatedly discovered as I set down these paths of learning unfamiliar topics: the low-level details are not as frightening as they may seem at first.

  • When I was writing Clojure, the idea of working with JVM bytecode seemed daunting. Then, I spent time learning about it and found it was not so difficult after all.

  • Coming from garbage-collected languages, the idea of using a language with manual memory management sounded very tricky and error-prone, but it’s not actually that bad with the right techniques.

  • Learning assembly language and manually generating x64 machine code was intimidating at first, but it just takes perseverance to read through the vast Intel documentation before being able to create something simple that works. Assembly language itself is actually very simple to understand.

  • Learning to use a graphics API, particularly one like D3D12 or Vulkan, seemed like a monumental task due to the vast amount of prerequisite knowledge. And indeed it is, because there is no happiness to be found in modern graphics APIs. However, once you dip a foot in and start learning, you start to pick things up, things get more familiar, and thus things get a bit easier.

Not everything is like this—some things are just really difficult and may even take a lifetime to master. For instance, I’ve made no attempt at an optimising compiler because there is no limit to how complicated that gets when taken to the fullest extent (see LLVM). Also, when working with modern graphics APIs like D3D12, even industry professionals are prone to make subtle memory management mistakes thanks to the overly convoluted nature of these APIs.

But for many things, all it takes is the initial step of dipping your toes in the water, then as you begin uncovering new territory, everything becomes a lot clearer and less daunting.

Thanks for reading. To possibly receive future updates, you may choose to subscribe via email.

Permalink

From Prototype to Production: The Business Case for Chachaml Part 3

This is the final article in our series, Clojure Meets Production MLOps: How chachaml Delivers AI-Native Workflows.

If you’re joining here, you may want to start with the earlier posts:

In this final article, we’ll focus on the business side. We’ll look at what adopting chachaml means for engineering teams, how it helps shorten the path from prototype to production, and why that matters when you’re building AI systems that need to last.

Real-World Use Case #1: End-to-End Model Development Inside a Clojure System.

A team builds a recommendation engine for a Clojure app. It uses data from their existing databases and services. That data moves through feature pipelines and into model training.

With chachaml, every training run is tracked automatically. Everything—parameters, metrics, artifacts, and results—is gathered in one place, making it easy to review and contrast your experiments. 

Once the model looks good, the team registers it, handles versioning, and promotes it through the model registry. After that, deployment is just part of the flow, and they monitor it performance from there. 

The whole process happens in a single environment: 

Data Ingestion → Training → Tracking → Registry → Deployment

Example: End-to-end run in a single Clojure form

;; 30-second quickstart — full end-to-end tracking
(ml/with-run {:experiment "quickstart"}
  (ml/log-params {:lr 0.01 :epochs 50})
  (ml/log-metric :accuracy 0.94)
  (ml/log-artifact "model" {:weights [1.0 2.0] :bias 0.3}))

There’s no need to piece together multiple tools or move between different systems.

Benefits

  • Single technology stack.
  • Less operational complexity.
  • Faster iteration.
  • Easier team collaboration.
  • Better visibility across the ML lifecycle.

For teams already using Clojure, this makes it much easier to take machine learning projects from development to production.

Real-World Use Case #2: Hybrid Python + Clojure ML Infrastructure.

For machine learning, many teams don’t use just one language.

A common setup is a Python data science team working alongside a Clojure production platform.

Data scientists train and evaluate models in Python. The production application runs in Clojure.

Chachaml helps connect both sides.

Teams can send runs, metrics, and artifacts using simple HTTP APIs. 

If they are working in Python, they can connect with chachaml using scikit-learn or libpython-clj2. All artifacts are stored in a shared S3-compatible storage, so everyone—across different teams and environments—can access what they require.

Here’s the general flow: 

Python Training → chachaml Tracking → Shared Artifact Storage → Clojure Production Platform

Example: Hybrid Python + Clojure workflow

;; Python side: push run via HTTP (shown above)
;; Clojure side: load artifact and serve
(let [model (reg/load-model "iris-classifier")]
  (predict model new-features))

;; Shared S3 artifact storage available to both sides
(artifact/store! run-id "model" model {:backend :s3})

This setup comes with some real perks: 

  • Python tooling for data science.
  • Clojure for production systems.
  • Shared experiment tracking.
  • Centralized artifact management.
  • Less duplication across teams.
  • A consistent view of the ML lifecycle.

Teams don’t have to replace their preferred tools; all components are centralized, making ML operations much easier to handle. 

What This Means for CTOs and Engineering Leaders

➜ Strategic Benefits

For engineering leaders, machine learning isn’t just about building models anymore—it’s become all about keeping operations running smoothly. 

These days, as teams adopt more AI systems, they manage a bunch of tools, workflows, and deployment setups. Before the team knows it, every aspect quickly becomes complicated and hard to handle. 

A unified approach changes the game. 

  • Decrease tool sprawl.
  • Improve governance.
  • Increase reproducibility.
  • Lower operational complexity.

And as machine learning becomes part of core business systems, those benefits become more important.

➜ Evaluating Long-Term ML Platform Decisions

Choosing an MLOps platform is a long-term architecture decision. It guides teams on building and deploying models. It also helps them with how they run and troubleshoot them. 

FactorPython-Centricchachaml + Clojure
Stack consistencyLowHigh
Operational complexityHighLower
REPL workflowLimitedNative
JVM integrationModerateExcellent

If a team already works with Clojure and the JVM, sticking with that ecosystem reduces complexity. 

Teams can reduce the challenges of managing multiple platforms, so workflows move faster and the infrastructure remains less complex. 

The idea isn’t to avoid Python completely—it’s mainly about simplifying workflows by using one solid platform instead of many. 

Flexiana’s View on the Future of MLOps in Clojure

➜ The Ecosystem Is Entering a New Phase

Machine learning in Clojure isn’t just an experiment anymore. It used to be all about building models, managing data, and doing research, but that’s changed. 

Now, more teams are actually putting machine learning systems in place and handling everything that comes with keeping them up and running. 

AI is emerging in everyday business tools, too, which means teams need better ways to track experimental progress, manage their models, assess results, and collaborate across projects. 

As adoption grows, operational tooling becomes just as important as model development. That’s usually a sign that an ecosystem is maturing.

➜ Why chachaml Matters Beyond One Library

chachaml is part of a larger shift in the Clojure ecosystem.

The challenge for most organizations is no longer training a model. It’s managing machine learning systems over time. That requires tracking, governance, deployment workflows, monitoring, and collaboration.

These are operational problems, and they need operational tools.

At Flexiana, we’ve spent years building production systems, working with machine learning workloads in Clojure, and helping teams manage hybrid ML environments that combine Clojure services with Python-based tooling. Those experiences highlighted a common gap: Teams could build models but lacked the infrastructure to run them in production.

As machine learning becomes a standard part of software architecture, Clojure teams will need tools that support the full lifecycle of those systems.

That’s why chachaml matters. It’s not only about machine learning. It brings production ML to Clojure with lasting infrastructure. 

To Sum Up

chachaml really steps up Clojure’s MLOps game. This platform puts everything in one place. Teams can track their experiments, manage pipelines, monitor models, and stay on top of everything—all without the usual hassle. Collaboration actually feels easy here. 

The built-in MCP support is a solid perk, as it lets AI agents work directly with the ML data and operations.

At Flexiana, we’re excited about this. It feels like a big step toward making machine learning in Clojure more reliable and ready for serious use. 

Get started with chachaml. Talk to Flexiana.

Review the source code, documentation, and examples on GitHub to see how chachaml works in practice. Connect the chachaml MCP server to tools like Claude Code or Continue and explore machine learning data through natural language queries. 

If you’re evaluating machine learning infrastructure or planning a production ML system, we’d be happy to talk.

We help teams with:

  • Clojure ML architecture
  • Production MLOps platforms
  • Hybrid JVM/Python systems
  • AI infrastructure strategy
  • Enterprise software development

Whether you’re building a new ML platform or improving an existing one, we can help you design an approach that fits your systems, workflows, and long-term goals.

The post From Prototype to Production: The Business Case for Chachaml Part 3 appeared first on Flexiana.

Permalink

I am sorry, but everyone is getting syntax highlighting wrong

Translations: Russian

Syntax highlighting is a tool. It can help you read code faster. Find things quicker. Orient yourself in a large file.

Like any tool, it can be used correctly or incorrectly. Let’s see how to use syntax highlighting to help you work.

Christmas Lights Diarrhea

Most color themes have a unique bright color for literally everything: one for variables, another for language keywords, constants, punctuation, functions, classes, calls, comments, etc.

Sometimes it gets so bad one can’t see the base text color: everything is highlighted. What’s the base text color here?

The problem with that is, if everything is highlighted, nothing stands out. Your eye adapts and considers it a new norm: everything is bright and shiny, and instead of getting separated, it all blends together.

Here’s a quick test. Try to find the function definition here:

and here:

See what I mean?

So yeah, unfortunately, you can’t just highlight everything. You have to make decisions: what is more important, what is less. What should stand out, what shouldn’t.

Highlighting everything is like assigning “top priority” to every task in Linear. It only works if most of the tasks have lesser priorities.

If everything is highlighted, nothing is highlighted.

Enough colors to remember

There are two main use-cases you want your color theme to address:

  1. Look at something and tell what it is by its color (you can tell by reading text, yes, but why do you need syntax highlighting then?)
  2. Search for something. You want to know what to look for (which color).

1 is a direct index lookup: color → type of thing.

2 is a reverse lookup: type of thing → color.

Truth is, most people don’t do these lookups at all. They might think they do, but in reality, they don’t.

Let me illustrate. Before:

After:

Can you see it? I misspelled return for retunr and its color switched from red to purple.

I can’t.

Here’s another test. Close your eyes (not yet! Finish this sentence first) and try to remember what color your color theme uses for class names?

Can you?

If the answer for both questions is “no”, then your color theme is not functional. It might give you comfort (as in—I feel safe. If it’s highlighted, it’s probably code) but you can’t use it as a tool. It doesn’t help you.

What’s the solution? Have an absolute minimum of colors. So little that they all fit in your head at once. For example, my color theme, Alabaster, only uses four:

  • Green for strings
  • Purple for constants
  • Yellow for comments
  • Light blue for top-level definitions

That’s it! And I was able to type it all from memory, too. This minimalism allows me to actually do lookups: if I’m looking for a string, I know it will be green. If I’m looking at something yellow, I know it’s a comment.

Limit the number of different colors to what you can remember.

If you swap green and purple in my editor, it’ll be a catastrophe. If somebody swapped colors in yours, would you even notice?

What should you highlight?

Something there isn’t a lot of. Remember—we want highlights to stand out. That’s why I don’t highlight variables or function calls—they are everywhere, your code is probably 75% variable names and function calls.

I do highlight constants (numbers, strings). These are usually used more sparingly and often are reference points—a lot of logic paths start from constants.

Top-level definitions are another good idea. They give you an idea of a structure quickly.

Punctuation: it helps to separate names from syntax a little bit, and you care about names first, especially when quickly scanning code.

Please, please don’t highlight language keywords. class, function, if, elsestuff like this. You rarely look for them: “where’s that if” is a valid question, but you will be looking not at the if the keyword, but at the condition after it. The condition is the important, distinguishing part. The keyword is not.

Highlight names and constants. Grey out punctuation. Don’t highlight language keywords.

Comments are important

The tradition of using grey for comments comes from the times when people were paid by line. If you have something like

of course you would want to grey it out! This is bullshit text that doesn’t add anything and was written to be ignored.

But for good comments, the situation is opposite. Good comments ADD to the code. They explain something that couldn’t be expressed directly. They are important.

So here’s another controversial idea:

Comments should be highlighted, not hidden away.

Use bold colors, draw attention to them. Don’t shy away. If somebody took the time to tell you something, then you want to read it.

Two types of comments

Another secret nobody is talking about is that there are two types of comments:

  1. Explanations
  2. Disabled code

Most languages don’t distinguish between those, so there’s not much you can do syntax-wise. Sometimes there’s a convention (e.g. -- vs /* */ in SQL), then use it!

Here’s a real example from Clojure codebase that makes perfect use of two types of comments:

Disabled code is gray, explanation is bright yellow

Light or dark?

Per statistics, 70% of developers prefer dark themes. Being in the other 30%, that question always puzzled me. Why?

And I think I have an answer. Here’s a typical dark theme:

and here’s a light one:

On the latter one, colors are way less vibrant. Here, I picked them out for you:

Notice how many colors there are. No one can remember that many.

This is because dark colors are in general less distinguishable and more muddy. Look at Hue scale as we move brightness down:

Basically, in the dark part of the spectrum, you just get fewer colors to play with. There’s no “dark yellow” or good-looking “dark teal”.

Nothing can be done here. There are no magic colors hiding somewhere that have both good contrast on a white background and look good at the same time. By choosing a light theme, you are dooming yourself to a very limited, bad-looking, barely distinguishable set of dark colors.

So it makes sense. Dark themes do look better. Or rather: light ones can’t look good. Science ¯\_(ツ)_/¯

But!

But.

There is one trick you can do, that I don’t see a lot of. Use background colors! Compare:

The first one has nice colors, but the contrast is too low: letters become hard to read.

The second one has good contrast, but you can barely see colors.

The last one has both: high contrast and clean, vibrant colors. Lighter colors are readable even on a white background since they fill a lot more area. Text is the same brightness as in the second example, yet it gives the impression of clearer color. It’s all upside, really.

UI designers know about this trick for a while, but I rarely see it applied in code editors:

If your editor supports choosing background color, give it a try. It might open light themes for you.

Bold and italics

Don’t use. This goes into the same category as too many colors. It’s just another way to highlight something, and you don’t need too many, because you can’t highlight everything.

In theory, you might try to replace colors with typography. Would that work? I don’t know. I haven’t seen any examples.

Using italics and bold instead of colors

Myth of number-based perfection

Some themes pay too much attention to be scientifically uniform. Like, all colors have the same exact lightness, and hues are distributed evenly on a circle.

This could be nice (to know if you have OCD), but in practice, it doesn’t work as well as it sounds:

OkLab l=0.7473 c=0.1253 h=0, 45, 90, 135, 180, 225, 270, 315

The idea of highlighting is to make things stand out. If you make all colors the same lightness and chroma, they will look very similar to each other, and it’ll be hard to tell them apart.

Our eyes are way more sensitive to differences in lightness than in color, and we should use it, not try to negate it.

Let’s design a color theme together

Let’s apply these principles step by step and see where it leads us. We start with the theme from the start of this post:

First, let’s remove highlighting from language keywords and re-introduce base text color:

Next, we remove color from variable usage:

and from function/method invocation:

The thinking is that your code is mostly references to variables and method invocation. If we highlight those, we’ll have to highlight more than 75% of your code.

Notice that we’ve kept variable declarations. These are not as ubiquitous and help you quickly answer a common question: where does thing thing come from?

Next, let’s tone down punctuation:

I prefer to dim it a little bit because it helps names stand out more. Names alone can give you the general idea of what’s going on, and the exact configuration of brackets is rarely equally important.

But you might roll with base color punctuation, too:

Okay, getting close. Let’s highlight comments:

We don’t use red here because you usually need it for squiggly lines and errors.

This is still one color too many, so I unify numbers and strings to both use green:

Finally, let’s rotate colors a bit. We want to respect nesting logic, so function declarations should be brighter (yellow) than variable declarations (blue).

Compare with what we started:

In my opinion, we got a much more workable color theme: it’s easier on the eyes and helps you find stuff faster.

Shameless plug time

I’ve been applying these principles for about 8 years now.

I call this theme Alabaster and I’ve built it a couple of times for the editors I used:

It’s also been ported to many other editors and terminals; the most complete list is probably here. If your editor is not on the list, try searching for it by name—it might be built-in already! I always wondered where these color themes come from, and now I became an author of one (and I still don’t know).

Feel free to use Alabaster as is or build your own theme using the principles outlined in the article—either is fine by me.

As for the principles themselves, they worked out fantastically for me. I’ve never wanted to go back, and just one look at any “traditional” color theme gives me a scare now.

I suspect that the only reason we don’t see more restrained color themes is that people never really thought about it. Well, this is your wake-up call. I hope this will inspire people to use color more deliberately and to change the default way we build and use color themes.

Permalink

Statistics made simple

I have a weird relationship with statistics: on one hand, I try not to look at it too often. Maybe once or twice a year. It’s because analytics is not actionable: what difference does it make if a thousand people saw my article or ten thousand?

I mean, sure, you might try to guess people’s tastes and only write about what’s popular, but that will destroy your soul pretty quickly.

On the other hand, I feel nervous when something is not accounted for, recorded, or saved for future reference. I might not need it now, but what if ten years later I change my mind?

Seeing your readers also helps to know you are not writing into the void. So I really don’t need much, something very basic: the number of readers per day/per article, maybe, would be enough.

Final piece of the puzzle: I self-host my web projects, and I use an old-fashioned web server instead of delegating that task to Nginx.

Static sites are popular and for a good reason: they are fast, lightweight, and fulfil their function. I, on the other hand, might have an unfinished gestalt or two: I want to feel the full power of the computer when serving my web pages, to be able to do fun stuff that is beyond static pages. I need that freedom that comes with a full programming language at your disposal. I want to program my own web server (in Clojure, sorry everybody else).

Existing options

All this led me on a quest for a statistics solution that would uniquely fit my needs. Google Analytics was out: bloated, not privacy-friendly, terrible UX, Google is evil, etc.

What is going on?

Some other JS solution might’ve been possible, but still questionable: SaaS? Paid? Will they be around in 10 years? Self-host? Are their cookies GDPR-compliant? How to count RSS feeds?

Nginx has access logs, so I tried server-side statistics that feed off those (namely, Goatcounter). Easy to set up, but then I needed to create domains for them, manage accounts, monitor the process, and it wasn’t even performant enough on my server/request volume!

My solution

So I ended up building my own. You are welcome to join, if your constraints are similar to mine. This is how it looks:

It’s pretty basic, but does a few things that were important to me.

Setup

Extremely easy to set up. And I mean it as a feature.

Just add our middleware to your Ring stack and get everything automatically: collecting and reporting.

(def app
  (-> routes
    ...
    (ring.middleware.params/wrap-params)
    (ring.middleware.cookies/wrap-cookies)
    ...
    (clj-simple-stats.core/wrap-stats))) ;; <-- just add this

It’s zero setup in the best sense: nothing to configure, nothing to monitor, minimal dependency. It starts to work immediately and doesn’t ask anything from you, ever.

See, you already have your web server, why not reuse all the setup you did for it anyway?

Request types

We distinguish between request types. In my case, I am only interested in live people, so I count them separately from RSS feed requests, favicon requests, redirects, wrong URLs, and bots. Bots are particularly active these days. Gotta get that AI training data from somewhere.

RSS feeds are live people in a sense, so extra work was done to count them properly. Same reader requesting feed.xml 100 times in a day will only count as one request.

Hosted RSS readers often report user count in User-Agent, like this:

Feedly/1.0 (+http://www.feedly.com/fetcher.html; 457 subscribers; like FeedFetcher-Google)

Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 6 subscribers)

Feedbin feed-id:1373711 - 142 subscribers

My personal respect and thank you to everybody on this list. I see you.

Graphs

Visualization is important, and so is choosing the correct graph type. This is wrong:

Continuous line suggests interpolation. It reads like between 1 visit at 5am and 11 visits at 6am there were points with 2, 3, 5, 9 visits in between. Maybe 5.5 visits even! That is not the case.

This is how a semantically correct version of that graph should look:

Some attention was also paid to having reasonable labels on axes. You won’t see something like 117, 234, 10875. We always choose round numbers appropriate to the scale: 100, 200, 500, 1K etc.

Goes without saying that all graphs have the same vertical scale and syncrhonized horizontal scroll.

Insights

We don’t offer much (as I don’t need much), but you can narrow reports down by page, query, referrer, user agent, and any date slice.

Not implemented (yet)

It would be nice to have some insights into “What was this spike caused by?”

Some basic breakdown by country would be nice. I do have IP addresses (for what they are worth), but I need a way to package GeoIP into some reasonable size (under 1 Mb, preferably; some loss of resolution is okay).

Finally, one thing I am really interested in is “Who wrote about me?” I do have referrers, only question is how to separate signal from noise.

Performance. DuckDB is a sport: it compresses data and runs column queries, so storing extra columns per row doesn’t affect query performance. Still, each dashboard hit is a query across the entire database, which at this moment (~3 years of data) sits around 600 MiB. I definitely need to look into building some pre-calculated aggregates.

One day.

How to get

Head to github.com/tonsky/clj-simple-stats and follow the instructions:

Let me know what you think! Is it usable to you? What could be improved?

Permalink

Clojure on Fennel part four: Parsing (again)

Other than that, the parsing is complete, and we can look at the compiler part of the ClojureFnl project. But that’s gonna be in the next post.

Sike!

Wasn’t my intention to fake out like this again, but while working on the compiler, I had an important realization that led me to redesign the parser completely. And it’s a bit of a shame, because I already had a decent part of the compiler working, being able to run the REPL and do various cool things with it. But, better sooner than later, I guess.

I had a good amount of the post about the compiler already written too, but now I’ll have to discard all of that. So instead, I decided to talk about the new parser by explaining why the old one failed me. And to do so, I’ll give you a brief look on how the compiler currently works with parsed code.

Old parser and compiler

So, in the previous post, we’ve reached a point where the parser can read code into a tagged tree. I spent a fair amount of time discussing how I wanted this specifically, and now it feels like I’m backpedalling, but tagged tree is not the way forward. Here’s the idea.

Currently, we’re reading this expression (+ 1 2) into:

[:code
 [:list
  [:symbol "+"]
  [:whitespace " "]
  [:number "1"]
  [:whitespace " "]
  [:number "2"]]]

The compiler then walks this tree, taking the first value of each node to decide what to do. First, it sees :code - that’s just an entry point, containing multiple top-level expressions. Next, it sees :list and dispatches to a function dedicated to compiling lists.

This function checks the first element of the list, sees that it is a :symbol, and checks whether this symbol is somehow special. While + doesn’t look that special, it is to the compiler, because Clojure’s + and Fennel’s + are not the same thing. In Clojure, + is a function, in Fennel, it is a special.

So the compiler replaces + with clojure_core_ns.add, and then proceeds over the rest of the forms in the list, recursively calling itself over each. In the end, we get this, written into a string builder:

(clojure_core_ns.add 1 2)

This is a somewhat simplified explanation, because I’m omitting scope resolution, symbol shadowing, and other such things that the compiler currently tracks. For actual specials, like Clojure’s let*, the story is a bit different too. The compiler has a dedicated compile-special function for all Clojure specials provided by cljlib as Fennel macros. And that’s where things started to go haywire.

Look at this code:

(def ^:private foo 42)

It’s read like this:

[:list
 [:symbol "def"]
 [:whitespace " "]
 [:metadata
  [:metadata-entry [:keyword ":private"]]
  [:whitespace " "]
  [:symbol "foo"]]
 [:whitespace " "]
 [:number "42"]]

Here’s a problem - the parser reads metadata ^:private, and metadata in Clojure is attached to symbols, so the grammar I use reads the next value after the metadata and wraps it into a single node:

[:metadata
 [:metadata-entry [:keyword ":private"]]
 [:whitespace " "]
 [:symbol "foo"]]

The compiler, while compiling this list, sees def as the first symbol and enters compile-special.def. But def expects a symbol to bind the value to, while here we have a different node type, called :metadata. So my compiler had to account for all cases where metadata can appear - and it wasn’t easy to do.

I tried to side-step this by always expecting metadata, because it can appear almost anywhere, and discarding it, because Fennel’s concept of metadata is a bit different. In which I mostly succeeded. But this wasn’t the only problematic thing to support.

Here’s another example, now from Edamame:

(defn- read-token
  "Read in a single logical token from the reader"
  ^String [#?(:clj rdr :cljs ^not-native rdr :cljr rdr) _kind initch]
  (loop [sb #?(:clj (StringBuilder.)
               :cljs (StringBuffer.)
               :cljr (StringBuilder.))
         ch initch]
    (if (or (whitespace? ch)
            (macro-terminating? ch)
            (nil? ch))
      (do (when ch
            (r/unread rdr ch))
          (str sb))
      (recur #?(:clj (.append sb ch) :cljs (.append sb ch) :cljr (.Append sb (str ch))) (r/read-char rdr)))))

It reads into this tagged tree:

[:list
 [:symbol "defn-"]
 [:whitespace " "]
 [:symbol "read-token"]
 [:whitespace "\n  "]
 [:string "\"Read in a single logical token from the reader\""]
 [:whitespace "\n  "]
 [:metadata
  [:metadata-entry [:symbol "String"]]
  [:whitespace " "]
  [:vector
   [:conditional
    [:list
     [:keyword ":clj"]
     [:whitespace " "]
     [:symbol "rdr"]
     [:whitespace " "]
     [:keyword ":cljs"]
     [:whitespace " "]
     [:metadata
      [:metadata-entry [:symbol "not-native"]]
      [:whitespace " "]
      [:symbol "rdr"]]
     [:whitespace " "]
     [:keyword ":cljr"]
     [:whitespace " "]
     [:symbol "rdr"]]]
   [:whitespace " "]
   [:symbol "_kind"]
   [:whitespace " "]
   [:symbol "initch"]]]
 [:whitespace "\n  "]
 [:list
  [:symbol "loop"]
  [:whitespace " "]
  [:vector
   [:symbol "sb"]
   [:whitespace " "]
   [:conditional
    [:list
     [:keyword ":clj"]
     [:whitespace " "]
     [:list [:symbol "StringBuilder."]]
     [:whitespace "\n               "]
     [:keyword ":cljs"]
     [:whitespace " "]
     [:list [:symbol "StringBuffer."]]
     [:whitespace "\n               "]
     [:keyword ":cljr"]
     [:whitespace " "]
     [:list [:symbol "StringBuilder."]]]]
   [:whitespace "\n         "]
   [:symbol "ch"]
   [:whitespace " "]
   [:symbol "initch"]]
  [:whitespace "\n    "]
  [:list
   [:symbol "if"]
   [:whitespace " "]
   [:list
    [:symbol "or"]
    [:whitespace " "]
    [:list [:symbol "whitespace?"] [:whitespace " "] [:symbol "ch"]]
    [:whitespace "\n            "]
    [:list
     [:symbol "macro-terminating?"]
     [:whitespace " "]
     [:symbol "ch"]]
    [:whitespace "\n            "]
    [:list [:symbol "nil?"] [:whitespace " "] [:symbol "ch"]]]
   [:whitespace "\n      "]
   [:list
    [:symbol "do"]
    [:whitespace " "]
    [:list
     [:symbol "when"]
     [:whitespace " "]
     [:symbol "ch"]
     [:whitespace "\n            "]
     [:list
      [:symbol "r/unread"]
      [:whitespace " "]
      [:symbol "rdr"]
      [:whitespace " "]
      [:symbol "ch"]]]
    [:whitespace "\n          "]
    [:list [:symbol "str"] [:whitespace " "] [:symbol "sb"]]]
   [:whitespace "\n      "]
   [:list
    [:symbol "recur"]
    [:whitespace " "]
    [:conditional
     [:list
      [:keyword ":clj"]
      [:whitespace " "]
      [:list
       [:symbol ".append"]
       [:whitespace " "]
       [:symbol "sb"]
       [:whitespace " "]
       [:symbol "ch"]]
      [:whitespace " "]
      [:keyword ":cljs"]
      [:whitespace " "]
      [:list
       [:symbol ".append"]
       [:whitespace " "]
       [:symbol "sb"]
       [:whitespace " "]
       [:symbol "ch"]]
      [:whitespace " "]
      [:keyword ":cljr"]
      [:whitespace " "]
      [:list
       [:symbol ".Append"]
       [:whitespace " "]
       [:symbol "sb"]
       [:whitespace " "]
       [:list [:symbol "str"] [:whitespace " "] [:symbol "ch"]]]]]
    [:whitespace " "]
    [:list [:symbol "r/read-char"] [:whitespace " "] [:symbol "rdr"]]]]]]

Yes, it’s abysmal, but bear with me. I had to work with this, after all, thinking that this is a blessing.

The compiler sees defn- and enters the compile-special.defn- function. Defn expects a function name, then a vector for its arguments. Here, instead of the vector, we have metadata node again. This is fine, since I just mentioned above that I managed to sidestep this problem.

Since this is an arglist, we need to compile it in a special way, adding its symbols to the function scope, etc. However, instead of arguments, we see the conditional node:

[:vector
 [:conditional
  [:list
   [:keyword ":clj"] [:whitespace " "] [:symbol "rdr"] [:whitespace " "]
   [:keyword ":cljs"] [:whitespace " "] [:metadata [:metadata-entry [:symbol "not-native"]] [:whitespace " "] [:symbol "rdr"]] [:whitespace " "]
   [:keyword ":cljr"] [:whitespace " "] [:symbol "rdr"]]]
 [:whitespace " "]
 [:symbol "_kind"]
 [:whitespace " "]
 [:symbol "initch"]]

I didn’t think it was allowed in Clojure to use conditional reading inside forms like this. But apparently it’s OK, and my compiler failed to deal with it.

So now, any node can not only be a metadata node, but also a conditional node. This complicates things, but at this point I’m still thinking that I can persevere.

So I did. I added support for almost all specials provided by cljlib. The main thing that was left to do were macros. And then it hit me:

I can’t do macros like that!

Why? Why, of course, because macros don’t emit tagged trees that my compiler understands. They emit code!

Here’s a simple macro:

(defmacro unless [test & body]
  `(when (not ~test)
     ~@body))

We can parse it:

[:list
 [:symbol "defmacro"] [:whitespace " "] [:symbol "unless"] [:whitespace " "]
 [:vector
  [:symbol "test"] [:whitespace " "] [:symbol "&"] [:whitespace " "] [:symbol "body"]]
 [:whitespace " "]
 [:backtick
  [:list
   [:symbol "when"] [:whitespace " "]
   [:list [:symbol "not"] [:whitespace " "] [:unquote [:symbol "test"]]]
   [:whitespace " "] [:unquote-splicing [:symbol "body"]]]]]

We can probably even compile it to something that we could then call during the compiler step. However, what will (unless true (println 42)) thing return?

(when (not true) (println 42)), of course. It’s a Lisp macro, what else did you expect?

But what does the compiler expect?

[:list
 [:symbol "when"]
 [:whitespace " "]
 [:list [:symbol "not"] [:whitespace " "] [:symbol "true"]]
 [:whitespace " "]
 [:list [:symbol "println"] [:whitespace " "] [:number "42"]]]

Oh, it wants this.

I have to convert macro’s output into a string, parse it, and feed it into the compiler if I want macros to work at all. And that’s BAD. If only I realized this sooner!

This was the final nail in the coffin of the tagged tree approach for my project. I knew I had to rewrite the parser to emit actual data structures that I’ll be able to emit from macros as well, and compile them.

New parser reader

So I decided that I need a proper lisp reader that will produce data structures:

;; Welcome to Fennel Proto REPL 0.6.4-dev
;; Fennel version: 1.7.0-dev
;; Lua version: PUC Lua 5.5
;; Work directory: ~/Projects/fennel/ClojureFnl/
>> (local reader (require :impl.reader))
nil
>> (reader.read-string "(def ^:private foo 42)")
(def foo 42)
>> (local {: meta : second} (require :clojure.core))
nil
>> (meta (second (reader.read-string "(def ^:private foo 42)")))
{:private true}

As can be seen, the new reader module provides the read-string function that produces data structures. Notably, there are no longer any metadata nodes - metadata is assigned to the symbol foo in this case.

Same goes for the reader conditionals:

>> (reader.read-string "[0 #?(:clj 1 :cljfnl 2) #?@(:clj [2 3] :cljfnl [3 4]) 5]")
[0 2 3 4 5]

These are now correctly spliced at read time.

This also solved macro problems for the most part:

>> (reader.read-string "`(+ ~x ~@y ~z)")
(clojure.core/seq
 (clojure.core/concat
  (clojure.core/list (quote clojure.core/+))
  (clojure.core/list x)
  y
  (clojure.core/list z)))

And it matches what Clojure itself does:

Clojure 1.12.5
user=> (read-string "`(+ ~x ~@y ~z)")
(clojure.core/seq
 (clojure.core/concat
  (clojure.core/list (quote clojure.core/+))
  (clojure.core/list x)
  y
  (clojure.core/list z)))

Due to this change, the compiler doesn’t have to know about syntax quote (`) at all, and thus macros will correctly return these kinds of lists.

So, now we can read Clojure code into data structures. Previously, the parser returned [:code A B ...] kind of result, where A, B, and the rest are all top-level forms. New reader also does this, but returns them as a list:

>> (reader.read-string-all "1 \"str\" :keyword ::namespaced #:map{:key :val}")
(1 "str" :keyword :user/namespaced {:map/key :val})

However, there’s a problem.

Consider the ::namespaced keyword. Both my reader and Clojure read it as :user/namespaced, but this is because the default namespace in the REPL is user. For my parser, it’s mostly an arbitrary choice because it doesn’t know anything about runtime.

Clojure 1.12.5
user=> (read-string "::x")
:user/x
user=> (ns foo)
nil
foo=> (read-string "::x")
:foo/x

And when compiling a file, the file might have an ns declaration, or even multiple of them:

(ns foo)

(println ::x)

(ns bar)

(println ::x)

If we were to run this code, we would see :foo/x then :bar/x.

My parser currently reads all input, meaning it transforms all top-level forms from text to data. However, this also means that it doesn’t understand anything about namespaces.

Previously, this was fine because the compiler handled this - the tagged tree didn’t resolve anything. Currently, however, we need to construct namespaced keys, and we need to know the namespace. But it’s not possible unless we parse the input expression by expression, instead of all at once.

Hence, I had to update the grammar so it could parse expression by expression. This way, I could read source code form by form, and compile each form separately. And if the compiler encounters an ns declaration, it could update some state, so the reader would know how to read namespaced keywords, and other things that may include namespaces.

But we still need to pass this state to the reader. And my reader is based on a PEG grammar.

Luckily for me, the lpeg library I use has a Carg function that can pass additional arguments into the PEG parser. This way I can write my transformation function T:

(fn T [tag patt]
  (/ (* (Ct (* (Cc tag) patt)) (Carg 1))
     (fn [node state]
       (case node
         ;; ----8<----
         [:macro-keyword bo data]
         (read-macro-keyword data bo state)
         [:conditional data]
         (read-conditional data state)
         [:conditional-splicing data]
         (conditional-splicing data state)
         ;; ----8<----
         _ _))))

This state can be passed anew after reading each expression, or mutated in place - either way, the reader now knows how to access state. The reader itself is still stateless.

The reader now supports all Clojure syntax and produces data structures:

;; Clojure                                          ;; Fennel
1                                                   ;; 1
0.5                                                 ;; 0.5
1e10                                                ;; 10000000000.0
1/2                                                 ;; 1/2
1N                                                  ;; 1N
1.33333333333333333333333333333333M                 ;; 1.33333333333333333333333333333333M
0xFF                                                ;; 255
017                                                 ;; 15
16rFF                                               ;; 255
\c                                                  ;; "c"
"string"                                            ;; "string"
\u0041                                              ;; "A"
\o101                                               ;; "A"
:keyword                                            ;; :keyword
:namespaced/keyword                                 ;; :namespaced/keyword
::auto-keyword                                      ;; :user/auto-keyword
symbol                                              ;; symbol
namespaced/symbol                                   ;; namespaced/symbol
#'var                                               ;; (var var)
'quoted-symbol                                      ;; (quote quoted-symbol)
(list)                                              ;; (list)
[vector]                                            ;; [vector]
{hash map}                                          ;; {hash map}
#:namespaced{:map 42}                               ;; {:namespaced/map 42}
#::{:auto :namespaced}                              ;; {:user/auto :namespaced}
#{hash set}                                         ;; #{set hash}
#"regexp"                                           ;; "regexp"
@dereferencing                                      ;; (clojure.core/deref dereferencing)
#(+ %1 %2)                                          ;; (fn* [p__0__ p__1__] (+ p__0__ p__1__))
^:meta data                                         ;; data
#_discard                                           ;;
[#?(:cljfnl "reader") #?@(:cljfnl [:conditionals])] ;; ["reader" :conditionals]
nil                                                 ;; nil
true                                                ;; true
false                                               ;; false
##NaN                                               ;; .nan
##Inf                                               ;; .inf
#inst "2022"                                        ;; #inst "2022-01-01T00:00:00.000-00:00"
#uuid "c6e8050a-789b-4305-b518-8f0f7c31da7a"        ;; #uuid "c6e8050a-789b-4305-b518-8f0f7c31da7a"

Note about data structures: even though we’re working in Fennel, the produced maps, vectors and lists are custom data structures that are implemented in the cljlib library. These are implementations of persistent data structures I’ve talked about in part 2.

Further work

With that, I can work on the compiler, now for real. It’s a shame that I basically have to do all the work on the compiler again from scratch, because the underlying data has changed so much, but this simplifies a lot of things, so I’m OK with that.

One thing that the reader still doesn’t support yet is read-time evaluation with #=expr. This requires a working compiler, and I’ll have to first implement it, then integrate it back into the reader.

Another thing I was thinking about was to drop the LPEG parser altogether. Maybe, when I have the compiler working and have support for all Clojure runtime semantics in place, I’ll make a fork of the Edamame parser and replace the current reader with it, adding support for ClojureFnl via reader conditionals. This will remove a C library dependency, which is a good thing for distribution. I still have another C library for arbitrary-precision numbers, but it can be worked around.

But, lesson learned - when implementing a lisp, even if it is just a transpiler, don’t try to cut corners, and make a proper reader.

Next post, for sure, will be about the compiler!

Comment via email

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.