Learn linear least squares with me

Permalink

biff.datastar, biff.ring

A couple more Biff 2 libraries are out the door:

• biff.datastar: the dumbest/awesomest possible way to make a reactive, server-side-rendered web app. Over the past several years I haven't put much priority on making it easy to make fancy reactive/real-time/collaborative UIs since my UI needs are typically pretty simple. But this architecture is actually really nice even when you don't need the fancy stuff.

• biff.ring: mostly some Biff-related plumbing code. I think the defroute macro is pretty nice. There's a cool wrap-csrf-protection middleware that doesn't use tokens.

The last "big" Biff 2 library I need to fix up and release will be biff.tasks, the one for all the CLI tasks. Other than that there's biff.authentication, the email-powered authentication module thing (now with a default sign-in form included) and a few other doodads. And then a starter app. And some documentation that ties all the libraries together, not just documentation for the individual libraries (which I've been writing as I go).

I'm still shooting to have all that released before the conj, which is... coming up. On the bright side, in the window of time between writing the first draft of this post and sending it out, I've already finished editing the biff.tasks code and only need to write the documentation. So I'd say we're well on our way.

Permalink

Design Ethical AI Rewards and Policies

Permalink

Fibers in Jolt: green threads that fit core.async

Jolt now provides opt-in support to run go blocks on fibers instead of OS threads. This post will discuss how that works along with the different trade-offs made compared to other implementations, and why core.async turns out to be a great fit for the mechanism. All the numbers below are from an Apple M1 Pro with 10 cores, measured with the harness in bench/fibers on the current tree.

The problem with a thread per process

As you probably know, core.async's programming model is best served by cheap green processes that communicate over channels. Jolt's original implementation backed every go block with a real OS thread. Using system threads affords the same semantics, and it has a nice property that there's nothing special about a go body. But real threads have significant overheads putting a limit on the number you can reasonably have, and they're not particularly cheap to start.

Let's take a look at what that ceiling looks like for threads and fibers when spawning K processes that each immediately park on an empty channel:

As the number of threads goes up, they effectively stop working. My machine runs out somewhere around 4,080 live threads, and by that point, the spawn cost has already gone up 3x. Memory story isn't encouraging either with a parked process costing 4,160 bytes of live heap on a fiber while sitting at 68,729 bytes on a thread. That's a 16.5x difference as a base, and measured as peak RSS rather than live bytes the gap widens to 44x since a thread needs a guard page along with a real stack mapping. So the motivation to have a light weight mechanism should be pretty obvious.

What a fiber is here

A fiber consists of a record holding a state, a body thunk, a continuation slot, an intrusive run queue link, a slice of per-fiber dynamic state, and the carrier it belongs to. In addition, the scheduler needs a little bookkeeping to track the pending step, the fiber's registered monitors, and the interrupt depth it parked at. Parking captures the current continuation with call/1cc and jumps to the scheduler, and that continuation is what gets invoked when resuming. That's all there is to it at a high level, and on Chez a bare continuation switch measures just 8.6 ns.

The current fiber lives in a Chez virtual register costing about 2 ns to read. Since the cost is so low, a scheduler can do millions of switches per second. The full scheduler yield, including swapping the fiber's dynamic slice and arming the preemption timer described further down, comes to around 137 ns.

Another consideration here is that a continuation on Chez is a stack segment, which is not free. A completed fiber that holds no continuation costs just 108 bytes, but the moment it parks, the cost jumps to about 4,177 bytes. Interestingly, that number barely moves with stack depth:

So a fiber isn't actually cheap because its stack is small, but due to 4 KB being much less than the 69 KB an OS thread costs, which makes it possible to have hundreds of thousands of them running concurrently.

Why core.async fits

On the JVM, go had to be a macro that CPS-transforms its body into a state machine because the JVM had no continuations when core.async was originally written. That's the key reason why <! and >! only work lexically inside a go block. Putting a parking take inside a function then calling it from a go body does not work since the macro cannot see past the call boundary to rewrite it.

Jolt, on the other hand, has real continuations, so there's no need for the transform to park. A fiber's <! registers a waiter on the channel and captures its continuation, wherever it happens to be. With this approach, parking works through arbitrary call depth, helper functions, callbacks, or even eval. The whole limitation core.async has on the JVM goes away on the Chez runtime.

Jolt ships its own native channels, but it implements the same design for the waiter protocol. A channel operation that cannot complete immediately registers a handler and waits to be woken. The only difference from threads is that they wait on a condition variable while fibers wait by parking. The channel core doesn't need to know which it is talking to. All it has to do is commit to a handler under its lock, write a mailbox, and call a wake function. The immediate-completion path, where a buffered value or a waiting putter is already there, captures nothing and never touches the scheduler at all.

Adding fibers meant adding a second wake strategy, and the blocking variants fall out from that as well. On a fiber, <!! and >!! park exactly the way <! and >! do. Blocking semantics are preserved, in that the process does not proceed until the value arrives.

Paying for the continuation only when you need it

Yet, having 4 KB per parked fiber still bothered me since for a large class of go bodies it should be avoidable. When the park site is visible to the compiler, the rest of the body can be turned into a closure of a few hundred bytes rather than necessitating a whole stack segment.

And that brings us back to the JVM's transform, which I originally avoided, but the big question was how to avoid inheriting the JVM's limitation along with it. Having to prove that the pass can rewrite every park in a body means a closed world analysis with anything calling park within the body resulting in a compilation time error.

Jolt makes the choice per park site instead, with the continuation park being used as the runtime fallback. A CPS pass in clojure.core.async rewrites the body where it can so that a park it rewrote stores a closure and switches with no capture, but a park it could not rewrite is left as written and parks by capturing. Both mechanisms coexist inside a single fiber, and can be mixed freely. Since there is no static claim to defend, a park inside a called function, a try, a nested fn, a collection literal, or reached through eval still works. It just keeps the continuation park, which is what a park cost before any of this existed.

The result on a parked process:

That's a 5.5x difference in memory. The park and resume round trip, meanwhile, is 1,040 to 1,049 ns rewritten against 984 to 1,018 ns captured, so the actual win here is memory usage rather than time.

It's also worth noting that alts! still captures, because threading a continuation through the waiter registration would be its own major piece of work, and so does any park inside a try, because the rewrite would have to carry the exception frame explicitly. The pass also treats a bare fn as opaque, since it cannot see what the closure is handed to, and that covers a larger set than it sounds like since binding, dosync and locking all hand their body over as a function, and a park inside any of them takes the capture too. All of these are correctness-preserving fallbacks rather than failures, however. A rewritten park does not rewind the dynamic-wind chain on the way back in, so a park that sits inside a wind has to be one the pass left alone.

Preemption, and the locks it forced

A cooperative scheduler assumes that go bodies reach a channel op reasonably often. When a body is pure computation instead, the fiber holds its carrier for as long as it runs, and every fiber queued behind it is simply stuck since fibers cannot migrate carriers. That creates potential for an unbounded starvation window.

The way to deal with the problem is to make the scheduler preemptive. Chez polls an engine timer at procedure calls and loop back edges, which means even a tight Scheme loop is preemptible, and the timer handler can turn the fiber's quantum into a yield. The default quantum is about 0.45 ms. A queued fiber stuck behind a fiber spinning in a bare loop on the same carrier gets to run within about a millisecond, and a 200 ms compute-bound spin gets preempted around 265 times.

The clojure.core.async/*fiber-preempt-ticks* var sets the quantum, subject to a floor, and is read once when the carrier pool starts. No value turns preemption off, so code that wants effectively cooperative behaviour can ask for a very long quantum instead. However, it's worth noting that preemption cannot help with a fiber that's inside a blocking foreign call because the timer is only polled in Scheme.

To ensure that preemption works safely, every lock in the runtime routes through a common counting wrapper, and the scheduler refuses to switch a fiber that holds one, re-arming on a short retry so the preemption lands just after the region instead of being dropped. That works because those regions measure around 55 ns against a 0.45 ms quantum, but the locks whose region is a user body are a special case. These include locking, dosync, a delay being forced, and java.util.concurrent.locks.ReentrantLock. Those regions are as long as the caller's code and the caller may park inside them, so they must carry ownership in a field keyed on the fiber rather than in an OS mutex. A field survives a context switch, and no counted lock is held while user code runs, which makes a long locking body preemptible.

The upshot for anyone writing jolt code is that a lock is a lock. You can hold a monitor across a <!, run a transaction that parks in the middle, and force a delay whose body blocks on a channel, with exclusion holding in each case.

Proving the design using formal tooling

The awkward thing about concurrency work is that it's both notoriously difficult to reason about and to test exhaustively. So, I decided to try proving certain properties of the design using Z3 through the chiasmus MCP to help ensure that my approach was sound. The pattern is to state the rule along with the property it is supposed to enforce, then have the solver either hand back a counterexample or report that none exists.

The lock ownership rule is a good example to walk through because it is small enough to show in full. The entire question here is which identity an acquire writes into the owner field and what the next acquire does with it. In the runtime that comes down to two pieces:

;; who is asking: the FIBER when there is one, else the OS thread's identity
(define (monitor-self) (or (jolt-current-fiber) (current-interrupt-box)))

;; and what the acquire does with the answer
(let ((me (monitor-self)))
  (let loop ()
    (let ((owner (vector-ref m monitor-i-owner)))
      (cond
        ((eq? owner me) (vector-set! m monitor-i-count (fx+ 1 (vector-ref m monitor-i-count))))
        ((not owner)    (vector-set! m monitor-i-owner me)
                        (vector-set! m monitor-i-count 1))
        (else           (monitor-wait! m) (loop))))))

The model, written in SMT-LIB, consists of four facts. Two execution contexts have identities, and the design either gives them the same one, which is the thread they share, or different ones, which is the fiber. The first context takes a free lock and parks inside the section without releasing. The second then runs the acquire decision exactly as the code writes it. And the property under test is that no state has both contexts inside the section at once.

(assert (! (= by_thread (= id_f1 id_f2)) :named identity-model))
(assert (! (= owner_after_f1 id_f1)      :named f1-owns))
(assert (! f1_in_section                 :named f1-still-inside))
(assert (! (= f2_enters (or (= owner_after_f1 NONE)
                            (= owner_after_f1 id_f2)))
                                         :named f2-lock-decision))
(assert (! (and f1_in_section f2_enters) :named seeking-violation))

The correspondence we're interested in is that by_thread records which branch monitor-self took, since the thread branch is the one that hands two fibers on a carrier the same identity, and f2_enters is the disjunction of the two cond arms that get in without waiting.

Asking whether that violation is reachable at all comes back SAT, and the assignment the solver hands back is the bug itself: by_thread true, both identities equal, and both contexts inside the section. Pinning the design to context identity and asking the same question comes back UNSAT. Here the unsat core names the identity choice alongside the acquire rule, which says the property depends on that choice rather than holding by accident of how the rest of the model happened to be written.

I could have reasoned through this by hand and been fairly sure, but having a formal proof takes the guesswork out of the rule itself. The solver quantifies over every assignment the model admits, so an UNSAT is exhaustive rather than a sample, and a SAT identifies the exact assignment that breaks it.

Of course, there is a limit to how much a solver can help since it proves a property of the rule I described to it, rather than of the code itself, and it knows nothing about implementation details such as Chez mutexes or the winder chain. Hence, the result can only be as strong as the model is faithful. However, there is a lot of value in knowing that the approach is fundamentally sound, while the actual implementation can be covered by the tests.

Not every invariant has a shape that lends itself well to this approach, and you have to know when to reach for it. The things generally worth formalizing are the rules for the load-bearing decisions that determine whether the approach itself is sound or not.

How this compares to Go and the JVM

Go's goroutines start with a small stack, around 2 KB, and grow by copying when they need more. Since the Go runtime has precise stack maps, it allows relocating a goroutine's stack, so goroutines can migrate freely between OS threads, allowing the scheduler to steal work. A goroutine that blocks on IO parks on the netpoller, and a goroutine that makes a genuinely blocking syscall causes its processor to be handed to another thread.

JVM virtual threads keep their stack as heap-allocated chunks that mount and unmount from a carrier thread. Unmounting copies the stack out while mounting copies it back. A virtual thread on the JVM can also remount on a different carrier than the one it last ran on.

Unfortunately, Jolt cannot do either, which leads us to the central trade-off. A Chez continuation captured on one OS thread raises "attempt to return to stale foreign context" when you try to resume it on another. So a fiber is bound to its carrier for life. There is no way to load balance the work since an idle carrier cannot take another carrier's queued fibers.

Preemption means the fibers sharing a carrier at least take turns, avoiding a starvation problem. What is left is that a carrier's work cannot be moved somewhere else, which shows up as skew. You can see both halves in the scaling benchmark. Forty CPU-bound fibers across carriers scale nearly linearly, from 903 Mops/sec on one carrier to 6,499 on ten, which is 7.2x on my 10-core machine. Give one fiber ten times the work of the others and the batch takes 112 ms, which is how long that one fiber takes.

Go and the JVM are able to rebalance because they have a stack representation that can be moved around. Jolt doesn't have a similar mechanism to lean on, so the carrier pool acts as a throughput knob, and growing it does not rescue work that is already skewed onto one carrier.

Blocking IO, which is where this usually falls apart

Another key challenge for a green thread system comes from blocking operations such as read on a fiber pinned to a carrier. Since continuations cannot migrate, everything queued up behind it ends up having to wait for it to finish.

Jolt's socket layer sets O_NONBLOCK and treats EAGAIN as "wait for readiness". Waiting means asking a per-process poller, kqueue on macOS and epoll on Linux, to report when the fd is ready. If there is a current fiber, the poller registers the fd and the fiber parks. And when there is not, the caller does a plain blocking wait on its own thread. The user-facing code is identical in both cases, so the same socket code implicitly works on a fiber and on a thread.

The wait has to be collect-safe, because Chez's collector stops the world, and a thread sitting inside a foreign call that is not marked collect-safe still counts as active. As a result, a collection from any other thread fails outright with "cannot collect when multiple threads are active". A poller stuck in kevent is essentially blocked all the time, which means that getting this wrong would result in the process never being able to collect. A full collect must succeed while the poller is blocked, and the failure mode is easy to miss.

Another tricky bit is that registration races need a control pipe. A fiber can register an fd while the poller is already inside kevent, and that registration has to interrupt the wait rather than sit there until the next unrelated event. The pipe read end is permanently in the poller's set, a registration writes a byte, and the poller drains pending registrations on every wake. This approach avoids needing timed polls or doing sleep in the wait path.

Finally, the commit to park has to be atomic with the wake, which is the same race the channel layer has, and gets solved the same way. The fiber marks itself parked under the poller's table lock, the poller collects woken fibers under that same lock and resumes them after releasing it.

The numbers, including the ones that do not flatter fibers

We can see how this trade-off shows up in channel throughput:

Two processes ping-ponging are actually a good margin slower on fibers than on threads. When two fibers land on different carriers, every handoff requires a cross-thread wakeup to take a lock, signal a condition variable, then wake another OS thread. That's strictly more work than two live OS threads doing handoffs directly between each other. But when the pool is pinned to a single carrier, the benchmark runs at 1.89 µs. Since the handoff is now a continuation switch that happens on the same thread, it's 1.9x faster than thread communication.

The fan-in case, which is closer to what people actually build, goes the other way giving 1.65x in favor of fibers. Having eight producers and one consumer is a shape where holding eight OS threads would be significantly more expensive.

Context switch costs, for calibration:

Memory usage characteristics are generally good, and stay flat under churn. Creating 16,000 fibers in waves of 2,000, with a full collect between batches, settles at about 234,000 bytes and stays there from the second wave on, since fibers release their memory as they finish.

What this means in practice

Fibers are provided as an opt-in mechanism which is enabled using clojure.core.async/*go-backend*. The var defaults to :thread, and you bind it to :fiber around the spawn. The thread backend has no pinning story to worry about, and it remains the right default for code that does unpredictable things. The pool size is managed using clojure.core.async/*fiber-carrier-count*, which defaults to the machine's processor count and gets read when the pool starts.

The rough guidance is that if you have many processes that spend most of their time parked, fibers win by a lot, on both spawn cost and memory. If you have a small number of processes doing tight channel handoffs then you have to pin the pool. A process that just computes for a long time is fine since the scheduler preempts it. The case that still needs care is a process that blocks a carrier on something the poller does not cover, and that's where thread should be used, since it always spawns a real OS thread regardless of the backend setting.

The part I find most satisfying is that adding fibers ended up being a matter of implementing a different wake strategy because core.async's channel protocol doesn't assume what a waiter is. And having real continuations means parking is no longer confined to places where the macro can see it. Thus, Jolt avoids the single most annoying restriction of core.async on the JVM, while keeping the compiler transform around as a memory optimization for the cases where it applies.

Permalink

“Code was never the hard part” is an insult to all programmers

Permalink

Program images and portable Scheme backends for Jolt

Two features landed in Jolt recently, both of which fell out from loosening the coupling between Jolt and the host runtime. The first is the ability to serialize program images in the style of Common Lisp and Smalltalk, and the second is to have a portable Scheme layer decoupled from the Chez runtime. Let's take a look at what these things buy Jolt in practical terms.

Images: a black box for your program

If you've ever had to support a production system like a web application, then you know the value of having good logging. What typically happens is that you sprinkle statements through the code, ship them somewhere searchable, and then use them as bread crumbs when something breaks. When a system has errors, you have to first reconstruct what it was doing to understand what happened. If your logging didn't capture a critical piece of information, then the investigation devolves into guess work and attempts to reconstruct the state which caused the error. This can be a particularly frustrating experience when production goes down at 3am.

The trouble with logging is that it forces you to guess the question before you know it. Each log line is a projection of program state chosen in advance, often being composed of two or three things that seemed relevant when you wrote the call. If the actual cause happened to be in the fourth thing, the log tells you nothing of value. You can't go back and ask a different question, because the values are gone, and you likely have no way to access them even if they're still in memory. All you have to work with is the rendering you decided on when you wrote the code originally.

As a result, people tend to over-log defensively, which creates a different type of problem where you end up with a volume of logs that add noise and make tracing harder while often still missing the fields that actually mattered.

But what if I told you that there was a better way, and you didn't have to rely on logging at all? This is precisely what jolt.image lets us do. Instead of choosing what to record ahead of time, you can just dump the whole state of the program at the time of the error to disk. Then you can just copy the file to your local machine, load up the state in the REPL and then poke around in it to see what happened.

Here's what that looks like in practice. To dump the state when an error happens, all you have to do is call image/dump! in the exception handler:

(try
  (process-batch! batch)
  (catch Exception e
    (image/dump! (str "crash-" (random-uuid) ".jimg")
                 {:error   (Throwable->map e)
                  :batch   batch
                  :pending @work-queue})
    (throw e)))

Or skip the enumeration entirely and take the whole program:

(catch Exception e
  (image/dump-world! "crash.jimg")
  (throw e))

dump-world! walks the var table and writes every data var's root so that nothing in your code has to declare what its state consists of up front. The image is architecture agnostic, so an image written on an arm64 server will restore fine on your x86-64 desktop. Once you've copied the file to your local machine, you just have to open it in a REPL:

$ jolt repl
user=> (require '[jolt.image :as image])
user=> (image/restore-world! "crash.jimg")
412
user=> (filter #(nil? (:price %)) @app.core/current-batch)
({:id 4182, :sku "B-77", ...})

What comes back are the values that were present in memory at the time the problem occurred. Maps, records, cycles and shared structure are all intact with their respective metadata attached, and functions come back callable as well. A named function resolves to the live one, while an anonymous closure travels as its source form along with its captured values to get compiled back on restore. So you can actually call the function that failed, on the data that failed in the REPL and see exactly what went wrong. You can now ask a strictly larger set of questions than any log file can answer, and you didn't have to know any of them in advance.

You can think of the program image as a black box recorder. Just like an aircraft stores the instrument states to allow investigators to decide afterwards what to look at, a program image gives you all the information needed to debug the problem.

At this point, a keen reader might ask how this approach handles open resources such as a socket or a file port that can't be serialized. The approach I landed on was to have dump-world! write them as stub records by default. Once the image is restored, you can list them by calling (image/stubs), and either register a resolver that reopens them or swap live values in by hand from the REPL using (image/register-stub-resolver! kind-or-pred f).

One limitation is that a closure over a compile-time constant refuses to dump because the constant gets folded into compiled code and can't be recovered while the stored source still needs it. Closures built by partial and comp have the same problem for a related reason. The image's header is also checked on read so that you don't get stale data from an incompatible build. While dump-world! will dump everything it can, dump! is strict by default, and names the path to the offending object instead of writing something subtly incomplete. It's useful in cases where you want to be explicit about having the whole state available.

Sending a program to someone

While post-mortem debugging is an obvious use case, another interesting application is to facilitate collaboration. An image is a program state you can hand to another person which opens up some intriguing possibilities.

For teaching, that means you can put someone directly into the middle of a running system, with real data loaded, without them having to build or seed anything. Imagine being able to say "here's the image with the pipeline implemented halfway through; go look at stage 2 in it." For a bug report, it means a colleague can reproduce your problem exactly, because they are just loading your state. They can poke at it, change something, dump it again, and send it back to you.

It's a whole different relationship with a program than what we're used to since, traditionally, source code is treated as the artifact that we pass around. Jolt moves the needle closer to how Smalltalk and the Lisp machines worked, and how save-lisp-and-die still works in Common Lisp today. The program is a live thing you keep that can evolve over time as opposed to being a recipe you can run.

Durable session state

Another interesting application would be use cases such as desktop apps, where the user can save their session and then get back to where they left off when restarting the app. I built a TodoMVC example illustrating what that looks like in practice. You can click around, add or modify tasks, then dump the state and reload it in a fresh instance.

Portable Scheme backends

Originally, my goal for Jolt was to build a Clojure implementation on Chez Scheme. However, Jack Rusher pointed out that it would be possible to factor out a portable runtime and compiler making Chez just one target among several. Gambit was the obvious choice for a second host since it has a JavaScript backend making it possible to run Jolt in a browser. If you visit the official site you can now try playing in an interactive REPL, which is Jolt running in the browser.

The refactor split the Scheme part of the compiler into three distinct layers. The core is written in portable Scheme implementing collections, sequences, the reader, the printer, vars, multimethods. This is ordinary Scheme that any serious implementation runs unchanged.

Next there is the adapter contract where the host shows through. Every host capability goes through an sa-* entry point, and the contract file lists 72 names grouped into tiers: system (clocks, environment, exit), threads, eval, introspect (continuation frames for backtraces), ffi, native-compile, and image. A target can either implement a tier or degrade it honestly so that an absent capability raises a message-carrying error or returns empty. That last property is key for making partial ports actually usable. The Gambit version runs with ffi, native-compile and image degraded, and declares its capabilities explicitly.

Finally, target-owned files are the two pieces nobody can share. These include the adapter itself, and the hash kernel. For example, the Chez version uses unsafe fixnum operations that other Schemes spell differently. On the compiler side, per-target differences go through a primitive table with the main entry being the unsafe-op prefix, so a target that maps it to the empty string simply gets checked operations everywhere stating whether it's safe, portable, or slower.

The dialect-specific work is both smaller than you'd guess and duller than you'd hope. Most of it involves mapping records to their parent types, the hashtable API, fx operation spellings, the shape of error objects, and making the hash function produce bit-identical output to Chez. The Gambit port weighs in at about 6,000 lines, and a good fraction of it, including the seed itself, is generated on Chez rather than having to be written from scratch. Cross-minting the seed from a known working build is the trick that keeps a new target from having to bootstrap itself.

Why more than one Scheme is worth having

The immediate payoff is reach. Gambit compiles to a single JavaScript file, so Jolt now runs in a browser allowing for a REPL on the front page of the site using the real compiler and standard library evaluating directly on the client. It's not terribly fast, but works for a demo.

Scheme is a whole family of languages with different dialects each optimizing for different use cases. So, the deeper payoff here is in opening up an ecosystem of implementations that made different bets. All of them share core language semantics, but each dialect puts its own twist on the language providing a runtime optimized for different use cases. Jolt can now piggyback on this whole ecosystem providing a Clojure layer on top.

Chez makes an excellent default since it's fast, relatively small, feature-rich, with real threads, an FFI, and native compilation. Gambit gets you to JavaScript and C. Meanwhile, a whole-program optimizing compiler in the Stalin lineage is a different proposition entirely; it affords aggressive closure and type analysis producing tiny output that suits a small binary shipped to a constrained device where startup and footprint dominate. Such a compiler typically has no runtime eval at all, which sounds disqualifying until you notice that the seed is already cross-minted on Chez. This way the compiler can live on one Scheme while the emitted program runs on another.

The key part here is that the program is the same Clojure regardless of which host you target. What changes are the capabilities available, which have to be stated in the contract providing clear boundaries for what can be expressed by each runtime. Thanks to many existing Scheme implementations, the same code can now run on a server, in a browser tab, as a tiny static binary, or get embedded in existing programs.

A program shouldn't be trapped in the process that started it, nor should it be married to the runtime it was first compiled for. Lisps were always meant to be flexible, and Jolt embraces this philosophy.

Permalink

Call for Proposals. July 2026 Survey

Greetings folks!

Clojurists Together is pleased to announce that we are opening our Q3 2026 funding round for Clojure Open Source Projects. Applications will be accepted through the 24th of August 2026 (midnight Pacific Time). We are looking forward to reviewing your proposals! More information and the application can be found here.

We will be awarding up to $29,000 USD for a total of 4-5 projects. The $2k funding tier is for experimental projects or smaller proposals, whereas the $9k tier is for those that are more established. Projects generally run 3 months, however, the $9K projects can run between 3 and 12 months as needed. We expect projects to start around mid-September 2026.

A BIG THANKS to all our members for your continued support. We also want to encourage you to reach out to your colleagues and companies to join Clojurists Together so that we can fund EVEN MORE great projects throughout the year.

And Now the Survey…

We surveyed members in July to find out what what issues were top of mind and the types of initiatives they would like us to focus on for this round of funding. While our goal for the survey is to surface the broadest, most consistently-raised themes, it is not meant to be prescriptive, as we are always interested in nurturing new ideas and approaches. As always, there was a lot of great input and we hope it will be useful in informing your project proposals.

Select Themes:

Demonstrated Impact of Past Funding Roughly three-quarters of respondents draw on CJT-funded work on a near-daily to weekly basis, with the remainder spread across occasional, project-dependent, or passive-interest use. This is strong evidence that past funding has produced tools and libraries with real, sustained utilization — which is why we exist!

Adoption and Growth of Clojure Continue to be of Concern. This theme is closely linked to employment challenges cited. These themes require broader or more strategic solutions that may be best addressed by the Core Team. However, Clojurists Together can support smaller and more focused efforts. Some ideas include:

• Facilitating discovery of existing tools and documentation currently residing in disparate spaces so that both new and seasoned users can quickly reference new and updated content.
• Support for developers willing to present the Clojure story to their professional or tech-adjacent communities.
• In addition to the mentoring already underway, development of self-directed training that is easy and cost-effective to scale.
  
  
  

About 88% of Members Surveyed are Using AI tools in some capacity - with members calling out the need for Clojure-specific support.

Developer Experience Tools are Respondents' Top Priority for Clojure and ClojureScript with Error Messaging identified in the top 4 for both. There is plenty of work that needs to get done in these categories. The good news is that the Clojure core team along with the CLI Task Force is actively working on improving the user experience of the command-line tooling. More to come in the few months….

July 2026 Survey Summary:

This summary includes a selection of member comments.

1. Who We Heard From: Respondent Profile & Platform Usage

Before weighing the themes below, it’s worth noting who answered this survey. The respondent base skews heavily toward long-tenured Clojure developers, and server-side JVM use dominates how members actually deploy Clojure.

Tenure

Of 41 respondents, the overwhelming majority have used Clojure for a long time:

*87.8% reported 6 years or more of Clojure experience, with only a small remainder spread across 1–5 years and under 1 year combined. This is an important caveat for every other theme in this report: the feedback is disproportionately the voice of veteran users, not newcomers.

Mentoring vs. beginner status

Respondents identified overwhelmingly as mentors rather than newcomers. Combined with the tenure data above, this confirms the survey sample is dominated by experienced members who are already invested in growing the community.

Platforms targeted (41 respondents, select-many)

*Clojure on the JVM server remains the dominant deployment target by far (97.6%), with ClojureScript in the browser a strong secondary use case (68.3%). Notably, ClojureDart shows minimal current usage (4.9%) despite being repeatedly and enthusiastically flagged in the open-ended “magic wand” and ecosystem-support answers (Section 8) — a gap between current adoption and member enthusiasm worth factoring into funding decisions.

2. Quantitative Priorities: Areas for Improvement

Members were asked which areas of Clojure and ClojureScript most need improvement (select-many). Developer Experience Tools ranked #1 in both languages, and data/error-handling concerns dominate the Clojure-specific results.

Clojure (35 respondents)

ClojureScript (26 respondents)

Write-in responses (ClojureScript, 1 mention / 3.8% each): reduced or near-zero NPM dependency, ability to do full-stack development without a separate backend, less reliance on NPM generally, AI-supported development, and “N/A, I don’t use ClojureScript."

3. Theme: Developer Experience Is the Top Tooling Priority

“Developer Experience Tools” was the single highest-ranked improvement area for both Clojure (45.7%) and ClojureScript (42.3%), and it recurs throughout the open-ended answers as well. For Clojure specifically, error messages and data analysis/processing frameworks tied for second place (40% each) — well ahead of documentation, IDE support, and debuggers. For ClojureScript, build tooling (26.9%) and documentation (23.1%) stand out as the next-biggest gaps after developer experience, suggesting the ClojureScript toolchain still feels heavier to maintain than the Clojure one.

4. Theme: 88% of member respondents use AI tools — but Clojure-Specific Support Is Lagging

Claude Code was named most often in the open-ended answers, alongside Cursor, Copilot, Gemini, Aider, ECA, bhauman’s MCP Server, Amazon Kiro (via CP in IntelliJ), Several members pointed specifically to REPL-driven, Clojure-aware tooling (e.g., clojure-mcp / clj-nrepl-eval integrations) as the feature that makes AI genuinely useful for Clojure — but also noted that generic AI tools frequently mishandle Clojure’s syntax (parentheses/brackets) and that few tools understand Clojure idioms well.

What members use AI for (38 respondents, select-many)

• One respondent noted that we should include ‘Security Reviews’ and ‘Agentic Coding’ as an option in future surveys.

Sub-themes:

• Concern that in an increasingly AI-generated-code world, Clojure loses its traditional advantages (conciseness, expressiveness) if hiring/tooling decisions default to more “mainstream” languages the models are best trained on.
• Mixed sentiment: alongside enthusiastic adopters, a vocal minority is skeptical of or opposed to AI-driven development altogether.

Supporting comments:

“clj-nrepl-eval from bhauman/clojure-mcp-light is central. REPL is the killer feature for AI assisted Clojure dev compared to other languages.”

“I wish ECA would work well with local AI models using Ollama. I dont want to use big tech companies… I dont trust them.”

“Each client provides a chatbot, which might be inside the IDE but I have no idea how to make it work with Clojure and not mess up the brackets.”

“The sad truth is in an ever increasing LLM driven development world there is less incentive to use Clojure than something like Rust. All the downsides in making that switch are alleviated if LLMs are doing the coding for you.”

“I like to use free models and run them locally, if there is a large amount of repeatable and we’ll defined work to do it can be good, like a refactor. Sometimes it’s good to use to test an idea or prototype I would not have time to do otherwise. I generally take it that if an llm agent can do something then it’s likely not that hard to do. If the llm struggles on something that should be simple it’s interesting to find out why.”

“I rarely write code “by hand” anymore. My workflow is primarily prompting various coding agents (Claude code, codex, open code using models via open router) and reviewing their output, but rarely dropping into the editor myself.”

“LLMs are a scourge upon the human race with no actual profitability, and I hope every day to see this bubble finally pop.” “It has basically taking over everything. Agent harnesses.”

“While our company doesn’t forbid the usage of (generative) AI tooling, it doesn’t encourage it either. It is up to each individual developer to use it or not. But the agreed contract is that whatever code a developer produces using AI tooling must meet the same established conventions (e.g., code style, idioms to be used, code and architecture estructure, etc) and quality levels of code produced by human developers. And that the code pushed by that developer must be owned by him/her, and that it is his/her own responsability to maintain, and fix if needed.”

5. Theme: Perception, Advocacy & the “Is Clojure Dying?” Narrative

The single most repeated theme in response to “the biggest challenge facing Clojure developers” was one of perception rather than technology: Clojure is widely seen — inside and outside the community — as niche, shrinking, or even dead, which makes it harder to justify on new projects, hire for, or pitch to business leadership and investors.

Supporting comments:

“It is a challenge using it on new projects and justifying it over mainstream alternatives. The biggest complaint I always hear is ‘how will we find developers’. " I think this is more of a perception challenge, the easiest way to reply would be to just point to success stories, or a very visible app or product.”

“Outreach. Many people think language is dead”

“In the world of startups Clojure is generally seen as a niche language and therefore a hindrance to selling a company and maybe even also just to getting funding (sometimes). A friend of mine is the CTO of a startup that was in talks for an acquisition, and the company backed out of the deal because Clojure was used.”

Members suggested amplifying success stories and visible production use cases, supporting community “influencers” and advocates, and funding outreach/evangelism efforts aimed at both developers and business decision-makers.

6. Theme: Jobs & Commercial Adoption

Closely tied to the perception theme is a concrete, recurring concern about the Clojure job market: too few open positions, hiring managers who default to languages with larger corporate backing, and no “gateway” framework (comparable to Rails or ML Frameworks) that pulls new developers into the language the way it once did.

Supporting comments:

“Lack of job opportunities. Big companies are quite skeptical about non-mainstream languages.”

“The people that make hiring decisions view developers as fungible goods, which then leads them to choose languages based on which one they believe will have the lowest salary/hourly cost which tend to be the languages with large corporate backers, and Clojure is not one of those languages.”

7. Theme: Library, Framework & Supply-Chain Health

Members flagged specific maintenance gaps in the ecosystem: unmaintained libraries with no clear owner, documentation gaps in widely-used projects. Support for projects, tools and platforms cited: (5) CIDER; (4) Malli; (3) ClojureDart, re-frame, Pathom, Babashka, Jank; (2) reagent, Reitit, Shadow-CLJS, Scicloj; (1) HugSQL, clj-kondo, duct, nrepl, datalevin, datahike, Fulcro, Datascript, Glojure, Grain, eca, Telemere, rama, replicant, http-kit, Clojure Civitas, Clay, Datastar, calva, ring, figwheel-main.

Supporting comments:

“Clojurists Together could act as a broker for finding maintainers for out-of-support libraries.”

“Some great projects could use better documentation; two examples of amazing libraries that could use better documentation being Malli and Specter.”

“Port all major libraries to tools.deps”

• Development of a Clojure language spec given the number of Clojure-like languages now emerging.
  

8. Theme: Community Health, Governance & Connection

The strength, generosity, and openness of the community is seen as a core strength - along with its engineering rigor. Members feedback included a desire for stronger central coordination, more inclusive and welcoming spaces, and a return to in-person connection.

Supporting comments:

“Clojure’s culture of engineering rigour is unmatched in the industry. I think clojurists’ attention to detail and care for their craft is a huge advantage right now in this age of slop and endless downtime. Also the community is warm, welcoming, and friendly, which is not the case anywhere else I “hang out” online”.

“Once people start using Clojure, they usually love it. REPL is great, Clojure is very fast, well designed language, also JVM interop has improved.”

“Resilience of Clojure communities and their support structures seems to be a challenge… In community spaces, some divide and disagreement often appear, and not everybody feels at home and supported.”

“I would create a Clojure foundation that would lead central decision-making for the continued growth and development of the language.” “Community in one digital place, a Clojure language spec.”

“There would be in-person meetups again!”, “That everyone and all events we’re in the same country :) I miss not being able to go everywhere!”

“I don’t know if this counts or not, but my favorite part of the ecosystem is how stable it is. I love that library updates rarely, if ever, break existing code. Having dealt with the churn and instability of the JS and Rails ecosystems, the fact that updates so rarely force me to do tedious work is a godsend.”

9. Theme: Growth, Outreach & Training

When asked directly what areas of the ecosystem need support, “advocacy,” “outreach,” “evangelism,” “mentoring,” and “community and growth” were named repeatedly and independently — more often than any single technical gap — reinforcing that members see growing and renewing the community as at least as urgent as improving the tools themselves.

Permalink

LISP — ภาษา 67 ปีก่อน ที่ยังมีชีวิตอยู่ในทุกภาษาที่คุณเขียน

คุณเขียน if-else ทุกวัน
คุณรัน code ใน terminal แล้ว REPL มันตอบกลับมา
คุณ lambda ใน Python, arrow function ใน JavaScript, closure ใน Rust

— ทั้งหมดนี้ เกิดจากภาษา LISP

และที่น่าทึ่งคือ... LISP ไม่เคยถูก planned ให้เป็นภาษาโปรแกรมด้วยซ้ำ

กระดาษแผ่นเดียวที่เปลี่ยนโลก

1958 — John McCarthy เริ่มพัฒนาแนวคิด LISP ที่ MIT
เมษายน 1960 — McCarthy วัย 32 ตีพิมพ์ paper ใน Communications of the ACM (vol. 3, หน้า 184-195)

"Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I"

ใน paper 12 หน้านี้ McCarthy เสนอไอเดียของภาษาโปรแกรมที่:

• จัดการกับสัญลักษณ์ (symbols) ไม่ใช่แค่ตัวเลข
• ใช้ recursion เป็นตัวควบคุมการทำงาน แทนการใช้ loop
• เก็บ function เป็น first-class citizen — ส่งต่อกันได้เหมือนข้อมูล

McCarthy เขียนมันขึ้นมาเป็น ทฤษฎีทางคณิตศาสตร์ — ไม่ได้ตั้งใจ implement

แต่ก่อน paper จะตีพิมพ์ — ระหว่างปี 1958-59 Steve Russell นักศึกษา grad student อ่าน manuscript

"I told him, 'Steve, why don't you program this eval?' and he said to me, 'Oh, I misread what you meant. I thought you meant I should implement the interpreter.'"

— John McCarthy, ACM interview

นักศึกษา เขียน interpreter ให้ทฤษฎีของอาจารย์ — และภาษา LISP ก็เกิด

code ตัวแรกที่ Russell เขียน ใช้เวลาแค่ 2-3 วัน (ตัว LISP 1.5 Programmer's Manual ฉบับเต็มออกตามมาทีหลัง ในปี 1962)

สิ่งที่ LISP สร้างไว้ (และเราใช้มันทุกวัน)

1. if-then-else

ย้อนไปปี 1958 — ภาษาส่วนใหญ่มีแค่ GOTO กับ branch แบบ assembly

McCarthy ให้กำเนิด cond (conditional expression) — จุดเริ่มต้นของ if-else ที่เราเห็นแทบจะในทุกภาษา

(cond ((< x 0) 'negative)
      ((= x 0) 'zero)
      (t 'positive))

C, Java, Python, JavaScript, Go, Rust — ภาษาเหล่านี้ได้รับมรดกนี้มาหมด Conditional branching ในทุกภาษาสมัยใหม่มีโครงสร้างแบบเดียวกับที่ McCarthy คิดไว้ตั้งแต่ก่อนมนุษย์ไปดวงจันทร์

2. Garbage Collection

ก่อน LISP — programmer จัดการ memory เอง 100% ทุกบรรทัดของ malloc และ free

LISP สร้าง garbage collection ตัวแรกของโลก — ต้นฉบับคือ mark-and-sweep algorithm (implement โดย Daniel Edwards นักศึกษา MIT)

ทุกวันนี้ GC คือ default ในเกือบทุกภาษา high-level — Java, Python, JavaScript, Go, C#, Ruby ล้วนใช้แนวคิดนี้ต่อยอด

3. Lambda — function เป็นข้อมูล

(lambda (x) (* x x))

LISP ทำให้ function เป็น first-class citizen — ส่ง function เป็น parameter ได้, return function ได้, เก็บลง variable ได้เหมือนเป็น string หรือ integer

นี่คือต้นทางของ:

• JavaScript — arrow function (x) => x * x (Brendan Eich ถูกจ้างไป Netscape เพื่อทำ Scheme ใน browser — แต่ management เปลี่ยนใจให้ syntax เหมือน Java)
• Python — lambda x: x * x
• Ruby — blocks, procs, lambdas (Matz บอกว่า "Ruby was a Lisp originally, in theory")
• Java — lambda expressions (Java 8, 2014 — ใช้เวลา 19 ปีกว่าจะตามทัน)
• C++ — lambda (C++11)

4. REPL — ปรัชญา "คุยกับภาษา"

Read-Eval-Print Loop — LISP ให้กำเนิดมันในทศวรรษ 1960s

ก่อนหน้านั้น: เขียน code → compile → run → debug → repeat
หลังจากนั้น: พิมพ์ expression → กด enter → เห็นผลทันที

ทุกวันนี้ถ้าคุณเปิด Python REPL (>>>), Node.js console, Ruby IRB, Chrome DevTools, Rust Playground, หรือ Elixir IEx — คุณกำลังนั่งอยู่ในห้องเรียนเดียวกับ programmer LISP เมื่อ 60 ปีที่แล้ว

5. Homoiconicity — code = data

'(+ 1 2)          ; ← นี่คือ list
(eval '(+ 1 2))   ; ← นี่คือ code ที่รัน list

LISP เขียนด้วย... LISP — code กับ data ใช้โครงสร้างเดียวกัน (S-expression)

แปลว่า โปรแกรมแก้โปรแกรมตัวเองได้ — ไม่ต้องใช้ parser แยก AST, ไม่ต้องเขียน transformer

นี่คือรากฐานของ macro system ที่ทรงพลังที่สุดในสายภาษาโปรแกรม

ไม่มีภาษาไหนทำได้เต็มระบบเท่า LISP — แต่แนวคิด "code as data" ไปอยู่ใน:

• Elixir — macro (รับ AST, คืน AST)
• Julia — macro + multiple dispatch ("We want a language that's homoiconic, with true macros like Lisp" — Julia manifesto, 2012)
• Rust — macro system (proc macro)
• Clojure — macro (ของ Clojure ก็คือ LISP dialect นั่นเอง)

มรดกจาก LISP

LISP (1958)
├── Scheme (1975) — minimalist, lexical scoping
│   ├── JavaScript (1995) — Brendan Eich ตั้งใจทำ Scheme-like ใน browser
│   │   └── arrow functions, closure, first-class functions
│   └── Racket (1995) — ภาษาเพื่อการสอนและการวิจัย
├── Common Lisp (1984) — ภาคอุตสาหกรรม, pragmatic
│   └── Emacs Lisp (1985) — editor scripting (GNU Emacs)
├── Clojure (2007) — LISP บน JVM, immutable by default
│   └── จุดประกาย functional programming ในโลก enterprise
└── Python, Ruby, Elixir, Julia, Rust, Swift — ทุกภาษาเอาแนวคิด LISP ไปปรับใช้

ทำไม LISP ถึงไม่ใช่ภาษาหลักวันนี้

ทั้งที่สร้างนวัตกรรมเกือบทุกอย่างที่เราใช้ — ทำไม LISP ถึงไม่ชนะ?

Paul Graham (ผู้ก่อตั้ง Y Combinator, แฟนพันธุ์แท้ LISP) อธิบายไว้ใน essay "Beating the Averages":

1. Syntax ต่างจากทุกอย่าง — วงเล็บทำให้คนกลัว
2. ไม่มี standard library ในยุคแรก — Common Lisp มากอบกู้ในปี 1984 แต่ช้าไปแล้ว
3. AI Winter — LISP ผูกกับ AI research; เมื่องบ AI หาย LISP ก็หายด้วย
4. Tooling — compiler ช้า, IDE ห่วย (เทียบกับ Visual Studio ยุค 90s)

Graham ยืนยันว่า:

"Lisp is a language that was discovered, not invented."

ฝั่งนักวิจารณ์ LISP (รวมถึงคนที่เคยใช้ใน production แล้วเปลี่ยนไปภาษาอื่น) ชี้ปัญหาเพิ่มเติมที่ Graham ไม่พูดถึง:

1. Dynamic typing ในระบบใหญ่ — refactoring ยาก, compiler จับ error ได้น้อย, debugging ใช้เวลานานใน codebase ระดับแสนบรรทัด
2. Performance จนถึงยุค 2000s — ก่อน SBCL และ Chez Scheme, LISP runtime บน hardware ยุคนั้นช้ากว่า C/C++ มากโดยเฉพาะงาน numerical
3. Fragmentation — Common Lisp กับ Scheme ไม่เคยตกลงกันได้; community แตกเป็นเสี่ยง ๆ แทนที่จะรวมพลัง

สรุป: ไม่มีสาเหตุเดียว — มันคือ perfect storm ของ syntax ต่าง + เกิดผิดเวลา + community แตก + ไม่มี corporate sponsor (ต่างจาก Java ที่ Sun ทุ่ม, C# ที่ Microsoft ทุ่ม)

มรดกที่มองไม่เห็น

บทความนี้ไม่ได้ตั้งใจจะบอกว่า "คุณควรเขียน LISP"

แต่ทุกครั้งที่คุณ:

numbers = [1, 2, 3]
squared = list(map(lambda x: x * x, numbers))

const result = data
  .filter(x => x.active)
  .map(x => x.value);

let squared: Vec<_> = numbers.iter().map(|x| x * x).collect();

— คุณกำลังเขียน LISP โดยไม่รู้ตัว

อ่านเพิ่ม

• McCarthy's Original 1960 Paper — "Recursive Functions of Symbolic Expressions"
• Paul Graham — "Beating the Averages"
• Paul Graham — "Revenge of the Nerds"
• Structure and Interpretation of Computer Programs (SICP) — ตำรา Scheme ในตำนานของ MIT
• Rich Hickey — "Simple Made Easy" — ปรัชญาเบื้องหลัง Clojure

📅 สิงหาคม 2026 | ⚠️ ตรวจสอบข้อมูล ณ วันที่เขียน

Permalink

Clojure Book, Massive Quality Update

I am very happy to announce that my Clojure book has received a massive update. Every line of my book was proofread and corrected by AI, and I read it to make sure that the AI corrections were right.

English is not my native language. Hence, before the AI era, the book I released had many grammatical errors. Now that AI has corrected them, my book is really good to read. In many places, AI has made my book terse and to the point. I’m very happy about it.

This AI proofread also prompted me to proofread the book myself so that no errors would slip by. I have done my best. What was done by AI in less than two hours took me more than three weeks to proofread. There were only a very small number of mistakes that AI made, which I corrected. The book is now far, far better.

I’m sure even Western / English audiences will find my book very enjoyable to read now.

Highlights of this new release are:

• The book is completely proofread twice, once by AI and then by me.
• I have released a video for the section “Creating Your Own Libraries.”
• The book is available in all formats: online, PDF, EPUB, and on Amazon.

I hope you all read my book. Please suggest corrections. And please tell me what material I could add so that my book becomes much better.

I thank all those who have suggested things to improve my book. Clojure has given me a lot, and I’m ready to give back as much as I can. This book is one tiny effort.

Permalink

Looking for my next role

Hi, I am Niki, and I am looking for my next role.

I am a π-shaped specialist:

• A software developer,
• A product designer specializing in UI and UX.

My mission is simplicity, performance and software that helps people.

Software Developer

I have worked with and built databases:

• Created DataScript, which inspired a whole series of databases (Datalevin, Datahike, re-db) and powered projects such as Roam Research, Athens Research, Logseq;
• Optimized Firebase-backed storage in Roam Research;
• Built a Postgres-backed DBaaS at Instant DB.

Sync engines:

• Postgres to browser at Instant DB;
• DataScript to Datomic at Cognician;
• Helped with SwarmJS/RON CRDTs.

Frontend:

• I’ve been building browser-based apps since 2005;
• Built my first web framework in 2010;
• Created a popular React wrapper for ClojureScript in 2014;
• Built my own virtual DOM implementation in 2023.

Performance:

• Fast EDN is 6x faster than Clojure’s standard EDN parser, on par with JSON on JVM;
• Reduced Roam Research DB load times from minutes to seconds.

UI/UX

I write Grumpy.Website, a blog on UI/UX with 2,000+ subscribers.

My articles have been referenced by Notion, ATP.fm, Daring Fireball and Marcin Wichary.

Earlier this year, I wrote a widely shared critique of excessive menu icons in macOS Tahoe. Apple later removed many of the icons discussed in the article.

Product Design

In 2024, I launched AlleKinos.de, which quickly reached 1,500 daily visitors without marketing or SEO.

I created Fira Code, which became one of the world’s most popular programming fonts.

My Clojure Sublimed extension became the go-to Clojure development environment in Sublime Text.

I also created many other smaller products, libraries, fonts, color schemes, all available on my GitHub.

The Rest

If you want to dive deeper, here’s the usual stuff:

• Projects
• Talks and podcasts
• LinkedIn
• GitHub

I also made a two-page PDF CV:

What am I looking for?

• Full-time or contract,
• Remote or Berlin,
• Individual contributor,
• Small, closely collaborating team,
• Ambitious product.

If you are working on a compiler, a database, an IDE, a programming language or another technically ambitious product, touching graphics, typography, algorithms, low-level programming, and you think my experience can help, let’s talk: niki@tonsky.me.

Permalink

Statistics made simple

I have a weird relationship with statistics: on one hand, I try not to look at it too often. Maybe once or twice a year. It’s because analytics is not actionable: what difference does it make if a thousand people saw my article or ten thousand?

I mean, sure, you might try to guess people’s tastes and only write about what’s popular, but that will destroy your soul pretty quickly.

On the other hand, I feel nervous when something is not accounted for, recorded, or saved for future reference. I might not need it now, but what if ten years later I change my mind?

Seeing your readers also helps to know you are not writing into the void. So I really don’t need much, something very basic: the number of readers per day/per article, maybe, would be enough.

Final piece of the puzzle: I self-host my web projects, and I use an old-fashioned web server instead of delegating that task to Nginx.

Static sites are popular and for a good reason: they are fast, lightweight, and fulfil their function. I, on the other hand, might have an unfinished gestalt or two: I want to feel the full power of the computer when serving my web pages, to be able to do fun stuff that is beyond static pages. I need that freedom that comes with a full programming language at your disposal. I want to program my own web server (in Clojure, sorry everybody else).

Existing options

All this led me on a quest for a statistics solution that would uniquely fit my needs. Google Analytics was out: bloated, not privacy-friendly, terrible UX, Google is evil, etc.

What is going on?

Some other JS solution might’ve been possible, but still questionable: SaaS? Paid? Will they be around in 10 years? Self-host? Are their cookies GDPR-compliant? How to count RSS feeds?

Nginx has access logs, so I tried server-side statistics that feed off those (namely, Goatcounter). Easy to set up, but then I needed to create domains for them, manage accounts, monitor the process, and it wasn’t even performant enough on my server/request volume!

My solution

So I ended up building my own. You are welcome to join, if your constraints are similar to mine. This is how it looks:

It’s pretty basic, but does a few things that were important to me.

Setup

Extremely easy to set up. And I mean it as a feature.

Just add our middleware to your Ring stack and get everything automatically: collecting and reporting.

(def app
  (-> routes
    ...
    (ring.middleware.params/wrap-params)
    (ring.middleware.cookies/wrap-cookies)
    ...
    (clj-simple-stats.core/wrap-stats))) ;; <-- just add this

It’s zero setup in the best sense: nothing to configure, nothing to monitor, minimal dependency. It starts to work immediately and doesn’t ask anything from you, ever.

See, you already have your web server, why not reuse all the setup you did for it anyway?

Request types

We distinguish between request types. In my case, I am only interested in live people, so I count them separately from RSS feed requests, favicon requests, redirects, wrong URLs, and bots. Bots are particularly active these days. Gotta get that AI training data from somewhere.

RSS feeds are live people in a sense, so extra work was done to count them properly. Same reader requesting feed.xml 100 times in a day will only count as one request.

Hosted RSS readers often report user count in User-Agent, like this:

Feedly/1.0 (+http://www.feedly.com/fetcher.html; 457 subscribers; like FeedFetcher-Google)

Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 6 subscribers)

Feedbin feed-id:1373711 - 142 subscribers

My personal respect and thank you to everybody on this list. I see you.

Graphs

Visualization is important, and so is choosing the correct graph type. This is wrong:

Continuous line suggests interpolation. It reads like between 1 visit at 5am and 11 visits at 6am there were points with 2, 3, 5, 9 visits in between. Maybe 5.5 visits even! That is not the case.

This is how a semantically correct version of that graph should look:

Some attention was also paid to having reasonable labels on axes. You won’t see something like 117, 234, 10875. We always choose round numbers appropriate to the scale: 100, 200, 500, 1K etc.

Goes without saying that all graphs have the same vertical scale and syncrhonized horizontal scroll.

Insights

We don’t offer much (as I don’t need much), but you can narrow reports down by page, query, referrer, user agent, and any date slice.

Not implemented (yet)

It would be nice to have some insights into “What was this spike caused by?”

Some basic breakdown by country would be nice. I do have IP addresses (for what they are worth), but I need a way to package GeoIP into some reasonable size (under 1 Mb, preferably; some loss of resolution is okay).

Finally, one thing I am really interested in is “Who wrote about me?” I do have referrers, only question is how to separate signal from noise.

Performance. DuckDB is a sport: it compresses data and runs column queries, so storing extra columns per row doesn’t affect query performance. Still, each dashboard hit is a query across the entire database, which at this moment (~3 years of data) sits around 600 MiB. I definitely need to look into building some pre-calculated aggregates.

One day.

How to get

Head to github.com/tonsky/clj-simple-stats and follow the instructions:

Let me know what you think! Is it usable to you? What could be improved?

Permalink

I Built a Clojure That Runs on Go(cljgo) . Then I Made It Run a Real AI Agent.

How dogfooding cljgo with a seven-language agent library turned into a Clojure port with zero reader conditionals — and a portability seam called koine that keeps the ugly parts somewhere else.

Building a language host(cljgo) is easy to fool yourself about. It runs your test suite, it runs your toy programs, and you conclude it works. So I went looking for something big enough to actually hurt.

I already had one: toolnexus, a library I maintain that gives any LLM real tool-calling — MCP servers, agent skills, your own functions, sub-agents — already ported across six languages (JavaScript, Python, Go, Java, C#, Elixir), all pinned to one behavioral contract. Porting it to Clojure would give Clojure developers something they didn’t have. Running that same port on cljgo would tell me whether cljgo is real.

The rule that made it interesting: no reader conditionals

The obvious way to write code for two hosts is #?(:clj ... :cljgo ...) sprinkled wherever they differ. I refused to do that, for a selfish reason and a principled one.

The selfish one: every conditional is a place where the two hosts can silently drift apart, and drift is precisely the bug this project exists to prevent.

The principled one: toolnexus should not know cljgo exists. A library about LLM tool-calling has no business containing opinions about which Clojure you run. If it does, then every future host — Babashka, ClojureCLR, whatever comes next — means editing the library.

So the host differences went somewhere else: koine, a small portability seam. File I/O, subprocesses, HTTP, JSON, environment, time, string handling above the BMP — anything the two hosts disagree about lives there and nowhere else. koine is where the conditionals go; toolnexus stays pure Clojure.

The result is one source tree with zero reader conditionals, verified in five execution modes — JVM main, JVM REPL, cljgo AOT, cljgo interpreted, cljgo REPL — with a CI gate that fails if the five disagree by a single assertion. Currently 395 tests, 1,614 assertions, five identical verdicts.

That means the whole dependency story for a Clojure developer is one line:

;; deps.edn — koine comes along with it
{:deps {net.clojars.muthuishere/toolnexus {:mvn/version "0.13.0"}}}

Any Clojure developer can use this and never think about cljgo. And the one cljgo developer — me, so far — gets the same library, unmodified. That’s the arrangement I wanted.

So what is toolnexus?

It is not a framework, and there is no runtime to deploy. It is a library built on one observation:

An MCP server tool, an agent skill, a function you wrote, an HTTP endpoint, a shell command, and a remote agent are the same thing to an LLM — a named, described, schema’d callable.

toolnexus unifies all of those behind one Tool interface, emits the schema in OpenAI / Anthropic / Gemini formats, and ships a client with the tool-calling loop already written: parallel and chained calls, hooks, retries, conversation memory, human-in-the-loop suspension, observability.

Concretely, you get:

• an MCP host — point it at an mcp.json and every server's tools (stdio and streamable-HTTP) join your registry
• agent skills — the open SKILL.md standard, loaded on demand rather than stuffed into the prompt
• your own functions, HTTP endpoints, and ten built-in tools (bash, read, write, edit, grep, glob…)
• sub-agents with scoped tool views and hard budgets, and remote A2A agents

All seven ports run the same examples/ fixtures and must produce the same behavior. The Clojure port is held to the same full conformance tier as the other six — it is not the junior member.

Enough architecture. Here is the thing running.

One agent, three tool sources, your terminal

Let’s build the thing every framework promises and rarely shows: an interactive agent you keep asking questions, backed by three different tool sources at once —

• an HTTP tool — one declaration that fetches the official Clojure events page
• an MCP server — @modelcontextprotocol/server-filesystem, spawned over stdio; all fourteen of its tools join the registry automatically
• an agent skill — a SKILL.md telling the model how to report events, loaded on demand

(Skills are the open SKILL.md standard — markdown instructions the model loads when the task matches. The catalog goes into the system prompt; the full text loads only on request. Progressive disclosure, not prompt stuffing.)

The skill — skills/clojure-events/SKILL.md:

---
name: clojure-events
description: Use when asked about Clojure community events, meetups or conferences. How to fetch and report them.
---
# How to report Clojure events

1. Fetch the official events page with your `clojure-events-page` tool — never
   answer from memory, events go stale.
2. List each event you find as: **name** — date — location (or "online").
3. Only report events that are actually on the page. If the page lists none,
   say so.
4. Close with the source link: https://clojure.org/community/events
5. Keep it a plain list, no marketing tone.

The program — src/chat.cljc, the whole thing:

(ns chat
  (:require [clojure.string :as str]
            [koine.env :as env]
            [koine.json :as json]
            [toolnexus.core :as tn]
            [toolnexus.client :as client]
            [toolnexus.http :as http]))

(def events-page
  (http/http-tool
   {:name        "clojure-events-page"
    :description "Fetches the official Clojure community events page and returns it as text."
    :method      :get
    :url         "https://clojure.org/community/events"
    :result-mode "text"}))

(def mcp-config
  {:mcpServers
   {:files {:type    "local"
            :command ["npx" "-y" "@modelcontextprotocol/server-filesystem" "."]
            :timeout 30000}}})

(defn -main [& _]
  (let [tk (tn/build {:skills   "skills"
                      :builtins false
                      :mcp      mcp-config
                      :tools    [events-page]})
        c  (client/create-client
            {:base-url "https://openrouter.ai/api/v1"
             :style    "openai"
             :model    (or (env/get-env "TN_MODEL") "openai/gpt-4o-mini")
             :api-key  (env/get-env "OPENROUTER_API_KEY")})]
    (loop []
      (print "you> ") (flush)
      (let [line (read-line)]
        (cond
          (or (nil? line) (#{"exit" "quit"} (str/trim line)))
          (do (println "bye.") (tn/shutdown! tk)) 
           (str/blank? line) (recur)
          :else
          (let [r (client/ask c line {:toolkit tk :id "cli" :on-event trace})]
            (println (str "agent> " (:text r)))
            (recur)))))))

Note client/ask with an :id — that is conversation memory. Every question you type continues the same conversation, so follow-ups like "which of those is soonest?" just work.

Run it:

$ cd clojure/examples && task clj-ex1

It greets you with the full inventory — one HTTP tool, fourteen files_* MCP tools, the skill loader, the clojure-events skill — suggests a few questions, and waits at you>.

Recorded with vhs, unedited.

The same file on cljgo — no build step

$ task cljgo-ex1

That runs the same chat.cljc through cljgo run — Clojure hosted on Go, interpreted directly, no JVM anywhere. The startup banner is worth reading once: cljgo prints net.clojars.muthuishere/toolnexus 0.13.0 — 16 namespace(s) with no Java interop and notes it pruned org.clojure/clojure ("cljgo IS the Clojure implementation") — that is the static no-interop check happening for real, on the same artifact from Clojars. If you want the 20 ms cold start, cljgo build will also AOT it to a self-contained native binary — but nothing here requires it.

A real session on cljgo, unedited:

Both projects ship in the repo under clojure/examples (clj-ex1 and cljgo-ex1) with the Taskfile; MODEL= overrides the OpenRouter model per run.

The parts the example didn’t show

The demo used three tool sources. The rest of the library is there too, in all seven ports, because parity is the product:

• Sub-agents — an agent is a tool: a registry of agents with scoped toolkit views, teams, and hierarchical budgets (:max-tokens, :max-turns, :max-wall-ms…) that stop loudly, never silently
• Personas — from-dir composes an identity from files (SOUL.md, MEMORY.md…); a memory tool lets an agent edit its own durable notes; start-agent gives it a heartbeat
• Human in the loop — a tool can suspend with a question; the run comes back pending and resumes when your host answers. MCP servers that elicit input mid-call are bridged onto the same mechanism
• Compaction — a :before-llm hook that keeps a long-lived agent inside its context window
• Remote agents — expose your toolkit as an A2A agent with serve, or consume someone else's; either way it is just more tools
• Built-in tools — bash, read, write, edit, grep, glob and friends, with per-tool toggles

The honest numbers

We benchmarked all seven ports plus a dozen competitor frameworks on one machine in one sitting, and published the table as measured. The Clojure port is currently the slowest port over MCP — ~5.5 ms p50 per request against Go’s 0.49 — and we know exactly why: ~2 ms per stdio round-trip in the JSON-RPC path, already logged as the top optimization target. With native tools it runs the same scenario in 1.7 ms, mid-table, ahead of LangChain.js and Mastra. And the number I actually care about: the two hosts agree to within 0.2 ms from the same source file. The parity claim survives being measured under load.

If a library only tells you its winning numbers, it’s an ad. Full table: performance page.

Three gotchas so you don’t find them the hard way

1. The API-key env chain prefers OPENAI_API_KEY for openai-style clients. If that's set and you're calling OpenRouter, pass :api-key explicitly (as the demo does) — otherwise your OpenAI key goes to OpenRouter and you get a confusing 401.
2. cljgo run evaluates top-level forms and doesn't call -main. Add a two-line entry file that requires the ns and calls it — cljgo-ex1's src/run_chat.cljc is exactly that — or build a binary.
3. Write non-ASCII characters directly in source, not as \uXXXX surrogate escapes — the JVM reader assembles the pair, cljgo's doesn't. (Yes, we found that one the interesting way.)

Try it, break it, tell me

If you are a Clojure developer, you can use toolnexus today on the JVM and never think about any of the above:

{:deps {net.clojars.muthuishere/toolnexus {:mvn/version "0.13.0"}}}

If you are curious whether Clojure-on-Go is real, clone the examples and run task cljgo-ex1. That is the same question I was asking, and running it is a better answer than anything I can write here.

• toolnexus — repo · docs, all seven languages · Clojars
• The examples in this article — clojure/examples
• cljgo — repo · Clojure hosted on Go
• koine — repo · the portability seam

Permalink

Leveling Up CIDER’s ClojureScript Support

Continuing the series on the notable changes in CIDER 2.0, let’s talk about ClojureScript - forever the trickier sibling in the CIDER family.

I’ll start with a confession I’ve made before: I rarely use ClojureScript myself, which is a big part of why its support in CIDER has historically lagged behind Clojure’s. Every “State of CIDER” survey reminds me of this, usually in the comments section, occasionally in all caps. So in the 2.0 cycle I decided to stop feeling vaguely guilty about it and actually do something - across every layer of the stack: CIDER itself, cider-nrepl, and Piggieback.

First, a strategic decision

The most important ClojureScript change in CIDER 2.0 isn’t a feature - it’s a decision about what not to build. Some of CIDER’s most powerful tools (the debugger, enlighten, tracing, profiling) are deeply tied to JVM runtime introspection, and porting them to ClojureScript would be a massive effort with a poor cost/benefit ratio. Rather than keeping them in eternal “maybe someday” limbo, we’ve explicitly scoped them as Clojure-only and focused the actual work on the things cljs users hit every day: evaluation, testing, error reporting, and clear behavior everywhere else.

That last part matters more than it sounds. Historically, invoking a JVM-only command in a ClojureScript REPL would fail in some confusing way - a cryptic error, a JVM-flavored result, or silence. Now the ops themselves report a clojure-only status, and CIDER tells you plainly that the command isn’t supported for ClojureScript. Knowing what a tool won’t do is half of trusting it.

What actually got better

• The regular test commands (cider-test-run-ns-tests and friends) now work in ClojureScript REPLs, asynchronous cljs.test/async tests included. Previously CIDER just refused, and you were stuck evaluating (run-tests) by hand like an animal.
• Expanding your own macros (e.g. ones brought in via :refer-macros) used to silently echo the form back unexpanded - a bug filed all the way back in 2017. The compiler environment is now threaded to the analyzer properly, and it just works.
• cider-nrepl now resolves the ClojureScript compiler environment through a provider chain, with a dedicated shadow-cljs provider - so the static-analysis ops keep working in a shadow REPL that never loads Piggieback.
• The new cider-tap viewer works with ClojureScript too: a runtime helper buffers tapped values and the JVM side streams them to Emacs. (Tapped cljs values aren’t inspectable - they live in the JS runtime - but you see them as they happen.)
• ClojureScript stack frames now render their ns/fn properly instead of degrading to nil/nil, and unqualified core vars resolve against cljs.core rather than falling back to clojure.core (which quietly broke things like indentation metadata).
• A recent ClojureScript on the classpath (whose Closure compiler wants JDK 21+) no longer crashes cider-nrepl at startup on an older JDK - you get a Clojure-only session instead of no session.
• Piggieback itself got a round of bug fixes in the 0.6.x/0.7.0 releases - it’s easy to forget it exists (which is rather the point of it), but it powers most cljs REPLs CIDER talks to.

The documentation kept pace too: the new full-stack Clojure + ClojureScript guide covers the two-REPLs-one-project setup that trips up nearly everyone, and the ClojureScript docs got a general refresh.

An unexpected assist

Fun aside: this is the area where AI coding agents helped me the most during the 2.0 cycle. My ClojureScript experience is limited, but between the excellent bug reports from the community and the ability to quickly prototype and test fixes against real shadow-cljs and figwheel setups, problems that had been “someone who knows cljs should look at this someday” for years finally got fixed. Make of that what you will.

What’s next

I keep pondering some form of “native” shadow-cljs support, given that shadow-cljs is what most ClojureScript users actually run these days. That’s still very much in the hammock phase, so don’t hold me to it - but the direction is clear: fewer moving parts, clearer errors, and honesty about what’s supported.

If you’re a ClojureScript user, I’d genuinely love to hear how 2.0 feels in your daily work - the feedback loop is what keeps this improving. Keep hacking!

Articles in the Series

• CIDER 2.0: Sky is the Limit
• Simplifying Session Management in CIDER
• Stepping Through Macros in CIDER
• Making CIDER More Discoverable
• Modernizing CIDER's Completion
• Closing the Find-Usages Gap in CIDER
• Sharpening CIDER's Debugging Tools
• Leveling Up CIDER's ClojureScript Support

Permalink

Introduction To honeysql - Clojure SQL Library

Notes

• Code
• honeysql
• Postgres and Clojure

Permalink

Eyre: Gathering System Facts

Back in medieval England, an eyre was a travelling court. Royal justices would ride out to a county, set up, and go through everything; crimes, taxes, who owned what, who owed what. Before they could rule on anything they had to know the full state of the place. So the first job was always the same. Count it all up.

Most config tools start the same way. Before they touch anything, they probe the system to check what&aposs already there. This is fact gathering. The tool looks at the machine, builds a picture of its current state, then decides what to do next.

Puppet has Facter, Chef has Ohai, Ansible has its setup module. They all have the same job. To profile the machine (OS, memory, network, filesystem) and hand back the results as data you can use. You can&apost manage a system well if you don&apost know what it looks like right now.

As part of cleaning up and modernising Spire, I&aposm putting out a new small library: Eyre. It gathers system facts through a shell. Spire will end up using it for it&aposs facts.

You give Eyre a function that runs a shell script and hands back the result as {:exit exit-code :out stdout :err stderr}. That&aposs it. Because you supply the executor, Eyre itself has zero dependencies. Whatever it needs is injected.

Put the following in a file gather.clj:

(ns gather
  (:require [babashka.process :as process]
            [clojure.pprint :as pprint]
            [eyre.core :as eyre]))

(defn make-exec [shell]
  (fn [script]
    (process/shell {:in script
                    :out :string
                    :err :string}
                   shell)))

(pprint/pprint
  (eyre/gather (make-exec "bash")))

then run it with babashka:

$ bb -Sdeps &apos{:deps {io.epiccastle/eyre {:mvn/version "0.1.1"}}}&apos gather.clj
{:shell
 {:type :bash,
  :version "5.3.15(1)-release",
  :shell "/bin/bash",
  :canonical-path "/usr/bin/bash"},
 :os
 {:family :linux,
  :kernel
  ...

You will see it dump all the facts it could find running as your user on a local bash shell.

What keys do we have?

(keys (eyre/gather (make-exec "bash"))
;;=> (:shell :os :hardware :users :filesystem :network :paths)

Lets just pull out the :shell portion of the response:

(:shell (eyre/gather (make-exec "bash"))
;;=>
{:type :bash,
 :version "5.3.15(1)-release",
 :shell "/usr/bin/bash",
 :login-shell "/bin/bash",
 :canonical-path "/usr/bin/bash"}

I can try launching it through other shells by changing "bash" to "zsh", "fish" or another shell and it continues to work.

(:shell (eyre/gather (make-exec "zsh"))
;;=>
{:type :zsh,
 :version "5.9.2",
 :shell "/usr/bin/zsh",
 :login-shell "/bin/bash",
 :canonical-path "/usr/bin/bash"}

(:shell (eyre/gather (make-exec "fish"))
;;=>
{:type :fish,
 :version "4.8.1",
 :shell "/usr/bin/fish",
 :login-shell "/bin/bash",
 :canonical-path "/usr/bin/bash"}

Here you can see the :login-shell continues to show the parent shell, while :shell shows the path of the shell process that you are running inside.

All decision on what to run in the executor is based on the :type of the shell. Eyre supports bash, zsh, sh, dash, ksh, busybox, fish, nushell, PowerShell and even cmd.exe. It can probe Linux, FreeBSD, NetBSD, macOS and Windows hosts.

Over SSH

Since Eyre just needs a function that runs a command and returns {:exit :err :out}, you&aposre not stuck running it locally. Plug in an executor that runs over SSH, and now you&aposre gathering facts from a remote machine instead.

Here&aposs what that looks like using clojuressh:

(ns gatherssh
  (:require [clojure.pprint :as pprint]
            [clojuressh.core :as ssh]
            [clojuressh.session :as session]
            [eyre.core :as eyre]))

(let [session (ssh/ssh "remotehost.com" {:username "remote-username"})
      exec (fn [script]
             @(ssh/exec session script {:out :string :err :string}))
      facts (eyre/gather exec)]
  (session/disconnect session)
  (pprint/pprint (:shell facts)))
;; =>
{:type :bash,
 :version "4.3.48(1)-release",
 :shell "/bin/bash",
 :login-shell "/bin/bash",
 :canonical-path "/bin/bash"}

Run:

bb -Sdeps &apos{:deps {io.epiccastle/eyre {:mvn/version "0.1.1"} io.epiccastle/clojuressh {:mvn/version "1.0.0"}}}&apos gatherssh.clj

AI in Development

LLM assisted coding provided two great benefits during development. The first was script translation. They are very competent at translating software from one language to another and certainly I do not know the idiosyncrasies of every shell.

The second was help setting up a significant test platform. Helping to write Packer scripts to build VMs, or Docker scripts to build containers, there was a lot of work here. Without AI doing a lot of that drudgery the library would not be tested across so many operating systems and shells.

Improvements

Running over the network brings a problem you don&apost get locally: latency. Every network shell call has a delay, and if you split fact gathering into lots of small calls, those delays stack up.

Right now, some of the probe scripts are joined together and run as one, so a slow connection doesn&apost pay round trip cost over and over. But there&aposs more to do. More scripts could be merged the same way. And beyond that, the gathering itself could be smarter. It could pull only the data you actually need instead of everything. These improvements will be left for later versions.

You can find the code here and the output documentation here.

I hope you find some uses for this tool.

Permalink