Learn linear least squares with me

Permalink

Program images and portable Scheme backends for Jolt

Two features landed in Jolt recently, both of which fell out from loosening the coupling between Jolt and the host runtime. The first is the ability to serialize program images in the style of Common Lisp and Smalltalk, and the second is to have a portable Scheme layer decoupled from the Chez runtime. Let's take a look at what these things buy Jolt in practical terms.

Images: a black box for your program

If you've ever had to support a production system like a web application, then you know the value of having good logging. What typically happens is that you sprinkle statements through the code, ship them somewhere searchable, and then use them as bread crumbs when something breaks. When a system has errors, you have to first reconstruct what it was doing to understand what happened. If your logging didn't capture a critical piece of information, then the investigation devolves into guess work and attempts to reconstruct the state which caused the error. This can be a particularly frustrating experience when production goes down at 3am.

The trouble with logging is that it forces you to guess the question before you know it. Each log line is a projection of program state chosen in advance, often being composed of two or three things that seemed relevant when you wrote the call. If the actual cause happened to be in the fourth thing, the log tells you nothing of value. You can't go back and ask a different question, because the values are gone, and you likely have no way to access them even if they're still in memory. All you have to work with is the rendering you decided on when you wrote the code originally.

As a result, people tend to over-log defensively, which creates a different type of problem where you end up with a volume of logs that add noise and make tracing harder while often still missing the fields that actually mattered.

But what if I told you that there was a better way, and you didn't have to rely on logging at all? This is precisely what jolt.image lets us do. Instead of choosing what to record ahead of time, you can just dump the whole state of the program at the time of the error to disk. Then you can just copy the file to your local machine, load up the state in the REPL and then poke around in it to see what happened.

Here's what that looks like in practice. To dump the state when an error happens, all you have to do is call image/dump! in the exception handler:

(try
  (process-batch! batch)
  (catch Exception e
    (image/dump! (str "crash-" (random-uuid) ".jimg")
                 {:error   (Throwable->map e)
                  :batch   batch
                  :pending @work-queue})
    (throw e)))

Or skip the enumeration entirely and take the whole program:

(catch Exception e
  (image/dump-world! "crash.jimg")
  (throw e))

dump-world! walks the var table and writes every data var's root so that nothing in your code has to declare what its state consists of up front. The image is architecture agnostic, so an image written on an arm64 server will restore fine on your x86-64 desktop. Once you've copied the file to your local machine, you just have to open it in a REPL:

$ jolt repl
user=> (require '[jolt.image :as image])
user=> (image/restore-world! "crash.jimg")
412
user=> (filter #(nil? (:price %)) @app.core/current-batch)
({:id 4182, :sku "B-77", ...})

What comes back are the values that were present in memory at the time the problem occurred. Maps, records, cycles and shared structure are all intact with their respective metadata attached, and functions come back callable as well. A named function resolves to the live one, while an anonymous closure travels as its source form along with its captured values to get compiled back on restore. So you can actually call the function that failed, on the data that failed in the REPL and see exactly what went wrong. You can now ask a strictly larger set of questions than any log file can answer, and you didn't have to know any of them in advance.

You can think of the program image as a black box recorder. Just like an aircraft stores the instrument states to allow investigators to decide afterwards what to look at, a program image gives you all the information needed to debug the problem.

At this point, a keen reader might ask how this approach handles open resources such as a socket or a file port that can't be serialized. The approach I landed on was to have dump-world! write them as stub records by default. Once the image is restored, you can list them by calling (image/stubs), and either register a resolver that reopens them or swap live values in by hand from the REPL using (image/register-stub-resolver! kind-or-pred f).

One limitation is that a closure over a compile-time constant refuses to dump because the constant gets folded into compiled code and can't be recovered while the stored source still needs it. Closures built by partial and comp have the same problem for a related reason. The image's header is also checked on read so that you don't get stale data from an incompatible build. While dump-world! will dump everything it can, dump! is strict by default, and names the path to the offending object instead of writing something subtly incomplete. It's useful in cases where you want to be explicit about having the whole state available.

Sending a program to someone

While post-mortem debugging is an obvious use case, another interesting application is to facilitate collaboration. An image is a program state you can hand to another person which opens up some intriguing possibilities.

For teaching, that means you can put someone directly into the middle of a running system, with real data loaded, without them having to build or seed anything. Imagine being able to say "here's the image with the pipeline implemented halfway through; go look at stage 2 in it." For a bug report, it means a colleague can reproduce your problem exactly, because they are just loading your state. They can poke at it, change something, dump it again, and send it back to you.

It's a whole different relationship with a program than what we're used to since, traditionally, source code is treated as the artifact that we pass around. Jolt moves the needle closer to how Smalltalk and the Lisp machines worked, and how save-lisp-and-die still works in Common Lisp today. The program is a live thing you keep that can evolve over time as opposed to being a recipe you can run.

Portable Scheme backends

Originally, my goal for Jolt was to build a Clojure implementation on Chez Scheme. However, Jack Rusher pointed out that it would be possible to factor out a portable runtime and compiler making Chez just one target among several. Gambit was the obvious choice for a second host since it has a JavaScript backend making it possible to run Jolt in a browser. If you visit the official site you can now try playing in an interactive REPL, which is Jolt running in the browser.

The refactor split the Scheme part of the compiler into three distinct layers. The core is written in portable Scheme implementing collections, sequences, the reader, the printer, vars, multimethods. This is ordinary Scheme that any serious implementation runs unchanged.

Next there is the adapter contract where the host shows through. Every host capability goes through an sa-* entry point, and the contract file lists 72 names grouped into tiers: system (clocks, environment, exit), threads, eval, introspect (continuation frames for backtraces), ffi, native-compile, and image. A target can either implement a tier or degrade it honestly so that an absent capability raises a message-carrying error or returns empty. That last property is key for making partial ports actually usable. The Gambit version runs with ffi, native-compile and image degraded, and declares its capabilities explicitly.

Finally, target-owned files are the two pieces nobody can share. These include the adapter itself, and the hash kernel. For example, the Chez version uses unsafe fixnum operations that other Schemes spell differently. On the compiler side, per-target differences go through a primitive table with the main entry being the unsafe-op prefix, so a target that maps it to the empty string simply gets checked operations everywhere stating whether it's safe, portable, or slower.

The dialect-specific work is both smaller than you'd guess and duller than you'd hope. Most of it involves mapping records to their parent types, the hashtable API, fx operation spellings, the shape of error objects, and making the hash function produce bit-identical output to Chez. The Gambit port weighs in at about 6,000 lines, and a good fraction of it, including the seed itself, is generated on Chez rather than having to be written from scratch. Cross-minting the seed from a known working build is the trick that keeps a new target from having to bootstrap itself.

Why more than one Scheme is worth having

The immediate payoff is reach. Gambit compiles to a single JavaScript file, so Jolt now runs in a browser allowing for a REPL on the front page of the site using the real compiler and standard library evaluating directly on the client. It's not terribly fast, but works for a demo.

Scheme is a whole family of languages with different dialects each optimizing for different use cases. So, the deeper payoff here is in opening up an ecosystem of implementations that made different bets. All of them share core language semantics, but each dialect puts its own twist on the language providing a runtime optimized for different use cases. Jolt can now piggyback on this whole ecosystem providing a Clojure layer on top.

Chez makes an excellent default since it's fast, relatively small, feature-rich, with real threads, an FFI, and native compilation. Gambit gets you to JavaScript and C. Meanwhile, a whole-program optimizing compiler in the Stalin lineage is a different proposition entirely; it affords aggressive closure and type analysis producing tiny output that suits a small binary shipped to a constrained device where startup and footprint dominate. Such a compiler typically has no runtime eval at all, which sounds disqualifying until you notice that the seed is already cross-minted on Chez. This way the compiler can live on one Scheme while the emitted program runs on another.

The key part here is that the program is the same Clojure regardless of which host you target. What changes are the capabilities available, which have to be stated in the contract providing clear boundaries for what can be expressed by each runtime. Thanks to many existing Scheme implementations, the same code can now run on a server, in a browser tab, as a tiny static binary, or get embedded in existing programs.

A program shouldn't be trapped in the process that started it, nor should it be married to the runtime it was first compiled for. Lisps were always meant to be flexible, and Jolt embraces this philosophy.

Permalink

Call for Proposals. July 2026 Survey

Greetings folks!

Clojurists Together is pleased to announce that we are opening our Q3 2026 funding round for Clojure Open Source Projects. Applications will be accepted through the 24th of August 2026 (midnight Pacific Time). We are looking forward to reviewing your proposals! More information and the application can be found here.

We will be awarding up to $29,000 USD for a total of 4-5 projects. The $2k funding tier is for experimental projects or smaller proposals, whereas the $9k tier is for those that are more established. Projects generally run 3 months, however, the $9K projects can run between 3 and 12 months as needed. We expect projects to start around mid-September 2026.

A BIG THANKS to all our members for your continued support. We also want to encourage you to reach out to your colleagues and companies to join Clojurists Together so that we can fund EVEN MORE great projects throughout the year.

And Now the Survey…

We surveyed members in July to find out what what issues were top of mind and the types of initiatives they would like us to focus on for this round of funding. While our goal for the survey is to surface the broadest, most consistently-raised themes, it is not meant to be prescriptive, as we are always interested in nurturing new ideas and approaches. As always, there was a lot of great input and we hope it will be useful in informing your project proposals.

Select Themes:

Demonstrated Impact of Past Funding Roughly three-quarters of respondents draw on CJT-funded work on a near-daily to weekly basis, with the remainder spread across occasional, project-dependent, or passive-interest use. This is strong evidence that past funding has produced tools and libraries with real, sustained utilization — which is why we exist!

Adoption and Growth of Clojure Continue to be of Concern. This theme is closely linked to employment challenges cited. These themes require broader or more strategic solutions that may be best addressed by the Core Team. However, Clojurists Together can support smaller and more focused efforts. Some ideas include:

• Facilitating discovery of existing tools and documentation currently residing in disparate spaces so that both new and seasoned users can quickly reference new and updated content.
• Support for developers willing to present the Clojure story to their professional or tech-adjacent communities.
• In addition to the mentoring already underway, development of self-directed training that is easy and cost-effective to scale.
  
  
  

About 88% of Members Surveyed are Using AI tools in some capacity - with members calling out the need for Clojure-specific support.

Developer Experience Tools are Respondents' Top Priority for Clojure and ClojureScript with Error Messaging identified in the top 4 for both. There is plenty of work that needs to get done in these categories. The good news is that the Clojure core team along with the CLI Task Force is actively working on improving the user experience of the command-line tooling. More to come in the few months….

July 2026 Survey Summary:

This summary includes a selection of member comments.

1. Who We Heard From: Respondent Profile & Platform Usage

Before weighing the themes below, it’s worth noting who answered this survey. The respondent base skews heavily toward long-tenured Clojure developers, and server-side JVM use dominates how members actually deploy Clojure.

Tenure

Of 41 respondents, the overwhelming majority have used Clojure for a long time:

*87.8% reported 6 years or more of Clojure experience, with only a small remainder spread across 1–5 years and under 1 year combined. This is an important caveat for every other theme in this report: the feedback is disproportionately the voice of veteran users, not newcomers.

Mentoring vs. beginner status

Respondents identified overwhelmingly as mentors rather than newcomers. Combined with the tenure data above, this confirms the survey sample is dominated by experienced members who are already invested in growing the community.

Platforms targeted (41 respondents, select-many)

*Clojure on the JVM server remains the dominant deployment target by far (97.6%), with ClojureScript in the browser a strong secondary use case (68.3%). Notably, ClojureDart shows minimal current usage (4.9%) despite being repeatedly and enthusiastically flagged in the open-ended “magic wand” and ecosystem-support answers (Section 8) — a gap between current adoption and member enthusiasm worth factoring into funding decisions.

2. Quantitative Priorities: Areas for Improvement

Members were asked which areas of Clojure and ClojureScript most need improvement (select-many). Developer Experience Tools ranked #1 in both languages, and data/error-handling concerns dominate the Clojure-specific results.

Clojure (35 respondents)

ClojureScript (26 respondents)

Write-in responses (ClojureScript, 1 mention / 3.8% each): reduced or near-zero NPM dependency, ability to do full-stack development without a separate backend, less reliance on NPM generally, AI-supported development, and “N/A, I don’t use ClojureScript."

3. Theme: Developer Experience Is the Top Tooling Priority

“Developer Experience Tools” was the single highest-ranked improvement area for both Clojure (45.7%) and ClojureScript (42.3%), and it recurs throughout the open-ended answers as well. For Clojure specifically, error messages and data analysis/processing frameworks tied for second place (40% each) — well ahead of documentation, IDE support, and debuggers. For ClojureScript, build tooling (26.9%) and documentation (23.1%) stand out as the next-biggest gaps after developer experience, suggesting the ClojureScript toolchain still feels heavier to maintain than the Clojure one.

4. Theme: 88% of member respondents use AI tools — but Clojure-Specific Support Is Lagging

Claude Code was named most often in the open-ended answers, alongside Cursor, Copilot, Gemini, Aider, ECA, bhauman’s MCP Server, Amazon Kiro (via CP in IntelliJ), Several members pointed specifically to REPL-driven, Clojure-aware tooling (e.g., clojure-mcp / clj-nrepl-eval integrations) as the feature that makes AI genuinely useful for Clojure — but also noted that generic AI tools frequently mishandle Clojure’s syntax (parentheses/brackets) and that few tools understand Clojure idioms well.

What members use AI for (38 respondents, select-many)

• One respondent noted that we should include ‘Security Reviews’ and ‘Agentic Coding’ as an option in future surveys.

Sub-themes:

• Concern that in an increasingly AI-generated-code world, Clojure loses its traditional advantages (conciseness, expressiveness) if hiring/tooling decisions default to more “mainstream” languages the models are best trained on.
• Mixed sentiment: alongside enthusiastic adopters, a vocal minority is skeptical of or opposed to AI-driven development altogether.

Supporting comments:

“clj-nrepl-eval from bhauman/clojure-mcp-light is central. REPL is the killer feature for AI assisted Clojure dev compared to other languages.”

“I wish ECA would work well with local AI models using Ollama. I dont want to use big tech companies… I dont trust them.”

“Each client provides a chatbot, which might be inside the IDE but I have no idea how to make it work with Clojure and not mess up the brackets.”

“The sad truth is in an ever increasing LLM driven development world there is less incentive to use Clojure than something like Rust. All the downsides in making that switch are alleviated if LLMs are doing the coding for you.”

“I like to use free models and run them locally, if there is a large amount of repeatable and we’ll defined work to do it can be good, like a refactor. Sometimes it’s good to use to test an idea or prototype I would not have time to do otherwise. I generally take it that if an llm agent can do something then it’s likely not that hard to do. If the llm struggles on something that should be simple it’s interesting to find out why.”

“I rarely write code “by hand” anymore. My workflow is primarily prompting various coding agents (Claude code, codex, open code using models via open router) and reviewing their output, but rarely dropping into the editor myself.”

“LLMs are a scourge upon the human race with no actual profitability, and I hope every day to see this bubble finally pop.” “It has basically taking over everything. Agent harnesses.”

“While our company doesn’t forbid the usage of (generative) AI tooling, it doesn’t encourage it either. It is up to each individual developer to use it or not. But the agreed contract is that whatever code a developer produces using AI tooling must meet the same established conventions (e.g., code style, idioms to be used, code and architecture estructure, etc) and quality levels of code produced by human developers. And that the code pushed by that developer must be owned by him/her, and that it is his/her own responsability to maintain, and fix if needed.”

5. Theme: Perception, Advocacy & the “Is Clojure Dying?” Narrative

The single most repeated theme in response to “the biggest challenge facing Clojure developers” was one of perception rather than technology: Clojure is widely seen — inside and outside the community — as niche, shrinking, or even dead, which makes it harder to justify on new projects, hire for, or pitch to business leadership and investors.

Supporting comments:

“It is a challenge using it on new projects and justifying it over mainstream alternatives. The biggest complaint I always hear is ‘how will we find developers’. " I think this is more of a perception challenge, the easiest way to reply would be to just point to success stories, or a very visible app or product.”

“Outreach. Many people think language is dead”

“In the world of startups Clojure is generally seen as a niche language and therefore a hindrance to selling a company and maybe even also just to getting funding (sometimes). A friend of mine is the CTO of a startup that was in talks for an acquisition, and the company backed out of the deal because Clojure was used.”

Members suggested amplifying success stories and visible production use cases, supporting community “influencers” and advocates, and funding outreach/evangelism efforts aimed at both developers and business decision-makers.

6. Theme: Jobs & Commercial Adoption

Closely tied to the perception theme is a concrete, recurring concern about the Clojure job market: too few open positions, hiring managers who default to languages with larger corporate backing, and no “gateway” framework (comparable to Rails or ML Frameworks) that pulls new developers into the language the way it once did.

Supporting comments:

“Lack of job opportunities. Big companies are quite skeptical about non-mainstream languages.”

“The people that make hiring decisions view developers as fungible goods, which then leads them to choose languages based on which one they believe will have the lowest salary/hourly cost which tend to be the languages with large corporate backers, and Clojure is not one of those languages.”

7. Theme: Library, Framework & Supply-Chain Health

Members flagged specific maintenance gaps in the ecosystem: unmaintained libraries with no clear owner, documentation gaps in widely-used projects. Support for projects, tools and platforms cited: (5) CIDER; (4) Malli; (3) ClojureDart, re-frame, Pathom, Babashka, Jank; (2) reagent, Reitit, Shadow-CLJS, Scicloj; (1) HugSQL, clj-kondo, duct, nrepl, datalevin, datahike, Fulcro, Datascript, Glojure, Grain, eca, Telemere, rama, replicant, http-kit, Clojure Civitas, Clay, Datastar, calva, ring, figwheel-main.

Supporting comments:

“Clojurists Together could act as a broker for finding maintainers for out-of-support libraries.”

“Some great projects could use better documentation; two examples of amazing libraries that could use better documentation being Malli and Specter.”

“Port all major libraries to tools.deps”

• Development of a Clojure language spec given the number of Clojure-like languages now emerging.
  

8. Theme: Community Health, Governance & Connection

The strength, generosity, and openness of the community is seen as a core strength - along with its engineering rigor. Members feedback included a desire for stronger central coordination, more inclusive and welcoming spaces, and a return to in-person connection.

Supporting comments:

“Clojure’s culture of engineering rigour is unmatched in the industry. I think clojurists’ attention to detail and care for their craft is a huge advantage right now in this age of slop and endless downtime. Also the community is warm, welcoming, and friendly, which is not the case anywhere else I “hang out” online”.

“Once people start using Clojure, they usually love it. REPL is great, Clojure is very fast, well designed language, also JVM interop has improved.”

“Resilience of Clojure communities and their support structures seems to be a challenge… In community spaces, some divide and disagreement often appear, and not everybody feels at home and supported.”

“I would create a Clojure foundation that would lead central decision-making for the continued growth and development of the language.” “Community in one digital place, a Clojure language spec.”

“There would be in-person meetups again!”, “That everyone and all events we’re in the same country :) I miss not being able to go everywhere!”

“I don’t know if this counts or not, but my favorite part of the ecosystem is how stable it is. I love that library updates rarely, if ever, break existing code. Having dealt with the churn and instability of the JS and Rails ecosystems, the fact that updates so rarely force me to do tedious work is a godsend.”

9. Theme: Growth, Outreach & Training

When asked directly what areas of the ecosystem need support, “advocacy,” “outreach,” “evangelism,” “mentoring,” and “community and growth” were named repeatedly and independently — more often than any single technical gap — reinforcing that members see growing and renewing the community as at least as urgent as improving the tools themselves.

Permalink

LISP — ภาษา 67 ปีก่อน ที่ยังมีชีวิตอยู่ในทุกภาษาที่คุณเขียน

คุณเขียน if-else ทุกวัน
คุณรัน code ใน terminal แล้ว REPL มันตอบกลับมา
คุณ lambda ใน Python, arrow function ใน JavaScript, closure ใน Rust

— ทั้งหมดนี้ เกิดจากภาษา LISP

และที่น่าทึ่งคือ... LISP ไม่เคยถูก planned ให้เป็นภาษาโปรแกรมด้วยซ้ำ

กระดาษแผ่นเดียวที่เปลี่ยนโลก

1958 — John McCarthy เริ่มพัฒนาแนวคิด LISP ที่ MIT
เมษายน 1960 — McCarthy วัย 32 ตีพิมพ์ paper ใน Communications of the ACM (vol. 3, หน้า 184-195)

"Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I"

ใน paper 12 หน้านี้ McCarthy เสนอไอเดียของภาษาโปรแกรมที่:

• จัดการกับสัญลักษณ์ (symbols) ไม่ใช่แค่ตัวเลข
• ใช้ recursion เป็นตัวควบคุมการทำงาน แทนการใช้ loop
• เก็บ function เป็น first-class citizen — ส่งต่อกันได้เหมือนข้อมูล

McCarthy เขียนมันขึ้นมาเป็น ทฤษฎีทางคณิตศาสตร์ — ไม่ได้ตั้งใจ implement

แต่ก่อน paper จะตีพิมพ์ — ระหว่างปี 1958-59 Steve Russell นักศึกษา grad student อ่าน manuscript

"I told him, 'Steve, why don't you program this eval?' and he said to me, 'Oh, I misread what you meant. I thought you meant I should implement the interpreter.'"

— John McCarthy, ACM interview

นักศึกษา เขียน interpreter ให้ทฤษฎีของอาจารย์ — และภาษา LISP ก็เกิด

code ตัวแรกที่ Russell เขียน ใช้เวลาแค่ 2-3 วัน (ตัว LISP 1.5 Programmer's Manual ฉบับเต็มออกตามมาทีหลัง ในปี 1962)

สิ่งที่ LISP สร้างไว้ (และเราใช้มันทุกวัน)

1. if-then-else

ย้อนไปปี 1958 — ภาษาส่วนใหญ่มีแค่ GOTO กับ branch แบบ assembly

McCarthy ให้กำเนิด cond (conditional expression) — จุดเริ่มต้นของ if-else ที่เราเห็นแทบจะในทุกภาษา

(cond ((< x 0) 'negative)
      ((= x 0) 'zero)
      (t 'positive))

C, Java, Python, JavaScript, Go, Rust — ภาษาเหล่านี้ได้รับมรดกนี้มาหมด Conditional branching ในทุกภาษาสมัยใหม่มีโครงสร้างแบบเดียวกับที่ McCarthy คิดไว้ตั้งแต่ก่อนมนุษย์ไปดวงจันทร์

2. Garbage Collection

ก่อน LISP — programmer จัดการ memory เอง 100% ทุกบรรทัดของ malloc และ free

LISP สร้าง garbage collection ตัวแรกของโลก — ต้นฉบับคือ mark-and-sweep algorithm (implement โดย Daniel Edwards นักศึกษา MIT)

ทุกวันนี้ GC คือ default ในเกือบทุกภาษา high-level — Java, Python, JavaScript, Go, C#, Ruby ล้วนใช้แนวคิดนี้ต่อยอด

3. Lambda — function เป็นข้อมูล

(lambda (x) (* x x))

LISP ทำให้ function เป็น first-class citizen — ส่ง function เป็น parameter ได้, return function ได้, เก็บลง variable ได้เหมือนเป็น string หรือ integer

นี่คือต้นทางของ:

• JavaScript — arrow function (x) => x * x (Brendan Eich ถูกจ้างไป Netscape เพื่อทำ Scheme ใน browser — แต่ management เปลี่ยนใจให้ syntax เหมือน Java)
• Python — lambda x: x * x
• Ruby — blocks, procs, lambdas (Matz บอกว่า "Ruby was a Lisp originally, in theory")
• Java — lambda expressions (Java 8, 2014 — ใช้เวลา 19 ปีกว่าจะตามทัน)
• C++ — lambda (C++11)

4. REPL — ปรัชญา "คุยกับภาษา"

Read-Eval-Print Loop — LISP ให้กำเนิดมันในทศวรรษ 1960s

ก่อนหน้านั้น: เขียน code → compile → run → debug → repeat
หลังจากนั้น: พิมพ์ expression → กด enter → เห็นผลทันที

ทุกวันนี้ถ้าคุณเปิด Python REPL (>>>), Node.js console, Ruby IRB, Chrome DevTools, Rust Playground, หรือ Elixir IEx — คุณกำลังนั่งอยู่ในห้องเรียนเดียวกับ programmer LISP เมื่อ 60 ปีที่แล้ว

5. Homoiconicity — code = data

'(+ 1 2)          ; ← นี่คือ list
(eval '(+ 1 2))   ; ← นี่คือ code ที่รัน list

LISP เขียนด้วย... LISP — code กับ data ใช้โครงสร้างเดียวกัน (S-expression)

แปลว่า โปรแกรมแก้โปรแกรมตัวเองได้ — ไม่ต้องใช้ parser แยก AST, ไม่ต้องเขียน transformer

นี่คือรากฐานของ macro system ที่ทรงพลังที่สุดในสายภาษาโปรแกรม

ไม่มีภาษาไหนทำได้เต็มระบบเท่า LISP — แต่แนวคิด "code as data" ไปอยู่ใน:

• Elixir — macro (รับ AST, คืน AST)
• Julia — macro + multiple dispatch ("We want a language that's homoiconic, with true macros like Lisp" — Julia manifesto, 2012)
• Rust — macro system (proc macro)
• Clojure — macro (ของ Clojure ก็คือ LISP dialect นั่นเอง)

มรดกจาก LISP

LISP (1958)
├── Scheme (1975) — minimalist, lexical scoping
│   ├── JavaScript (1995) — Brendan Eich ตั้งใจทำ Scheme-like ใน browser
│   │   └── arrow functions, closure, first-class functions
│   └── Racket (1995) — ภาษาเพื่อการสอนและการวิจัย
├── Common Lisp (1984) — ภาคอุตสาหกรรม, pragmatic
│   └── Emacs Lisp (1985) — editor scripting (GNU Emacs)
├── Clojure (2007) — LISP บน JVM, immutable by default
│   └── จุดประกาย functional programming ในโลก enterprise
└── Python, Ruby, Elixir, Julia, Rust, Swift — ทุกภาษาเอาแนวคิด LISP ไปปรับใช้

ทำไม LISP ถึงไม่ใช่ภาษาหลักวันนี้

ทั้งที่สร้างนวัตกรรมเกือบทุกอย่างที่เราใช้ — ทำไม LISP ถึงไม่ชนะ?

Paul Graham (ผู้ก่อตั้ง Y Combinator, แฟนพันธุ์แท้ LISP) อธิบายไว้ใน essay "Beating the Averages":

1. Syntax ต่างจากทุกอย่าง — วงเล็บทำให้คนกลัว
2. ไม่มี standard library ในยุคแรก — Common Lisp มากอบกู้ในปี 1984 แต่ช้าไปแล้ว
3. AI Winter — LISP ผูกกับ AI research; เมื่องบ AI หาย LISP ก็หายด้วย
4. Tooling — compiler ช้า, IDE ห่วย (เทียบกับ Visual Studio ยุค 90s)

Graham ยืนยันว่า:

"Lisp is a language that was discovered, not invented."

ฝั่งนักวิจารณ์ LISP (รวมถึงคนที่เคยใช้ใน production แล้วเปลี่ยนไปภาษาอื่น) ชี้ปัญหาเพิ่มเติมที่ Graham ไม่พูดถึง:

1. Dynamic typing ในระบบใหญ่ — refactoring ยาก, compiler จับ error ได้น้อย, debugging ใช้เวลานานใน codebase ระดับแสนบรรทัด
2. Performance จนถึงยุค 2000s — ก่อน SBCL และ Chez Scheme, LISP runtime บน hardware ยุคนั้นช้ากว่า C/C++ มากโดยเฉพาะงาน numerical
3. Fragmentation — Common Lisp กับ Scheme ไม่เคยตกลงกันได้; community แตกเป็นเสี่ยง ๆ แทนที่จะรวมพลัง

สรุป: ไม่มีสาเหตุเดียว — มันคือ perfect storm ของ syntax ต่าง + เกิดผิดเวลา + community แตก + ไม่มี corporate sponsor (ต่างจาก Java ที่ Sun ทุ่ม, C# ที่ Microsoft ทุ่ม)

มรดกที่มองไม่เห็น

บทความนี้ไม่ได้ตั้งใจจะบอกว่า "คุณควรเขียน LISP"

แต่ทุกครั้งที่คุณ:

numbers = [1, 2, 3]
squared = list(map(lambda x: x * x, numbers))

const result = data
  .filter(x => x.active)
  .map(x => x.value);

let squared: Vec<_> = numbers.iter().map(|x| x * x).collect();

— คุณกำลังเขียน LISP โดยไม่รู้ตัว

อ่านเพิ่ม

• McCarthy's Original 1960 Paper — "Recursive Functions of Symbolic Expressions"
• Paul Graham — "Beating the Averages"
• Paul Graham — "Revenge of the Nerds"
• Structure and Interpretation of Computer Programs (SICP) — ตำรา Scheme ในตำนานของ MIT
• Rich Hickey — "Simple Made Easy" — ปรัชญาเบื้องหลัง Clojure

📅 สิงหาคม 2026 | ⚠️ ตรวจสอบข้อมูล ณ วันที่เขียน

Permalink

Clojure Book, Massive Quality Update

I am very happy to announce that my Clojure book has received a massive update. Every line of my book was proofread and corrected by AI, and I read it to make sure that the AI corrections were right.

English is not my native language. Hence, before the AI era, the book I released had many grammatical errors. Now that AI has corrected them, my book is really good to read. In many places, AI has made my book terse and to the point. I’m very happy about it.

This AI proofread also prompted me to proofread the book myself so that no errors would slip by. I have done my best. What was done by AI in less than two hours took me more than three weeks to proofread. There were only a very small number of mistakes that AI made, which I corrected. The book is now far, far better.

I’m sure even Western / English audiences will find my book very enjoyable to read now.

Highlights of this new release are:

• The book is completely proofread twice, once by AI and then by me.
• I have released a video for the section “Creating Your Own Libraries.”
• The book is available in all formats: online, PDF, EPUB, and on Amazon.

I hope you all read my book. Please suggest corrections. And please tell me what material I could add so that my book becomes much better.

I thank all those who have suggested things to improve my book. Clojure has given me a lot, and I’m ready to give back as much as I can. This book is one tiny effort.

Permalink

Find accessible colours for the web

Permalink

Looking for my next role

Hi, I am Niki, and I am looking for my next role.

I am a π-shaped specialist:

• A software developer,
• A product designer specializing in UI and UX.

My mission is simplicity, performance and software that helps people.

Software Developer

I have worked with and built databases:

• Created DataScript, which inspired a whole series of databases (Datalevin, Datahike, re-db) and powered projects such as Roam Research, Athens Research, Logseq;
• Optimized Firebase-backed storage in Roam Research;
• Built a Postgres-backed DBaaS at Instant DB.

Sync engines:

• Postgres to browser at Instant DB;
• DataScript to Datomic at Cognician;
• Helped with SwarmJS/RON CRDTs.

Frontend:

• I’ve been building browser-based apps since 2005;
• Built my first web framework in 2010;
• Created a popular React wrapper for ClojureScript in 2014;
• Built my own virtual DOM implementation in 2023.

Performance:

• Fast EDN is 6x faster than Clojure’s standard EDN parser, on par with JSON on JVM;
• Reduced Roam Research DB load times from minutes to seconds.

UI/UX

I write Grumpy.Website, a blog on UI/UX with 2,000+ subscribers.

My articles have been referenced by Notion, ATP.fm, Daring Fireball and Marcin Wichary.

Earlier this year, I wrote a widely shared critique of excessive menu icons in macOS Tahoe. Apple later removed many of the icons discussed in the article.

Product Design

In 2024, I launched AlleKinos.de, which quickly reached 1,500 daily visitors without marketing or SEO.

I created Fira Code, which became one of the world’s most popular programming fonts.

My Clojure Sublimed extension became the go-to Clojure development environment in Sublime Text.

I also created many other smaller products, libraries, fonts, color schemes, all available on my GitHub.

The Rest

If you want to dive deeper, here’s the usual stuff:

• Projects
• Talks and podcasts
• LinkedIn
• GitHub

I also made a two-page PDF CV:

What am I looking for?

• Full-time or contract,
• Remote or Berlin,
• Individual contributor,
• Small, closely collaborating team,
• Ambitious product.

If you are working on a compiler, a database, an IDE, a programming language or another technically ambitious product, touching graphics, typography, algorithms, low-level programming, and you think my experience can help, let’s talk: niki@tonsky.me.

Permalink

Statistics made simple

I have a weird relationship with statistics: on one hand, I try not to look at it too often. Maybe once or twice a year. It’s because analytics is not actionable: what difference does it make if a thousand people saw my article or ten thousand?

I mean, sure, you might try to guess people’s tastes and only write about what’s popular, but that will destroy your soul pretty quickly.

On the other hand, I feel nervous when something is not accounted for, recorded, or saved for future reference. I might not need it now, but what if ten years later I change my mind?

Seeing your readers also helps to know you are not writing into the void. So I really don’t need much, something very basic: the number of readers per day/per article, maybe, would be enough.

Final piece of the puzzle: I self-host my web projects, and I use an old-fashioned web server instead of delegating that task to Nginx.

Static sites are popular and for a good reason: they are fast, lightweight, and fulfil their function. I, on the other hand, might have an unfinished gestalt or two: I want to feel the full power of the computer when serving my web pages, to be able to do fun stuff that is beyond static pages. I need that freedom that comes with a full programming language at your disposal. I want to program my own web server (in Clojure, sorry everybody else).

Existing options

All this led me on a quest for a statistics solution that would uniquely fit my needs. Google Analytics was out: bloated, not privacy-friendly, terrible UX, Google is evil, etc.

What is going on?

Some other JS solution might’ve been possible, but still questionable: SaaS? Paid? Will they be around in 10 years? Self-host? Are their cookies GDPR-compliant? How to count RSS feeds?

Nginx has access logs, so I tried server-side statistics that feed off those (namely, Goatcounter). Easy to set up, but then I needed to create domains for them, manage accounts, monitor the process, and it wasn’t even performant enough on my server/request volume!

My solution

So I ended up building my own. You are welcome to join, if your constraints are similar to mine. This is how it looks:

It’s pretty basic, but does a few things that were important to me.

Setup

Extremely easy to set up. And I mean it as a feature.

Just add our middleware to your Ring stack and get everything automatically: collecting and reporting.

(def app
  (-> routes
    ...
    (ring.middleware.params/wrap-params)
    (ring.middleware.cookies/wrap-cookies)
    ...
    (clj-simple-stats.core/wrap-stats))) ;; <-- just add this

It’s zero setup in the best sense: nothing to configure, nothing to monitor, minimal dependency. It starts to work immediately and doesn’t ask anything from you, ever.

See, you already have your web server, why not reuse all the setup you did for it anyway?

Request types

We distinguish between request types. In my case, I am only interested in live people, so I count them separately from RSS feed requests, favicon requests, redirects, wrong URLs, and bots. Bots are particularly active these days. Gotta get that AI training data from somewhere.

RSS feeds are live people in a sense, so extra work was done to count them properly. Same reader requesting feed.xml 100 times in a day will only count as one request.

Hosted RSS readers often report user count in User-Agent, like this:

Feedly/1.0 (+http://www.feedly.com/fetcher.html; 457 subscribers; like FeedFetcher-Google)

Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 6 subscribers)

Feedbin feed-id:1373711 - 142 subscribers

My personal respect and thank you to everybody on this list. I see you.

Graphs

Visualization is important, and so is choosing the correct graph type. This is wrong:

Continuous line suggests interpolation. It reads like between 1 visit at 5am and 11 visits at 6am there were points with 2, 3, 5, 9 visits in between. Maybe 5.5 visits even! That is not the case.

This is how a semantically correct version of that graph should look:

Some attention was also paid to having reasonable labels on axes. You won’t see something like 117, 234, 10875. We always choose round numbers appropriate to the scale: 100, 200, 500, 1K etc.

Goes without saying that all graphs have the same vertical scale and syncrhonized horizontal scroll.

Insights

We don’t offer much (as I don’t need much), but you can narrow reports down by page, query, referrer, user agent, and any date slice.

Not implemented (yet)

It would be nice to have some insights into “What was this spike caused by?”

Some basic breakdown by country would be nice. I do have IP addresses (for what they are worth), but I need a way to package GeoIP into some reasonable size (under 1 Mb, preferably; some loss of resolution is okay).

Finally, one thing I am really interested in is “Who wrote about me?” I do have referrers, only question is how to separate signal from noise.

Performance. DuckDB is a sport: it compresses data and runs column queries, so storing extra columns per row doesn’t affect query performance. Still, each dashboard hit is a query across the entire database, which at this moment (~3 years of data) sits around 600 MiB. I definitely need to look into building some pre-calculated aggregates.

One day.

How to get

Head to github.com/tonsky/clj-simple-stats and follow the instructions:

Let me know what you think! Is it usable to you? What could be improved?

Permalink

Painting with Gaussians

Last year I built an edge-aware pixelation tool to turn images into pixel art by deforming a grid so that it follows image edges instead of naively using a fixed grid over the picture. Using an edge adapter grid bent to inform the color and brightness of the pixels worked well for keeping edges crisp and preserving details.

I later realized the same edge information could also be applied in a context of digital painting. A painting program has to figure out where to put brush strokes, how big should they be, and which direction should they flow in. Much of that is already encoded in the edges since they mark the boundaries between regions. These are the contours around areas of objects that a brush would trace, and their absence indicates generally flat areas where a few broad strokes should suffice.

And so, I set off to see if I could make a program that paints in the style of a digital painting where marks derived from the image structure would resemble brush strokes. My goal was to make an interactive tool where you drag sliders around and the painting reforms itself in front of you. The project was also a great opportunity for me to test drive Jolt and see how well it works for building a non-trivial project.

In this post, I'll walk you through how it all came together. We'll see what ideas worked and which ones didn't. Most importantly, we'll find out whether the end result actually ends up resembling anything like a painting.

Why Gaussians?

The first question we need to consider is what a brush stroke is exactly in computational terms. A stroke of oil or acrylic is an elongated mark which has a center of color that fades toward its edges, and its orientation is the product of a brush being dragged across a canvas. It's translucent at the edges, and strokes overlap, allowing a painter to lay down broad blocks of color first, then build detail on top with smaller and more translucent marks to add finer detail.

It turns out that a 2D Gaussian splat maps onto this idea surprisingly well. It has a mean which is where the stroke lands, a covariance matrix representing how it's stretched and rotated, along with a color and opacity. The covariance can be used to encode the brush direction and its elongation with the major axis pointing along the stroke, and the minor axis across it. Rendering a field of splats with standard over-compositing where each one is occluding what's behind it by its alpha gives you a similar effect to a natural painting model that allows marks to layer and blend together. Of course, you don't get the same fidelity of actual paint, so the effect is closer to digital painting using a tool like GIMP or Krita.

There are already some implementations of this idea such as DrawingWithGaussians and 2d-gaussian-splatting-Art. However, both of them use a gradient descent approach where they seed random splats, and then iteratively nudge their positions, shapes, and colors until the rendered field has the appearance of a target image. That's the well known approach which is both slow and opaque. The worst part is that the end result ends up being simply a lossy reconstruction of the input image rather than looking like any sort of a painting.

Since I already had the solution for extracting edge information from the image, I didn't see the point of evolving the image blindly. Instead, the extracted edges can be used to guide the painting process because they tell us where the details are along with the orientation of the strokes. Between detail density and pixel colors I had all the information that I'd need without having to resort to gradient descent. I'd basically just need to trace the existing image. How hard could it be really?

So, I started following the reference rasterizer which uses additive blending where: pixel = background + Σ(intensity × color). Turns out, this approach works in the fitting regime because the optimizer learns colors that compensate for overlap. Unfortunately, seeding thousands of splats directly from pixel colors and rendering them additively creates a lot more overlap. With 1,200 splats on a 64×64 image, the sum hit 22.06 in some pixels, creating pure white blobs all over the image. Luckily, the problem can be solved by using the standard over-operator from alpha compositing to make each splat occlude what's behind it by its alpha so that the summed color never exceeds 1.0. Another benefit of this approach is that it cleanly separates color sampled from the image and opacity.

Edges Tell You Where to Paint

Every image in this section is the same photo run through the same pipeline, with one idea switched off at a time — same source, same stroke budget, same base size — so each step shows exactly what that one idea buys.

I started using the following source photo, and recorded the progress as I continued to improve the app to illustrate what each idea buys. Let's see how the painting evolves as new tricks are added to the mix.

I got a rather sad output which looked like a uniform mosaic with my initial renderer. Every splat had the same size, aspect ratio, and rotation, producing a regular grid of identical blobs. Not really looking like much of a painting so far.

An actual painter would vary their strokes using a few broad strokes for flat regions such as the sky or a smooth surface. Then, along edges and in textured areas like eyes or fabric, a smaller brush gets used to make numerous finer strokes that follow the contours of the objects.

One way to emulate this is by using a structure tensor to compute the image gradient, encoding how much and in which direction the color changes for each pixel. A 2×2 tensor is formed from the gradient outer product, and blurred over a neighborhood. Importantly, the tensor's eigenvectors will tell you three key things. The major eigenvector points across the contour, providing the direction of the strongest gradient. The minor eigenvector points along the edge, giving the direction of the brush stroke. And coherence, which is the ratio of eigenvalues, tells you whether the edge is a crisp contour or isotropic mush.

Each splat gets its own covariance from this tensor at its position, and gets elongated along the edge, with elongation being proportional to coherence. Flat areas stay round while the edges become thin, directional strokes that trace the contours of the objects in the scene. This is the classic painterly rendering trick from Litwinowicz and Hertzmann. With it in place, the rendering started to resemble something that looks like brushwork if you squint a bit. Here, the fur and the hat brim pick up direction, and the whiskers start to appear.

But here, I hit another problem because the structure tensor uses luminance gradients which are grayscale, making it blind to isoluminant color edges such as red lips against pale skin or a blue sign on a grey wall. Luckily, the Di Zenzo color tensor can be used to compute Sobel gradients per RGB channel. Their outer products can then be summed into one tensor, giving a chroma edge that drives orientation as strongly as a luma edge.

Edges Aren't Enough

While the structure tensor solves the problem of figuring out orientation, it tells you nothing about the density of the region. And without knowing that, it's not possible to figure out how many strokes need to go in that region and how small should they be.

You might be thinking that you could just use edge strength to figure this out, and decide on the number of strokes to use based on that. But doing so ends up missing the texture of the objects entirely. For example, a gravel path has little coherent edge structure but lots of high-frequency detail that deserves its own fine marks. A smooth cheek, on the other hand, has a single contour edge while its interior should stay broad. Conversely, a faint-but-real edge, such as subtle fabric folds or distant tree branches, has low absolute gradient magnitude but still needs to be rendered. So, relying solely on doing edge analysis makes it impossible to reproduce many of the important details present in the original image.

This is where the Haar wavelet, which I discussed in this post, comes into play. Running a multi-scale 2D Haar decomposition on the luminance produces a detail energy map by summing the absolute detail coefficients across scales for each cell. The map will contain high values in textured and edgy regions, and low values in ones lacking detail. But raw wavelet energy still has the problem of being absolute. A dark region with genuine texture produces less absolute energy than a bright region with moderate texture, leading to the dark details getting washed out.

What we need here, again, is a luma-relative detail map where each cell's energy is divided by its local mean brightness plus a fraction of the global mean. Now, dark regions can keep their detail, and since the map is fused with locally-normalized edge strength from the structure tensor, a faint contour in a flat region will still attract strokes.

And that's why both of these techniques are valuable here. The tensor carries the orientation and coherence for every stroke, while the wavelet identifies the density map needed to drive adaptive placement. Together they form a complete answer to the question of where detail lives and what shape it has. At this point, things are starting to come together, and there is enough information extracted from the source image to go beyond naive algorithmic stroke placement.

The detail map now decides where the small marks are spent, and the out-of-focus background turns into a smooth wash, while the strokes collect on areas of detail such as the fur and the eyes.

Killing the Grid

However, there's still one remaining problem that we haven't talked about yet. Even with adaptive orientation and density, the output still ends up looking off because the strokes end up having a faint regularity to them. Their placement and regular shape are an artifact of having a regular placement grid.

Using a stratified grid where you divide the image into cells and put one stroke in each of them will necessarily create artifacts at the edges. Incidentally, this is the same problem seen in JPEG compression at higher levels. It's possible to jitter the position within the cell to break the lattice or use rotated grids per detail level, but these workarounds don't address the underlying problem of the grid being regular.

And so, the grid approach needs to be abandoned in favor of placing strokes at positions derived from a Wang avalanche hash of their index. This method gives true white-noise coordinates with no periodicity. There's a subtlety here, however, because a simple linear hash (frac × i A) produces points that fall on Marsaglia hyperplanes leading to diagonal stripes which are even worse than the grid artifacts. You need a proper avalanche mix where each input bit affects every output bit.

Making It Look Painted: Noise as a Feature

At this point I had edge-aligned strokes at adaptive densities which smoothly paint across the whole canvas, but the result still felt too regular. Every stroke on a given edge had the same length, the same spacing, the same alignment, which is not what real brushwork looks like. I needed some way to emulate the hand wavering and changes in pressure to make the painted effect more plausible.

And what better way to add subtle structural variation than to use Perlin noise. I recently wrote a post showing how it can be used to create a flow animation. So, I naturally reached for it since it was fresh in my mind. Two decorrelated 2D Perlin channels can be used to form a smooth vector flow field. Then, each stroke's orientation can be derived by blending the structure tensor's edge angle with the Perlin flow angle, weighted by (1 − coherence). The stroke follows the contour faithfully on a strong edge with coherence approaching 1, while it has a more organic flow field in a flat region with coherence around 0. Using this trick makes the background look a bit like flowing brushwork thanks to the turbulence introduced by the noise factor.

The flow field is computed from a heavily-blurred copy of the structure tensor to diffuse edge orientations into surrounding flat areas. A stroke in a low-detail region follows the nearby feature since the flow of a distant contour ripples through the background. In areas where there aren't any features to guide the strokes, the Perlin vector field takes over to produce organic curves without a directional bias. Per-stroke size and color also need to have independent noise channels to avoid strokes looking identical even in uniform regions. This is the first version where the marks started actually resembling brushwork.

Of course, that all sounds good on paper, but in practice I started seeing a regular wavy pattern in the images. Digging into it, I discovered that the Perlin bend is a shared spatial field, so neighboring strokes end up creating a wave in phase, with every chain passing through a region getting an identical bend sequence. The result ends up looking like a coherent fabric-like weave. And it's particularly noticeable near moderate edges such as fingers or cloth folds. To get a bit of natural variation there would need to be a per-seed phase offset in the noise coords to make each stroke wobble independently.

Layering: How a Painter Builds a Canvas

With all the core pieces in place, the next step was to try doing the actual painting. My idea was to emulate a physical painting process where large splats can be used to define general color regions, and then to layer progressively smaller splats to add progressive detail on top. This way the painting would start with a broad underpainting; mid-tones would be added next, then glazes, and fine detail on top. Each layer is more translucent and specific than the one below it, augmenting the existing structure that's already been built up.

The base layer contains large, opaque strokes that fully cover the image, ensuring that there are no gaps that would create empty spaces. Next is the subject-adaptive broad tier derived from the wavelet detail map, which points to where the interesting content is. In the background where subjectness is low, strokes grow larger and sparser to create a bokeh effect, melting into a few soft daubs. Under the subject where detail is high, the strokes have to stay tight to capture the important features in the image. The mid tier isn't fundamentally different from the base layer, containing shorter and more translucent strokes representing glazes which bring out general shapes in the image. So, the overall mechanic here stays largely the same.

The fine tier is where the real brush strokes live. At this resolution single dabs are not sufficient, and each fine seed needs to trace a chain of tapered Gaussian segments stepped along the edge tangent, so they all fuse into a continuous line. Low-frequency Perlin noise is used to bend the chain slightly. Size and alpha taper toward the tail like a brush lifting. Colors need to be sampled from one side of the edge so the two sides' paints meet at the boundary instead of crossing it. On strong edges, fine strokes carry nearly opaque paint, emulating impasto liner strokes that sit on top of the glaze. The edge map is also used to inform the length of detail strokes, each following an unbroken segment without any breaks or sharp corners in it. Thus, the strokes have a lot of variance in terms of length and shape, the way a real paintbrush would.

Since each refinement level is more translucent, the details accumulate on the underpainting rather than scratching over it. One thing that becomes tricky at finer details, however, is ensuring that strokes hitting a color boundary fade toward the tail as they drift from their original color. When the mismatch becomes large enough, the chain needs to stop before emitting, which simulates the painter lifting the brush at the region boundary.

Here you can finally see the full pipeline with the same strokes as the previous image, but now built up in layers which become progressively more translucent and more color-specific at the top, along with an edge tier that restates the silhouettes from their own sides. You can judge for yourself whether the end result looks painted, but it seems pretty good to my eye.

The same crop before (left) and after (right) shows how the whiskers hold their color instead of smearing into the fur, and the eyes sit crisply against their surroundings.

The GPU Move

I started with the CPU pipeline, which ran at 55–112ms per render, and that was fine for a few tens of thousands of splats. It was also valuable as it led to a couple of performance optimizations seen in pull requests here and here. So, that ended up being a useful exercise for tuning the compiler, but a detailed painting would need hundreds of thousands of small strokes and no amount of tuning would help there. Every splat being a Clojure allocation means that the whole vector would have to cross to the GPU each frame.

At this point, the only viable solution was to move generation entirely to the GPU and do the work within an OpenGL context. Conveniently, the approach I was already following maps well to doing transform feedback with a geometry shader, feeding it a vertex program of candidate points. The shader, in turn, threshold-tests each against the detail map, runs the placement math, and emits packed splat records for the survivors. The buffer stays on the GPU, allowing the render pass to be done directly.

Doing heavy math on the GPU made the whole thing fast enough to tweak different parameters interactively. Controlling the number of splats being used, how small the splats can get, their hardness, and smoothing across different resolutions allows tailoring the end effect for each image being rendered. Most of these controls loosely map to physical painting concepts such as brush size, paint load, glaze transparency, stroke length, and hand steadiness. All that naturally fell out of the approach to model the problem as a painting exercise.

And a few more examples illustrating how the painter works with different kinds of scenes and settings:

Conclusion

In the days of generative image models, it's still fun to see what can be achieved using traditional image transformation techniques. Building a desktop app to prove the concept turned out to be a great exercise for Jolt and my Glimmer reactive GUI library on top of GTK. The exercise also shook out a lack of type hinting optimizations in Jolt, leading to general performance improvements, and helped prove out the nREPL-driven development workflow.

In the end, I'd call the experiment a success. While it could still be tuned further to produce even more realistic painting effects, the general concept works as well as I dared hope. Turns out that analyzing edges and textures of an image to extract its structure gives a lot of the same information a painter uses to decide where to put brush strokes. In that sense, the algorithm really is doing digital painting.

But the really fun part of the project was in combining a number of techniques, such as Perlin noise, wavelets, and edge detection, that I played around with previously in isolation. All these different tricks came together for this project, making it possible to build something greater than the sum of its parts. I find these are the most rewarding types of experiments where you can build on things you've previously learned and combine them in novel ways to make something new and unexpected. I hope you enjoyed the journey as much as I did working on the project.

As always, the project is open source, and can be found on GitHub.

Permalink

I Built a Clojure That Runs on Go(cljgo) . Then I Made It Run a Real AI Agent.

How dogfooding cljgo with a seven-language agent library turned into a Clojure port with zero reader conditionals — and a portability seam called koine that keeps the ugly parts somewhere else.

Building a language host(cljgo) is easy to fool yourself about. It runs your test suite, it runs your toy programs, and you conclude it works. So I went looking for something big enough to actually hurt.

I already had one: toolnexus, a library I maintain that gives any LLM real tool-calling — MCP servers, agent skills, your own functions, sub-agents — already ported across six languages (JavaScript, Python, Go, Java, C#, Elixir), all pinned to one behavioral contract. Porting it to Clojure would give Clojure developers something they didn’t have. Running that same port on cljgo would tell me whether cljgo is real.

The rule that made it interesting: no reader conditionals

The obvious way to write code for two hosts is #?(:clj ... :cljgo ...) sprinkled wherever they differ. I refused to do that, for a selfish reason and a principled one.

The selfish one: every conditional is a place where the two hosts can silently drift apart, and drift is precisely the bug this project exists to prevent.

The principled one: toolnexus should not know cljgo exists. A library about LLM tool-calling has no business containing opinions about which Clojure you run. If it does, then every future host — Babashka, ClojureCLR, whatever comes next — means editing the library.

So the host differences went somewhere else: koine, a small portability seam. File I/O, subprocesses, HTTP, JSON, environment, time, string handling above the BMP — anything the two hosts disagree about lives there and nowhere else. koine is where the conditionals go; toolnexus stays pure Clojure.

The result is one source tree with zero reader conditionals, verified in five execution modes — JVM main, JVM REPL, cljgo AOT, cljgo interpreted, cljgo REPL — with a CI gate that fails if the five disagree by a single assertion. Currently 395 tests, 1,614 assertions, five identical verdicts.

That means the whole dependency story for a Clojure developer is one line:

;; deps.edn — koine comes along with it
{:deps {net.clojars.muthuishere/toolnexus {:mvn/version "0.13.0"}}}

Any Clojure developer can use this and never think about cljgo. And the one cljgo developer — me, so far — gets the same library, unmodified. That’s the arrangement I wanted.

So what is toolnexus?

It is not a framework, and there is no runtime to deploy. It is a library built on one observation:

An MCP server tool, an agent skill, a function you wrote, an HTTP endpoint, a shell command, and a remote agent are the same thing to an LLM — a named, described, schema’d callable.

toolnexus unifies all of those behind one Tool interface, emits the schema in OpenAI / Anthropic / Gemini formats, and ships a client with the tool-calling loop already written: parallel and chained calls, hooks, retries, conversation memory, human-in-the-loop suspension, observability.

Concretely, you get:

• an MCP host — point it at an mcp.json and every server's tools (stdio and streamable-HTTP) join your registry
• agent skills — the open SKILL.md standard, loaded on demand rather than stuffed into the prompt
• your own functions, HTTP endpoints, and ten built-in tools (bash, read, write, edit, grep, glob…)
• sub-agents with scoped tool views and hard budgets, and remote A2A agents

All seven ports run the same examples/ fixtures and must produce the same behavior. The Clojure port is held to the same full conformance tier as the other six — it is not the junior member.

Enough architecture. Here is the thing running.

One agent, three tool sources, your terminal

Let’s build the thing every framework promises and rarely shows: an interactive agent you keep asking questions, backed by three different tool sources at once —

• an HTTP tool — one declaration that fetches the official Clojure events page
• an MCP server — @modelcontextprotocol/server-filesystem, spawned over stdio; all fourteen of its tools join the registry automatically
• an agent skill — a SKILL.md telling the model how to report events, loaded on demand

(Skills are the open SKILL.md standard — markdown instructions the model loads when the task matches. The catalog goes into the system prompt; the full text loads only on request. Progressive disclosure, not prompt stuffing.)

The skill — skills/clojure-events/SKILL.md:

---
name: clojure-events
description: Use when asked about Clojure community events, meetups or conferences. How to fetch and report them.
---
# How to report Clojure events

1. Fetch the official events page with your `clojure-events-page` tool — never
   answer from memory, events go stale.
2. List each event you find as: **name** — date — location (or "online").
3. Only report events that are actually on the page. If the page lists none,
   say so.
4. Close with the source link: https://clojure.org/community/events
5. Keep it a plain list, no marketing tone.

The program — src/chat.cljc, the whole thing:

(ns chat
  (:require [clojure.string :as str]
            [koine.env :as env]
            [koine.json :as json]
            [toolnexus.core :as tn]
            [toolnexus.client :as client]
            [toolnexus.http :as http]))

(def events-page
  (http/http-tool
   {:name        "clojure-events-page"
    :description "Fetches the official Clojure community events page and returns it as text."
    :method      :get
    :url         "https://clojure.org/community/events"
    :result-mode "text"}))

(def mcp-config
  {:mcpServers
   {:files {:type    "local"
            :command ["npx" "-y" "@modelcontextprotocol/server-filesystem" "."]
            :timeout 30000}}})

(defn -main [& _]
  (let [tk (tn/build {:skills   "skills"
                      :builtins false
                      :mcp      mcp-config
                      :tools    [events-page]})
        c  (client/create-client
            {:base-url "https://openrouter.ai/api/v1"
             :style    "openai"
             :model    (or (env/get-env "TN_MODEL") "openai/gpt-4o-mini")
             :api-key  (env/get-env "OPENROUTER_API_KEY")})]
    (loop []
      (print "you> ") (flush)
      (let [line (read-line)]
        (cond
          (or (nil? line) (#{"exit" "quit"} (str/trim line)))
          (do (println "bye.") (tn/shutdown! tk)) 
           (str/blank? line) (recur)
          :else
          (let [r (client/ask c line {:toolkit tk :id "cli" :on-event trace})]
            (println (str "agent> " (:text r)))
            (recur)))))))

Note client/ask with an :id — that is conversation memory. Every question you type continues the same conversation, so follow-ups like "which of those is soonest?" just work.

Run it:

$ cd clojure/examples && task clj-ex1

It greets you with the full inventory — one HTTP tool, fourteen files_* MCP tools, the skill loader, the clojure-events skill — suggests a few questions, and waits at you>.

Recorded with vhs, unedited.

The same file on cljgo — no build step

$ task cljgo-ex1

That runs the same chat.cljc through cljgo run — Clojure hosted on Go, interpreted directly, no JVM anywhere. The startup banner is worth reading once: cljgo prints net.clojars.muthuishere/toolnexus 0.13.0 — 16 namespace(s) with no Java interop and notes it pruned org.clojure/clojure ("cljgo IS the Clojure implementation") — that is the static no-interop check happening for real, on the same artifact from Clojars. If you want the 20 ms cold start, cljgo build will also AOT it to a self-contained native binary — but nothing here requires it.

A real session on cljgo, unedited:

Both projects ship in the repo under clojure/examples (clj-ex1 and cljgo-ex1) with the Taskfile; MODEL= overrides the OpenRouter model per run.

The parts the example didn’t show

The demo used three tool sources. The rest of the library is there too, in all seven ports, because parity is the product:

• Sub-agents — an agent is a tool: a registry of agents with scoped toolkit views, teams, and hierarchical budgets (:max-tokens, :max-turns, :max-wall-ms…) that stop loudly, never silently
• Personas — from-dir composes an identity from files (SOUL.md, MEMORY.md…); a memory tool lets an agent edit its own durable notes; start-agent gives it a heartbeat
• Human in the loop — a tool can suspend with a question; the run comes back pending and resumes when your host answers. MCP servers that elicit input mid-call are bridged onto the same mechanism
• Compaction — a :before-llm hook that keeps a long-lived agent inside its context window
• Remote agents — expose your toolkit as an A2A agent with serve, or consume someone else's; either way it is just more tools
• Built-in tools — bash, read, write, edit, grep, glob and friends, with per-tool toggles

The honest numbers

We benchmarked all seven ports plus a dozen competitor frameworks on one machine in one sitting, and published the table as measured. The Clojure port is currently the slowest port over MCP — ~5.5 ms p50 per request against Go’s 0.49 — and we know exactly why: ~2 ms per stdio round-trip in the JSON-RPC path, already logged as the top optimization target. With native tools it runs the same scenario in 1.7 ms, mid-table, ahead of LangChain.js and Mastra. And the number I actually care about: the two hosts agree to within 0.2 ms from the same source file. The parity claim survives being measured under load.

If a library only tells you its winning numbers, it’s an ad. Full table: performance page.

Three gotchas so you don’t find them the hard way

1. The API-key env chain prefers OPENAI_API_KEY for openai-style clients. If that's set and you're calling OpenRouter, pass :api-key explicitly (as the demo does) — otherwise your OpenAI key goes to OpenRouter and you get a confusing 401.
2. cljgo run evaluates top-level forms and doesn't call -main. Add a two-line entry file that requires the ns and calls it — cljgo-ex1's src/run_chat.cljc is exactly that — or build a binary.
3. Write non-ASCII characters directly in source, not as \uXXXX surrogate escapes — the JVM reader assembles the pair, cljgo's doesn't. (Yes, we found that one the interesting way.)

Try it, break it, tell me

If you are a Clojure developer, you can use toolnexus today on the JVM and never think about any of the above:

{:deps {net.clojars.muthuishere/toolnexus {:mvn/version "0.13.0"}}}

If you are curious whether Clojure-on-Go is real, clone the examples and run task cljgo-ex1. That is the same question I was asking, and running it is a better answer than anything I can write here.

• toolnexus — repo · docs, all seven languages · Clojars
• The examples in this article — clojure/examples
• cljgo — repo · Clojure hosted on Go
• koine — repo · the portability seam

Permalink

Leveling Up CIDER’s ClojureScript Support

Continuing the series on the notable changes in CIDER 2.0, let’s talk about ClojureScript - forever the trickier sibling in the CIDER family.

I’ll start with a confession I’ve made before: I rarely use ClojureScript myself, which is a big part of why its support in CIDER has historically lagged behind Clojure’s. Every “State of CIDER” survey reminds me of this, usually in the comments section, occasionally in all caps. So in the 2.0 cycle I decided to stop feeling vaguely guilty about it and actually do something - across every layer of the stack: CIDER itself, cider-nrepl, and Piggieback.

First, a strategic decision

The most important ClojureScript change in CIDER 2.0 isn’t a feature - it’s a decision about what not to build. Some of CIDER’s most powerful tools (the debugger, enlighten, tracing, profiling) are deeply tied to JVM runtime introspection, and porting them to ClojureScript would be a massive effort with a poor cost/benefit ratio. Rather than keeping them in eternal “maybe someday” limbo, we’ve explicitly scoped them as Clojure-only and focused the actual work on the things cljs users hit every day: evaluation, testing, error reporting, and clear behavior everywhere else.

That last part matters more than it sounds. Historically, invoking a JVM-only command in a ClojureScript REPL would fail in some confusing way - a cryptic error, a JVM-flavored result, or silence. Now the ops themselves report a clojure-only status, and CIDER tells you plainly that the command isn’t supported for ClojureScript. Knowing what a tool won’t do is half of trusting it.

What actually got better

• The regular test commands (cider-test-run-ns-tests and friends) now work in ClojureScript REPLs, asynchronous cljs.test/async tests included. Previously CIDER just refused, and you were stuck evaluating (run-tests) by hand like an animal.
• Expanding your own macros (e.g. ones brought in via :refer-macros) used to silently echo the form back unexpanded - a bug filed all the way back in 2017. The compiler environment is now threaded to the analyzer properly, and it just works.
• cider-nrepl now resolves the ClojureScript compiler environment through a provider chain, with a dedicated shadow-cljs provider - so the static-analysis ops keep working in a shadow REPL that never loads Piggieback.
• The new cider-tap viewer works with ClojureScript too: a runtime helper buffers tapped values and the JVM side streams them to Emacs. (Tapped cljs values aren’t inspectable - they live in the JS runtime - but you see them as they happen.)
• ClojureScript stack frames now render their ns/fn properly instead of degrading to nil/nil, and unqualified core vars resolve against cljs.core rather than falling back to clojure.core (which quietly broke things like indentation metadata).
• A recent ClojureScript on the classpath (whose Closure compiler wants JDK 21+) no longer crashes cider-nrepl at startup on an older JDK - you get a Clojure-only session instead of no session.
• Piggieback itself got a round of bug fixes in the 0.6.x/0.7.0 releases - it’s easy to forget it exists (which is rather the point of it), but it powers most cljs REPLs CIDER talks to.

The documentation kept pace too: the new full-stack Clojure + ClojureScript guide covers the two-REPLs-one-project setup that trips up nearly everyone, and the ClojureScript docs got a general refresh.

An unexpected assist

Fun aside: this is the area where AI coding agents helped me the most during the 2.0 cycle. My ClojureScript experience is limited, but between the excellent bug reports from the community and the ability to quickly prototype and test fixes against real shadow-cljs and figwheel setups, problems that had been “someone who knows cljs should look at this someday” for years finally got fixed. Make of that what you will.

What’s next

I keep pondering some form of “native” shadow-cljs support, given that shadow-cljs is what most ClojureScript users actually run these days. That’s still very much in the hammock phase, so don’t hold me to it - but the direction is clear: fewer moving parts, clearer errors, and honesty about what’s supported.

If you’re a ClojureScript user, I’d genuinely love to hear how 2.0 feels in your daily work - the feedback loop is what keeps this improving. Keep hacking!

Articles in the Series

• CIDER 2.0: Sky is the Limit
• Simplifying Session Management in CIDER
• Stepping Through Macros in CIDER
• Making CIDER More Discoverable
• Modernizing CIDER's Completion
• Closing the Find-Usages Gap in CIDER
• Sharpening CIDER's Debugging Tools
• Leveling Up CIDER's ClojureScript Support

Permalink

Introduction To honeysql - Clojure SQL Library

Notes

• Code
• honeysql
• Postgres and Clojure

Permalink

Eyre: Gathering System Facts

Back in medieval England, an eyre was a travelling court. Royal justices would ride out to a county, set up, and go through everything; crimes, taxes, who owned what, who owed what. Before they could rule on anything they had to know the full state of the place. So the first job was always the same. Count it all up.

Most config tools start the same way. Before they touch anything, they probe the system to check what&aposs already there. This is fact gathering. The tool looks at the machine, builds a picture of its current state, then decides what to do next.

Puppet has Facter, Chef has Ohai, Ansible has its setup module. They all have the same job. To profile the machine (OS, memory, network, filesystem) and hand back the results as data you can use. You can&apost manage a system well if you don&apost know what it looks like right now.

As part of cleaning up and modernising Spire, I&aposm putting out a new small library: Eyre. It gathers system facts through a shell. Spire will end up using it for it&aposs facts.

You give Eyre a function that runs a shell script and hands back the result as {:exit exit-code :out stdout :err stderr}. That&aposs it. Because you supply the executor, Eyre itself has zero dependencies. Whatever it needs is injected.

Put the following in a file gather.clj:

(ns gather
  (:require [babashka.process :as process]
            [clojure.pprint :as pprint]
            [eyre.core :as eyre]))

(defn make-exec [shell]
  (fn [script]
    (process/shell {:in script
                    :out :string
                    :err :string}
                   shell)))

(pprint/pprint
  (eyre/gather (make-exec "bash")))

then run it with babashka:

$ bb -Sdeps &apos{:deps {io.epiccastle/eyre {:mvn/version "0.1.1"}}}&apos gather.clj
{:shell
 {:type :bash,
  :version "5.3.15(1)-release",
  :shell "/bin/bash",
  :canonical-path "/usr/bin/bash"},
 :os
 {:family :linux,
  :kernel
  ...

You will see it dump all the facts it could find running as your user on a local bash shell.

What keys do we have?

(keys (eyre/gather (make-exec "bash"))
;;=> (:shell :os :hardware :users :filesystem :network :paths)

Lets just pull out the :shell portion of the response:

(:shell (eyre/gather (make-exec "bash"))
;;=>
{:type :bash,
 :version "5.3.15(1)-release",
 :shell "/usr/bin/bash",
 :login-shell "/bin/bash",
 :canonical-path "/usr/bin/bash"}

I can try launching it through other shells by changing "bash" to "zsh", "fish" or another shell and it continues to work.

(:shell (eyre/gather (make-exec "zsh"))
;;=>
{:type :zsh,
 :version "5.9.2",
 :shell "/usr/bin/zsh",
 :login-shell "/bin/bash",
 :canonical-path "/usr/bin/bash"}

(:shell (eyre/gather (make-exec "fish"))
;;=>
{:type :fish,
 :version "4.8.1",
 :shell "/usr/bin/fish",
 :login-shell "/bin/bash",
 :canonical-path "/usr/bin/bash"}

Here you can see the :login-shell continues to show the parent shell, while :shell shows the path of the shell process that you are running inside.

All decision on what to run in the executor is based on the :type of the shell. Eyre supports bash, zsh, sh, dash, ksh, busybox, fish, nushell, PowerShell and even cmd.exe. It can probe Linux, FreeBSD, NetBSD, macOS and Windows hosts.

Over SSH

Since Eyre just needs a function that runs a command and returns {:exit :err :out}, you&aposre not stuck running it locally. Plug in an executor that runs over SSH, and now you&aposre gathering facts from a remote machine instead.

Here&aposs what that looks like using clojuressh:

(ns gatherssh
  (:require [clojure.pprint :as pprint]
            [clojuressh.core :as ssh]
            [clojuressh.session :as session]
            [eyre.core :as eyre]))

(let [session (ssh/ssh "remotehost.com" {:username "remote-username"})
      exec (fn [script]
             @(ssh/exec session script {:out :string :err :string}))
      facts (eyre/gather exec)]
  (session/disconnect session)
  (pprint/pprint (:shell facts)))
;; =>
{:type :bash,
 :version "4.3.48(1)-release",
 :shell "/bin/bash",
 :login-shell "/bin/bash",
 :canonical-path "/bin/bash"}

Run:

bb -Sdeps &apos{:deps {io.epiccastle/eyre {:mvn/version "0.1.1"} io.epiccastle/clojuressh {:mvn/version "1.0.0"}}}&apos gatherssh.clj

AI in Development

LLM assisted coding provided two great benefits during development. The first was script translation. They are very competent at translating software from one language to another and certainly I do not know the idiosyncrasies of every shell.

The second was help setting up a significant test platform. Helping to write Packer scripts to build VMs, or Docker scripts to build containers, there was a lot of work here. Without AI doing a lot of that drudgery the library would not be tested across so many operating systems and shells.

Improvements

Running over the network brings a problem you don&apost get locally: latency. Every network shell call has a delay, and if you split fact gathering into lots of small calls, those delays stack up.

Right now, some of the probe scripts are joined together and run as one, so a slow connection doesn&apost pay round trip cost over and over. But there&aposs more to do. More scripts could be merged the same way. And beyond that, the gathering itself could be smarter. It could pull only the data you actually need instead of everything. These improvements will be left for later versions.

You can find the code here and the output documentation here.

I hope you find some uses for this tool.

Permalink

LangGraph isn't cheaper than LangChain — unless you opt out of its defaults

LangGraph isn't cheaper than LangChain — unless you opt out of its defaults

Cost-audit series, episode 4. This series began with an AI agent that burned 136M tokens overnight →.

When LangChain deprecated ConversationBufferMemory (the subject of episode 1 in this series), the official migration path was LangGraph. The pitch: explicit state management, you control exactly what flows where. More expressive, more controllable.

It is — but only if you reach for the controls. The default state model in LangGraph has the same unbounded-growth problem as the memory it replaced. Teams migrating to escape ConversationBufferMemory's cost curve often land on an identical curve, with new graph complexity on top.

This audit shows exactly where the default grows, what it costs, and what opt-outs exist.

The default: MessagesState + add_messages

The quickstart in LangGraph's own docs uses this pattern:

from langgraph.graph import StateGraph, MessagesState

def my_node(state: MessagesState):
    messages = state["messages"]
    response = llm.invoke(messages)   # sends ALL messages to the LLM
    return {"messages": [response]}

graph = StateGraph(MessagesState)
graph.add_node("agent", my_node)

MessagesState is a TypedDict with a single key, messages, backed by the add_messages reducer. Here's what that reducer does:

# langgraph/graph/message.py — add_messages (def at line 18; merge loop below)
def add_messages(left: Messages, right: Messages) -> Messages:
    # ... (coerces left/right to lists of BaseMessage) ...
    left_idx_by_id = {m.id: i for i, m in enumerate(left)}
    merged = left.copy()
    ids_to_remove = set()
    for m in right:
        if (existing_idx := left_idx_by_id.get(m.id)) is not None:
            if isinstance(m, RemoveMessage):
                ids_to_remove.add(m.id)
            else:
                merged[existing_idx] = m      # same id → update in place
        else:
            merged.append(m)                  # new id → APPEND (the list grows)
    merged = [m for m in merged if m.id not in ids_to_remove]
    return merged

Source: langgraph/graph/message.py

This is not a summarizer, not a window, not a trimmer. It is an append-only list. Every message ever added to state stays in state — and every node that reads state["messages"] sees the full list.

This is ConversationBufferMemory with a graph wrapper.

The cost math

Assume a conversational agent: 10 turns, 150 tokens per user message, 200 tokens per assistant reply (modest — a short answer each time).

After 10 turns, state["messages"] contains 20 messages = (10 × 150) + (10 × 200) = 3,500 tokens of accumulated history.

For the 11th call, the node sends all 3,500 tokens of prior history as context, then generates a new reply. Each further turn adds another 350 tokens (150 user + 200 assistant), so the 12th call sends 3,850, the 13th 4,200, and so on.

Total input tokens for a 20-turn conversation:

(Each call = 400 system + 150 current user + (turn−1) × 350 accumulated history.)

A naive estimate (flat 550 tokens/call × 20 calls) = 11,000 tokens.

Actual with add_messages default = ~77,500 tokens. 7× over.

With claude-haiku-4-5 ($0.80/M input, $4/M output) for a chatbot doing 500 conversations/day:

• Naive estimate: 11,000 × 500 × 30 × $0.80/M = $132/month
• Actual: 77,500 × 500 × 30 × $0.80/M = $930/month

That's $798/month of silent overspend on input tokens alone, just from the default accumulation — before you add nodes, tools, or memory.

Multiplier 1: multi-node graphs (each node pays the full state)

LangGraph's value over a simple chat loop is composing multiple nodes — a router, a tool-caller, a summarizer, a responder. Each node that reads state["messages"] pays the full token cost of the accumulated message list.

graph = StateGraph(MessagesState)
graph.add_node("router", route_node)      # reads state["messages"]
graph.add_node("tool_caller", tool_node)  # reads state["messages"]
graph.add_node("responder", respond_node) # reads state["messages"]

For a 3-node graph where each node reads messages, a single user turn that passes through all three nodes costs 3× the message-list tokens. After 10 turns with 3,500 accumulated tokens, one user message costs: 3 × 3,500 = 10,500 tokens just for message history, before any node-specific prompts.

Multiplier 2: interrupt_before / interrupt_after (human-in-the-loop)

LangGraph's human-in-the-loop feature pauses graph execution at a node boundary. When the graph resumes, it deserializes the full checkpointed state and re-injects it into the next node:

# langgraph/pregel/__init__.py — Pregel.astream (v0.2.60), the entrypoint
# that drives interruptible execution. Verbatim signature:
async def astream(
    self,
    input: Union[dict[str, Any], Any],
    config: Optional[RunnableConfig] = None,
    *,
    stream_mode: Optional[Union[StreamMode, list[StreamMode]]] = None,
    output_keys: Optional[Union[str, Sequence[str]]] = None,
    interrupt_before: Optional[Union[All, Sequence[str]]] = None,
    interrupt_after: Optional[Union[All, Sequence[str]]] = None,
    debug: Optional[bool] = None,
    subgraphs: bool = False,
) -> AsyncIterator[Union[dict[str, Any], Any]]:
    ...

Source: langgraph/pregel/__init__.py — search the file for async def astream (defined ~line 1683; interrupt_before/interrupt_after are the pause controls). When a run resumes after an interrupt, Pregel reloads the pending state from the checkpointer (the aget_tuple/aget_state path returns the full checkpoint blob — every message included) before continuing at the next node.

The cost: full state deserialization on every resume. If a workflow interrupts 3 times before completion (a common approval flow), and the state has 5,000 tokens of messages, the resumption overhead alone is 3 × 5,000 = 15,000 extra tokens — paid every time, even if the approval is just a "yes."

Multiplier 3: parallel fan-out (Send API)

LangGraph's Send API dispatches parallel subgraph invocations, each receiving a copy of state:

from langgraph.types import Send

def fanout_node(state: MessagesState):
    return [
        Send("worker_a", {"messages": state["messages"], "task": "summarize"}),
        Send("worker_b", {"messages": state["messages"], "task": "critique"}),
        Send("worker_c", {"messages": state["messages"], "task": "expand"}),
    ]

Source: langgraph/types.py

Each Send carries the full state["messages"] to the worker node. With 3 workers and 5,000 tokens of history: 15,000 tokens dispatched in the fan-out alone. If those workers themselves call an LLM, each call pays the full 5,000-token history again. Compare to the CrewAI quadratic problem from episode 3 — this is the same failure mode, different API.

The opt-outs (LangGraph actually provides them)

Unlike ConversationBufferMemory (which had no good trim story), LangGraph ships built-in tools to fix this. Teams just don't use them by default.

Trim messages before every LLM call

from langchain_core.messages import trim_messages

def my_node(state: MessagesState):
    trimmed = trim_messages(
        state["messages"],
        max_tokens=2000,          # hard cap
        strategy="last",          # keep the most recent
        token_counter=llm,        # use the model's tokenizer
        include_system=True,      # always keep the system message
        allow_partial=False,
    )
    response = llm.invoke(trimmed)   # sends trimmed history, not full list
    return {"messages": [response]}

Source: langchain_core/messages/utils.py

This keeps the full history in state (for checkpointing, human inspection) while capping what the LLM actually sees. Applying a 2,000-token cap on a 20-turn conversation reduces input tokens from ~77,500 to ~40,000 (2,000 tokens × 20 calls). ~48% cost reduction, one line change.

Pass only what the node needs

Instead of giving every node the full state["messages"], scope what each node receives:

class MyState(TypedDict):
    messages: Annotated[list[BaseMessage], add_messages]
    last_tool_result: str        # structured, compact
    task_description: str        # set once, doesn't grow

def tool_caller_node(state: MyState):
    # This node only needs the task + last result — not the full conversation
    prompt = f"Task: {state['task_description']}\nContext: {state['last_tool_result']}"
    response = llm.invoke(prompt)
    return {"last_tool_result": response.content}

The tool_caller_node never touches state["messages"] — it pays zero for message accumulation. Only the nodes that genuinely need conversational context receive it.

Summarize periodically (the right way)

def maybe_summarize(state: MessagesState):
    messages = state["messages"]
    if len(messages) > 20:                # threshold: tune to your cost tolerance
        summary = llm.invoke([
            SystemMessage("Summarize this conversation in 3 sentences."),
            *messages,
        ])
        return {
            "messages": [
                SystemMessage(f"Conversation summary: {summary.content}"),
                messages[-2],    # keep the last human message
                messages[-1],    # keep the last assistant message
            ]
        }
    return {}   # no change needed

This collapses the accumulated history into a single system message at the summarization trigger. After summarization, the effective context is ~200 tokens (summary + last exchange) instead of 3,500+. Insert maybe_summarize as an always-on node before expensive LLM calls.

Measuring your own graph

LangGraph's built-in tracing (via LangSmith) shows per-node token usage, but it's behind a paid tier for production volumes. For a free alternative that works with any JSONL export:

npm install -g @wartzar-bee/tokenscope
npx @wartzar-bee/tokenscope langgraph-session.jsonl

Output shows the session total, how much of each call is re-sent accumulated context versus new work, and where the growth is steepest — the same view that surfaced the 136M-token burn in episode 1.

Summary

LangGraph's MessagesState + add_messages default is ConversationBufferMemory under a new name. The graph model gives you explicit controls that the old memory API lacked — but the controls are opt-in. Without trim_messages, selective state reads, or periodic summarization, migration from LangChain to LangGraph buys you graph expressiveness at the same (or higher, for multi-node graphs) token cost.

The migration from LangChain to LangGraph is worth it — but only once you understand and explicitly opt out of these defaults. Otherwise you're paying for the complexity without the savings.

wartzar-bee builds tools for operating cost-efficient autonomous agents. tokenscope is free and open-source. Follow on dev.to →

Permalink

Sharpening CIDER’s Debugging Tools

The series on the notable changes in CIDER 2.0 rolls on. This time: the “what is my code actually doing?” toolbox - the debugger, tracing, enlighten, and the new tap viewer. This was the part of the release I enjoyed working on the most, and the part that needed the most love.

The debugger, dusted off

CIDER’s interactive debugger is one of its most impressive features and, paradoxically, one of its least reliable ones. Instrumenting arbitrary Clojure code is hard - the debugger rewrites your forms to capture locals at every step, and the corner cases are endless. Over the 2.0 cycle (and the 0.62.x releases of cider-nrepl) a whole family of long-standing instrumentation bugs got fixed:

• Record literals embedded in code survive instrumentation instead of being quietly downgraded to plain maps - which used to break protocol dispatch in anything that compiled routes or components into records (compojure users know the pain).
• defrecord/deftype inline methods no longer blow up with the infamous Unable to resolve symbol: STATE__ error - the instrumenter now sensibly skips the method bodies, which compile to real JVM methods that can’t capture debugger state. #dbg on a bare collection literal triggered the same error; fixed too. And heavily destructured argument lists used to crash instrumentation in their own special way - not anymore.
• A form too large to instrument (yes, that’s a JVM limitation - Method code too large!) now degrades gracefully: CIDER retries without local capture and tells you what happened, instead of surfacing a raw compiler error.
• Quitting a debug session uses nREPL’s interrupt machinery instead of the deprecated Thread.stop, so it keeps working on modern JDKs where Thread.stop is simply gone.

The UX got attention too. Quitting the debugger with q finally restores point to where you started the session - a request filed in 2016 - instead of stranding you at the last breakpoint. The force-step-out key works again. And all the debugger’s single-key commands are now proper named commands with a transient menu (?) listing them, so you’re never stuck trying to remember whether locals was l or L.

Tracing grew a home

clojure.tools.trace-style tracing has been in CIDER forever, but the output was always interleaved into the REPL, where it fought with your actual work. CIDER 2.0 gives traces a dedicated, live-streaming *cider-trace* buffer:

Calls fold and unfold (TAB, or F/U for everything at once), n/p move between calls, and . jumps to a function’s definition. cider-list-traced answers the eternal “wait, what did I even trace?”, and cider-untrace-all cleans the slate.

Enlighten, back from the dead

Enlighten - the mode that displays the values of locals inline as your code runs - has been in “experimental” limbo since 2016. It finally got a proper overhaul: a real test suite, fixes for the same record/deftype instrumentation bugs as the debugger (they share machinery), and - importantly - manners. You can now enlighten a single form with cider-enlighten-defun-at-point instead of flipping a global mode, and cider-enlighten-stop turns everything off at once, rather than making you re-evaluate every function in penance.

Every local and every intermediate result, right there in the buffer:

Tap into your programs

New in 2.0: cider-tap, a buffer that streams every value sent to tap> and lets you crack any of them open in the inspector with RET. tap> has quietly become the Clojure community’s favorite debugging primitive, and now you don’t need an external tool like Portal or Reveal for the basic workflow - though those remain great if you want more. (ClojureScript taps stream too; they’re just not inspectable, since the values live in the JS runtime.)

It’s println debugging, minus the println guilt.

The connective tissue

A few related quality-of-life items round out the picture: stack frames for top-level anonymous functions jump to their actual source instead of clojure.core/fn (a bug from 2020), ClojureScript frames render their ns/fn properly, and the macroexpansion tooling - a debugging tool in its own right - got a full makeover that deserves (and will get) its own article.

None of these tools is new. That’s rather the point: the 2.0 debugging story is mostly the existing tools becoming trustworthy. A debugger you don’t trust is worse than no debugger at all.

The debugging docs cover everything in detail. Keep hacking!

Articles in the Series

• CIDER 2.0: Sky is the Limit
• Simplifying Session Management in CIDER
• Stepping Through Macros in CIDER
• Making CIDER More Discoverable
• Modernizing CIDER's Completion
• Closing the Find-Usages Gap in CIDER
• Sharpening CIDER's Debugging Tools
• Leveling Up CIDER's ClojureScript Support

Permalink