cljs-ajax 0.9.0-beta1 is out

I’ve been trying to make a point of not talking about stuff that I’m doing and concentrating on actually delivering. I find that this avoids a sense of “false progress”, which is why I finally get to talk about something I’ve been working on for the last three months. Or more accurately, Codex on the free plan has been working on it.

The cljs-ajax library was started by the incredibly productive Dmitri Sotnikov, and transferred to me because he had many larger and more important projects. It was the most popular cljs http library by a small margin. The maintainer of the main competitor, cljs-http, (which is an excellent library that explores a different design space), r0man, sadly hit burn out on his project. And approximately six months later, I hit the same thing. Let’s be clear: this is not really a sign that I’m no longer burnt out on this project. Without Codex there’s no way this would have happened.

Burnout

One of the things you learn when maintaining an open source library of even the modest size of cljs-ajax is that engineering concerns predominate. I think everyone starts because they want to write some code, but you look at any project’s mailing list and it’s all about the build server.* cljs-ajax really suffered in this regard: even with all of the unit testing coverage, HTTP fundamentally requires integration testing. Not only that, it needs to work across a wide-variety of platforms, and against a shocking number of badly-behaved endpoints. And I had… no automated integration testing. Instead, I had a test server that I ran repl commands against. Which I needed to do for every last PR.

Equally, the deployment process was a pain in the neck. And relied on a bunch of secrets that I kept losing. I’m sure the Leiningen deploy process was well thought out, it was just more work than I particularly wished to do.

Needless to say, both of these things contributed to my burnout.

So what happened during burnout? Well, nothing very much. A bunch of people submitted PRs and I couldn’t work up the energy to test them. I had one approach to take over the project that I believe was a supply-chain attack. (A pretty low effort one compared to the xz attack.) Nothing really came to take the place. The Closure Library and Compiler was abandoned by Google and adopted by the ClojureScript team. And finally, a lot of the people who had contributed… stopped using Clojure and ClojureScript entirely. There’s been no actively-maintained general http libraries for ClojureScript for five years. On the other hand… fetch happened. Which has rendered a lot of the “cross-API” code rather redundant.

What didn’t happen is slightly remarkable: the code didn’t break. People are still using it and it still works. The innate conservatism of the Clojure project has finally rubbed off on the ClojureScript project and backwards compatiblity is still strong. This isn’t to say that old library references can’t be a problem, but it’s much better than you might expect from many platforms after a five year hiatus.

* Look at a really big project and it’s all about governance.

What now?

So, as the title says, I’ve published a beta version of Clojure 0.9.0. The differences between 0.8.4 and this are… zero. Instead I have:

  • Replaced Leiningen with cljs-build. I have no opinion about which is better, I’m just using the default option, as I did when Leiningen was employed in the first place. This also replaced doo and shadow with karma.
  • Developed a proper set of integration tests replacing my old manual process.
  • Developed a new CI process using github workflows and a release process to match, which also means the keys are now on my github account.
  • Fixed dependabot issues.
  • Lightly updated the documentation (including the first merged PR from rgkirch).

In other words, I’ve prioritised just getting the project into a fit shape and addressing some of the issues that caused so much burnout in the first place.

What’s next?

So, honestly, I have no idea what’s next. I plan to merge down the rest of the existing PRs. Then, I plan to triage the open issue list. There’s an obvious argument to be made for making fetch the default implementation, and maybe changing the API to be more fetch-like. Adding in clojure.spec would obviously be desirable, although I have no idea how workable that would be given some of the heroic things cljs-ajax does to support its very wide set of deployment scenarios. Building a proper objective framework for measuring the deploy-size implications of various options would be desirable since one of the unspoken principles of cljs-ajax has always been to avoid paying for stuff you don’t use. (Within reason, this isn’t C++.) Again, this is mostly engineering stuff. There’s no particular ambition for supporting anything other than the existing RPC model. Equally, I have no particular interest in achieving feature parity with clj-http or hato, although to the best of my knowledge cljs-ajax is still a pretty good, if unpopular, choice if you want an asynchronous HTTP API in Clojure.

But more importantly: I don’t know if there’s actually any interest in this project. There’s other alternatives. The research I’ve done suggests that, even after five years, cljs-ajax is still pretty popular in the community but I’m very outside the community these days, have no active Clojure/ClojureScript projects and I worry cljs-ajax may be a relic of a bygone age.

What if I don’t want to use LLM code?

There are many good arguments for not wanting to rely on LLM code, and especially code that isn’t fully understood by a human. These arguments are valid. I admire and understand those taking a stand against AI slop. For the record, this blog post is entirely human-produced. However, the situation with cljs-ajax is, quite simply, absolutely nothing is likely to happen at all without it. There’s no prospect of a new maintainer stepping in the way that I did. The community is too small and, given my previous experiences, I’d need to put a fair bit of effort into vetting anyone unknown. I still have a day job that occupies most of my time, and I use AI heavily there. The old code is still there. The latest release still has no AI-written code in production pathways, but that’s likely to change.

Permalink

Teaching LLMs to one-shot complex backends at scale, report #1

The LLM is not all that matters in AI coding. What the LLM is targeting matters a great deal. A simpler target that requires less reasoning will produce better results.

Attempts to get LLMs to produce complex backends have been lackluster. A recent paper, Constraint Decay: The Fragility of LLM Agents in Backend Code Generation, shows that even on a simple CRUD app, end-to-end success on the full test suite tops out at 33% once realistic structural constraints are imposed.

Conventional backends are made of many separate systems glued together, each with its own model and failure modes. Most of the failures observed in these benchmarks show up at the seams between these systems. The LLM is not asked to reason about one coherent system – it’s asked to coordinate across many.

Along those lines, we believe Rama is ideally positioned to take LLM coding to the next level for backends. Rama collapses the typical backend stack (databases, queues, stream processors, application logic) into one integrated system. The seams that current LLMs trip over largely don’t exist in a Rama application. A horizontally scalable, fault-tolerant backend is expressed as one coherent program rather than as glue across half a dozen systems.

In the past few months we’ve been working on a project to teach LLMs to one-shot complex backends at scale with Rama as the substrate. Our results so far are very promising, as I’ll review later in this post, but we have a ways to go. The major milestone we’re working towards is one-shotting the entire Matrix spec, which also has a thorough set of tests available that can be used to verify an implementation. What we’re looking to produce is:

  • A generated implementation of Matrix that passes all the reference tests
  • Transcript showing every step of how the LLM one-shotted the project
  • Benchmarks automatically written and executed by the LLM that demonstrate high performance and horizontal scalability

Matrix is orders of magnitude more difficult than the backends current LLMs can handle, particularly with these scalability and fault-tolerance requirements, so one-shotting it will be a huge milestone. However, the overarching goal is for this to work on any backend problem. We don’t expect what we’re building to one-shot every possible backend. Humans remain vastly better than LLMs at broad systems design where many tradeoffs must be considered. What we think is achievable, and what this project is targeting, is a workflow where humans assist with high-level design decisions and the agent handles lower-level decisions and implementation, including achieving horizontal scalability and fault-tolerance. By “fault tolerant” we mean the system continues operating correctly through infrastructure failures (e.g. node deaths) without data loss, data duplication, or downtime, and recovers automatically when failed components return.

Whether our goal is possible remains to be seen, but I’ll be documenting our progress as we go via these progress reports.

Our workflow

We work through the rama-ai-learn project, which we just open-sourced. It’s a benchmark and harness for measuring how well LLMs can produce production Rama code, along with the skill content the agent uses to do the work.

Each task we throw at an agent is a “challenge.” A challenge directory (example) contains a README.md stating operations, latency targets, and other constraints. It also has an interface the agent must implement. The directory also contains private artifacts that are encrypted before runs so the agent can’t see them: tests covering functional correctness, fault-tolerance, and performance, and a reference implementation. After an agent finishes its implementation and passes its own tests, the challenge runner runs the formerly encrypted tests to determine whether the agent succeeded or failed.

Agents are run inside a Docker container with full permissions. We capture every agent invocation’s full transcript, including thinking, tool uses, tool results, and the final response. Thinking is particularly valuable. It’s how we discover failure modes that don’t show up in the produced code, like an agent identifying a fault-tolerance gap, going back and forth on possible solutions, and then saying “this is getting complicated” and failing to address it at all.

Rama has Java and Clojure APIs. We’re focused on Clojure for now but will produce an equally capable Java version of the skills later. The REPL is the main reason, as with a long-running REPL session, the agent evaluates code and inspects results in milliseconds instead of constantly paying for JVM startup and dependency loading. We expect this gap to matter more as challenges get harder and converging on a correct design takes the agent many iterations.

Working on improving LLM performance involves making a new challenge and then iterating on the skill files until the agent passes it consistently. Then we move on to a new challenge that stresses a different type of application, a different capability of Rama, or an application of greater scope.

Current status

Three medium-complexity challenges now one-shot correctly on almost every run, all horizontally scalable and fault-tolerant:

  • Bank transfer: tracks funds for each user and supports deposits, transfers between users, balance reads, and inbound/outbound transfer history reads per user. Transfers must be exactly-once and fault-tolerant: no double-spends under retry, no transfer that lands on one side but not the other under failure, and no negative balances from insufficient funds.
  • Time series: records render latency measurements per URL and answers range-aggregate queries (cardinality, total, min, max) over any minute-bucket range from a single minute to many years. Queries must be fast across the full range of query lengths, whether five minutes or five years.
  • Auction: hosts auctions where sellers list items with expiration times, bidders place bids, and the highest bid at expiration wins. Supports reading a seller’s listings, a listing’s bids and current top bid, and a user’s notifications. Auctions must end automatically when their expiration time passes, and seller / winner / losing-bidder notifications must each be delivered exactly once and in chronological order.

The private tests verify performance and fault-tolerance characteristics using the with-event-hook macro that we added in the latest Rama release. With this we can capture and assert on computation being balanced, the number and types of underlying RocksDB operations, topology types chosen by the agent, force failures/retries, and more.

The bank transfer challenge is the easiest of the three. The main test is whether the agent recognizes that update latency in the hundreds of milliseconds is acceptable, and therefore that a microbatch topology is the right tool. Microbatch topologies have fault-tolerant exactly-once semantics for all computation. The challenge also verifies the agent chooses the optimal PState (Rama’s equivalent to databases) structures, especially that transfer logs are subindexed since they’re unbounded.

The hard part of the time series challenge is pre-aggregating the latency data at multiple granularities and then using a server side distributed query (called a “query topology”) to efficiently compute the range query while reading as few buckets as possible. The agent does a great job of reasoning through how many RocksDB operations will be done depending on the number of buckets read and then choosing the appropriate number of granularities. It also recognizes a query topology is appropriate since that’s more efficient than many roundtrips between the client and worker nodes.

The auction module is the hardest, with multiple features with differing performance characteristics, polymorphic data (for notification types), and time-based behavior. Getting notifications of auction results to have exactly-once delivery with fault-tolerance is easy to get wrong. The agent sometimes lands on something close to the reference design – a stream topology for listings and bids and a microbatch topology for expirations and notifications. What surprised me is it also sometimes produces a design I had never considered. Using just a stream topology, notifications are stored as a map rather than a list, keyed deterministically by the listing’s timestamp, the listing ID, and the notification type. Because the key is deterministic, a retried expiration rewrites the same keys with the same values, achieving exactly-once by making delivery of the notification an idempotent operation.

The published benchmarks show LLMs struggling with far simpler backends. They measure single-instance applications with no scalability or fault-tolerance requirements. Our challenges add exactly-once semantics, fault tolerance, horizontal scalability, and performance constraints that the tests actively verify. With seams removed, the model can spend its reasoning budget on the application requirements rather than a host of random technical details tangential to the application.

Skill files

The skill files we’ve developed have gone through many iterations in order to pass these challenges consistently. At first, they were basically just the Rama documentation translated to Markdown files. The top-level skill had core information needed to program Rama (concepts, dataflow syntax, paths), while less-needed information was put in reference files.

The agent consistently made the same mistakes even when the correct guidance was loaded in context. An agent that reads a reference at the start of a session would fail to revisit it at the moment of a specific decision. Some examples:

  • Rather than refer to the var bound by (defmodule FooModule ...) directly as FooModule , it would try to construct it as (FooModule.)
  • Unbounded locations in a PState would not be subindexed
  • Object would be used for schemas instead of something precise
  • Partitioners would be missing, especially before writes, causing the module to write to the wrong locations

We researched best practices on making skills and then did a major restructuring of the skills to instruct the agent to use a phased approach:

  1. Implicit spec. Derive every edge case and invariant the protocol implies but doesn’t state. Produces an IMPLICIT_SPEC.md document.
  2. Plan. Design the depots, PState schemas, and topologies before writing any code. Produces PLAN.md .
  3. Plan validation. Review the plan against a checklist of scenarios (e.g. race conditions, failures/retries). Produces PLAN_VALIDATION.md with a pass or fail verdict.
  4. Implement. Write the module source, adhering strictly to the plan.
  5. Implementation validation. Review the implementation against a checklist of common mistakes. Produces IMPLEMENTATION_VALIDATION.md .
  6. Tests. Write thorough tests covering every protocol method and every edge case from the implicit spec.
  7. Test validation. Review the test code against a checklist of common mistakes. Produces TEST_VALIDATION.md .
  8. Run tests. Run the test suite.

Three things make this effective. Requiring an artifact at every phase prevents the agent from silently skipping a step. The validation phases are written as explicit checklists, and LLMs are reliable at walking a checklist item-by-item without forgetting entries. And the phase split lets the agent focus on smaller pieces of work at a time.

A few language patterns turned out to make a big difference. Negative constraints with a reason work much better than positive suggestions at preventing the agent from going down wrong paths. Including a reason is critical as otherwise the agent comes up with some random justification to ignore it (e.g. “Do NOT write any topology code in this phase — implementations without plans produce wrong PState schemas, incorrect partition alignment, and unnecessary disk I/O”). We specify explicit default choices and require justification to deviate (e.g. “default to microbatch; if stream, state which API requires it” forces the agent to justify the choice it’s making, while “choose stream or microbatch” lets the agent go with whatever it already decided).

When working on the auction challenge, we started to run into issues with compaction. When the agent would compact in the middle of a run, it would frequently skip critical instructions. So we made some changes to the skill to make it compaction-resistant. Instead of all the instructions living in the skill itself, we changed the skill to instruct the agent to copy pre-written templates for all five artifacts into the implementation directory. Each template contains the full checklist and instructions for filling it in, plus a signpost at the top saying what phase it belongs to and what comes next. The agent’s job is to “fill in” each template, not write from scratch. We also instruct it to put signposts in comments in the implementation files.

After compaction, whatever file the agent reads, whether an unfilled template, the source code, or a partially completed validation, the signposts tell it where it is in the workflow and what to do next. The key insight is that critical instructions living in context get lobotomized by compaction, but instructions embedded in files on disk are permanent. The templates and signposts ensure the agent always has a roadmap. It knows where it came from (which artifacts are already filled in) and where it needs to go next (what the current template is asking for).

The phases, validations, and signposts are a redundant structure that constrain the agent’s search space and funnel it towards a correct solution. There are some critical instructions that we put at the top of the skill to help with that funneling.

One of the most important is emphasizing that the agent is building a production-worthy application. Without this explicit emphasis, the agent would frequently say things like “This code is just for testing locally, so failures are rare and I don’t need to consider them”. We actually lie in the challenge prompt by saying “You are building a production Rama module. It will be deployed under real conditions — node failures, processing retries, concurrent clients, high throughput.”

Other critical instructions at the top of the skill are “Never trade I/O efficiency for code simplicity” and “Never trade fault tolerance for code simplicity”. LLMs have a strong bias for what they consider “simple” code, willing to sacrifice important performance and fault-tolerance properties in the process. In reality, the code to achieve efficiency and fault-tolerance in these challenges barely requires any more code, so these instructions cause the agent to search a little harder to find the right solution.

The rest of the skill is information about Rama that the agent uses to design and implement. One of the most impactful pieces of information was adding latency numbers for RocksDB operations (which Rama uses in the implementation of PStates). We provide approximate numbers for the cost of RocksDB reads, writes, iterator seeks, and iterator reads. During challenge runs, you can see the agent do the math to estimate the cost of various strategies to inform potential design decisions. In the time series challenge, for example, this is how the agent decides to use multiple granularities and how many granularities to use.

We only put general, high-level information in the skills. All examples in the skills are unrelated to the challenges we’ve made. Watching the agent reason from the high-level properties of Rama to implementing correct solutions for each challenge is very cool to see.

Orchestration

Until recently, we had the agent orchestrate itself through all the phases of the skill. We prompted it with the challenge instructions and let it walk the whole process in one session. That stopped working on the latest challenge we’re working on (described below). Once an agent committed to a design early in the session, it tunneled on that design for the rest of the run, even when its own later thinking surfaced problems with it. The validation phases in particular would rationalize away findings with justifications that didn’t make sense.

We made a custom orchestration script to fix this. It invokes the agent once per phase with a fresh context. Each phase reads only its own doc and the artifacts produced by earlier phases, then stops. Validation phases are instructed to be adversarial against the artifacts from the previous agent. The rationalization loops we saw before largely don’t have a place to form, because the agent doing validation is no longer the same agent that wrote the plan or implementation.

As a side effect, much of the compaction-resistance work we did earlier is less relevant. Phases are short enough that compaction rarely fires inside one. That said, the compaction-resistant structure is still worth having if only to lessen the amount of information loaded in context when reading a skill, since so many of the instructions are in the artifacts on disk.

However, there are major issues with the orchestration script which makes me think we’ll ultimately abandon it. First off, it’s unrealistic that this orchestration script would be used by actual developers since it’s contrary to normal usage of coding agents.

It also adds significant latency. Each phase invocation pays a fresh-context cost (re-reading the skill, re-reading prior artifacts, re-loading references) before it can do any real work. In order to lower latency, the script has accumulated a lot of logic about what to do when a validation phase fails. At first the orchestration was strict. A single test failure kicked the run all the way back to the implementation phase, which then had to flow forward through implementation validation, test writing, test validation, and test running again. Most of that re-work was unnecessary, because test failures usually surface small, localized bugs the agent can fix without running most of the phases again. To avoid that overhead we introduced the minor-fail / major-fail distinction in the validation phases (only major changes trigger another adversarial review) and added a “finish” phase at the end where the agent stays in a single context and iterates on the module and tests until the test suite passes. Both changes reduced wasted work, but they grew the state machine considerably.

Instead, we’re thinking of having a top-level agent handle orchestration and delegate to subagents for execution of each phase. We’ll have to make sure we can still capture equivalent transcripts, and we’ll have to see what kinds of mistakes the top-level agent makes during orchestration.

Finally, a couple of small things we include as part of orchestration of the implementation phase are worth mentioning. After the agent finishes writing the module, it must verify the namespace loads cleanly before finishing the phase. Additionally, it runs a linter and fixes all errors before moving on. The linting hooks we made for Rama statically catch things like arity mismatches, using invalid dataflow forms, or referencing undeclared PStates. Including both of these steps as part of the implementation phase saves a lot of time in what would later be a test failure causing phases to retry.

Current challenge

We’re currently working on the “fanout” challenge, a social-media-style backend with profiles, posts, follows, and a per-user merged timeline of posts from accounts the user follows. It’s based on our Twitter-scale Mastodon implementation. The module the agent must implement runs alongside a provided social-graph module that stores the social graph in an optimized way so that fanout can be balanced even for a heavily unbalanced social graph. We wrote in detail how fanout at scale works in this blog post.

The spec requires that merged timelines be kept in memory (writing those to disk drops throughput by 15x), and that fanout be balanced and fair. A post by an account with millions of followers cannot delay fanout for a post by an account with ten. Because timelines are kept in memory, it needs to find an alternate way to make it fault-tolerant. The correct approach for that is to reconstruct lost timelines on read by querying for the recent posts of followees.

The agent has not yet produced an implementation that satisfies the non-functional constraints. It gets the functionality correct and is designing and implementing the reconstruction process, but it’s failing:

  • Minimizing data in the cache and using memory and GC-efficient data structures. The correct solution uses a ring buffer of long values for just the user ID and post ID for each entry in the timeline. The agent so far is always using a TreeMap for the timeline and also storing the post content in the cache, which massively increases memory usage unnecessarily since that can just be fetched in the query topology that fetches a timeline page.
  • Fairness is not handled at all. It eagerly delivers posts to all followers immediately instead of spreading out delivery for large users over a longer period of time, violating the spec.

We’re iterating on more guidance and validation steps to get the agent to make the correct decisions for these. It’s possible these are the kinds of details LLMs are unable to get right, since they’re high-level design decisions that require reasoning about runtime tradeoffs. So it’s fine if we have to make the challenge easier by telling it how to achieve these properties. But it would be better if we can get the agent to come to these conclusions on its own.

Model choice

We’re currently testing with Opus 4.6. We tried 4.7, but its responses include only summaries of the model’s reasoning rather than the raw thinking blocks. That’s a real problem for skill development, as seeing the model’s thought process is critical to determine how to get it to stop going down wrong paths.

We plan to test with other models, especially Codex, once we make more progress.

Open questions

We have many open questions to resolve:

  • How many challenges do we need to be confident we’re not inadvertently overfitting? There’s nothing specific to the challenges in the skills / orchestration, but their structure might be tuned to the particular patterns LLMs follow for these particular problems.
  • Matrix is too big a project to implement the whole thing via one iteration through our phase structure. How should orchestration be structured, and how should the agent split up the work? How does it iterate on a design as it learns more through implementation?
  • The Matrix test suite will be helpful not just for evaluating a Matrix implementation, but it could be helpful for the agent to use during development to iterate and check exact behavior. Will we be able to get Matrix to one-shot without access to the tests?
  • What’s the best way for developers to use these skills in practice? How should artifact generation be handled?
  • How can larger projects be efficiently parallelized?
  • What’s the impact of generating Rama applications that need to integrate with lots of existing infrastructure?

LLM usage during development

LLMs are helpful for building this project, though they can be actively detrimental if not used properly.

The most useful application has been transcript analysis. We had the LLM build a transcript analysis script (scripts/analyze-latest-transcript.py) with subcommands for searching thinking blocks, tool calls, validation artifacts, and so on. The LLM then uses that script freely when we ask it to dig into why an agent made a particular mistake without us needing to give permission constantly for it to run custom shell or Python commands to analyze transcripts.

We also use LLMs to build the harness and tooling for the project. LLMs built the orchestration script that runs challenges, and they’re helpful for assisting in making new challenges.

We also have an alignment-scoring pass that runs after each challenge. It compares the agent’s implementation and tests to the reference implementation and tests, producing a short summary of differences. It’s not a significant part of the workflow, but reading a short summary is a faster way to spot a regression than diffing files directly.

LLMs are useful for brainstorming skill updates after we identify a failure mode, but only as assistants. They’re not good at understanding why they made a mistake. They always think they have an answer, even when the problems are subtle. They cheat like crazy, trying to add challenge-specific info or examples into generic reference files. So LLM suggestions can help with critical analysis, but they don’t substitute for it.

Conclusion

I expect future progress reports to be much shorter than this one, since I’ll just talk about new progress. I hope these progress reports are useful to anyone using LLMs or developing skills of their own, and I hope to get useful ideas and feedback from anyone following along.

Permalink

Data Governance in Versioned Systems

Data Governance in Versioned Systems

May 2026

If a database remembers everything, how do you delete things?

This is the first question people ask when they encounter immutable data systems. It’s a fair question. GDPR right-to-erasure requests, accidental PII ingestion, retention policies — these aren’t edge cases. Any system that stores data for real workloads needs to answer them.

The second question is access control: who gets to see what? A database that lets you fork, branch, and time-travel is powerful, but it also means the attack surface is wider than a traditional system where yesterday’s state is already overwritten.

This note describes what works today across Datahike and its ecosystem, what’s planned, and what’s not solved yet.

Purge in Datahike

Datahike has a purge operation that removes datoms from a database’s indices — both the current and the history index of the resulting commit.

What is this syntax? 
require('[datahike.api :as d])

;; Before purge
d/q('[:find ?n :where [?e :name ?n]] @conn)
;; => #{["Alice"] ["Bob"]}

;; Purge Alice (requires :keep-history? true)
d/transact(conn [[:db.purge/entity [:name "Alice"]]])

;; After purge — gone from current state AND from the new commit's history
d/q('[:find ?n :where [?e :name ?n]] @conn)
;; => #{["Bob"]}

d/q('[:find ?n :where [?e :name ?n]] d/history(@conn))
;; => #{["Bob"]}
(require '[datahike.api :as d])

;; Before purge
(d/q '[:find ?n :where [?e :name ?n]] @conn)
;; => #{["Alice"] ["Bob"]}

;; Purge Alice (requires :keep-history? true)
(d/transact conn [[:db.purge/entity [:name "Alice"]]])

;; After purge — gone from current state AND from the new commit's history
(d/q '[:find ?n :where [?e :name ?n]] @conn)
;; => #{["Bob"]}

(d/q '[:find ?n :where [?e :name ?n]] (d/history @conn))
;; => #{["Bob"]}

Two things purge is not. It’s not a soft delete: it rewrites the affected path through the persistent sorted set, and the new commit’s index roots no longer reach a node containing Alice. It’s also not retroactive across the commit graph: each commit owns its own index roots, so the pre-purge commit still points at the old nodes containing the datom. Until d/gc-storage sweeps that intermediate commit, the bytes remain on disk and (d/commit-as-db conn <pre-purge-uuid>) can still see Alice. The recipe in the next section closes that loop.

How DELETE disposes of data — and how purge differs

Compare to a traditional DELETE. In PostgreSQL, DELETE marks the row’s heap tuple dead; the bytes live on in several places:

  • In the heap page as a dead tuple. Plain VACUUM marks the space reusable and may defragment within the page, but never returns space to the OS or zeroes bytes. VACUUM FULL rewrites the table into a new file; the old file is unlinked but its disk blocks are not zeroed.
  • In WAL segments until they’re recycled, and indefinitely in archived WAL if archive_mode=on.
  • In replicas, which received the original WAL and run their own VACUUM cycle.
  • In backups — base backups, pg_dump archives, PITR base + WAL.

Proving the row is gone requires forensic checks across each of those layers, and there’s no manifest of where it lived.

Purge changes one part of that picture: in the live store, deletion is explicit and structurally locatable. The persistent sorted set is a tree of content-addressed nodes; the purge transaction rewrites the affected path; the new commit’s roots no longer reach a node containing the datom; a subsequent gc-storage with an appropriate cutoff sweeps the unreachable intermediate commits. The commit graph is itself a manifest — you can walk every branch head’s ancestors and list precisely which snapshots ever held the datom.

Where Datahike does not differ from PostgreSQL: backups, storage backends with their own versioning, and replication targets. We come back to that in Backups and storage-layer history.

Garbage collection in Datahike

d/gc-storage is the sweep that physically reclaims storage. The full mechanics — cutoff dates, branch-heads-always-kept, online vs. offline GC — are in Branches as Values, Merges as Queries. The piece that matters for governance is how it composes with purge:

What is this syntax? 
require('[superv.async :refer [<?? S]])

;; 1. Purge — produces a new commit whose roots no longer reach Alice.
d/transact(conn [[:db.purge/entity [:name "Alice"]]])

;; 2. gc-storage WITHOUT a cutoff only reclaims storage on deleted branches.
;;    The pre-purge commit on :db is a live intermediate commit, so its
;;    tree nodes (containing Alice) survive this sweep.
<??(S d/gc-storage(conn))

;; 3. gc-storage WITH a cutoff is what physically evicts those nodes.
;;    Pick a cutoff that exceeds your longest-running reader's lifetime.
let [seven-days-ago new java.util.Date(System/currentTimeMillis() - 7 * 24 * 60 * 60 * 1000)]:
  <??(S d/gc-storage(conn seven-days-ago))
end
(require '[superv.async :refer [<?? S]])

;; 1. Purge — produces a new commit whose roots no longer reach Alice.
(d/transact conn [[:db.purge/entity [:name "Alice"]]])

;; 2. gc-storage WITHOUT a cutoff only reclaims storage on deleted branches.
;;    The pre-purge commit on :db is a live intermediate commit, so its
;;    tree nodes (containing Alice) survive this sweep.
(<?? S (d/gc-storage conn))

;; 3. gc-storage WITH a cutoff is what physically evicts those nodes.
;;    Pick a cutoff that exceeds your longest-running reader's lifetime.
(let [seven-days-ago (java.util.Date. (- (System/currentTimeMillis)
                                         (* 7 24 60 60 1000)))]
  (<?? S (d/gc-storage conn seven-days-ago)))

This is the recipe most people get wrong on first read: purge + cutoff-GC, not purge alone. Plain gc-storage is a safe maintenance op — it leaves intermediate commits alone — which is exactly why it doesn’t finish the job for erasure. The pre-purge commit gets swept on the first cutoff-GC pass after it ages out of the grace window. In practice that means erasure has a tail measured in your GC cadence, not in milliseconds, which most compliance regimes accept.

The cutoff has to comfortably exceed your longest-running reader’s lifetime. Datahike’s distributed readers walk storage directly without coordinating with a writer, so a snapshot vanishing mid-query is a real failure mode.

Branch heads are always kept regardless of cutoff. So the recipe assumes the post-purge state is the head you want to keep; if the datom also exists on another branch, you purge there too before sweeping.

Secondary indices

Datahike’s secondary indices — Scriptum (Lucene full-text), Proximum (HNSW vector), Stratum (columnar) — are first-class versioned state. Indices are CoW-forked on branch, persisted with each commit, and restored on connect.

For governance, the question is whether purge propagates. It does: a purge transaction routes a retraction event (-transact with :added? false) to every secondary index covering an affected attribute, the same way :db/retract does. After purging Alice on a database with a Scriptum index over :person/name and :person/bio, a full-text search for “Alice” returns nothing, a vector KNN over her embedding skips her, and any columnar aggregate excludes her row.

On storage reclamation, Stratum and Proximum are konserve-backed: d/gc-storage sweeps their unreachable blobs alongside the primary indices, following the same pattern (Stratum: columnar rewrite; Proximum: HNSW mark-delete).

Scriptum is the exception. Its Lucene segments live on the writer node’s local filesystem, outside konserve. Scriptum’s -sec-mark returns the empty set, so d/gc-storage can’t reach them, and Lucene’s own delete model is tombstones-until-segment-merge — the bytes linger inside a segment file until Lucene merges that segment away. For full erasure on Scriptum you may need to force a segment merge and make sure the writer’s filesystem snapshot policy doesn’t pin old segments.

Backups and storage-layer history

Datahike does not escape the backup and archive problem. A backup of the konserve store taken before purge+GC contains the pre-purge nodes. Storage backends with their own versioning hold them too:

  • S3 object versioning, if enabled, retains every overwritten key as a prior version.
  • ZFS / btrfs snapshots of the konserve directory retain the pre-purge tree.
  • A git-backed konserve backend retains the pre-purge commit graph in .git even after a purge rewrite on the working tree.
  • Logical replicas of the konserve store hold whatever they received.

Purge + cutoff-GC reach the live store; erasure across backups and storage-layer history is a separate procedure — identify the destinations that contain the datom, then replay the purge against them or rewrite them. PostgreSQL doesn’t ship a tool for this either; in any storage-versioned environment it’s an operational policy question, not a database feature.

The common operational workaround is crypto-shredding — encrypt sensitive values per-user and “delete” by destroying the key. The bytes remain in WAL and backups but become unreadable; available to Datahike users on the same terms, with the same regulatory gray zone about whether key destruction counts as deletion.

What Datahike does give you here is a manifest of where the datom lived. Because history and branches are first-class, you can walk the commit graph from every branch head and list every snapshot that ever referenced the datom. That’s not what you get in PostgreSQL, where “which copies of this row exist?” is answered by inspecting WAL archives, backups, and replicas as independent unstructured investigations.

Multi-branch purge

Purge is a transaction on a single branch. If the same datom is reachable from another branch’s head, or from any of its commits inside the GC window, it lives there too. Structural sharing means the bytes are physically one copy in konserve — but the paths to reach them are independent.

The practical procedure: purge on every branch you control, delete branches you no longer need, then run cutoff-GC. Data reachable only from deleted branches gets reclaimed.

For databases with hundreds of agent-created branches, this gets expensive. Each branch’s purge is its own transaction and writes its own tree path. Optimization for the agent-fanout case is on the roadmap.

Access control

GDPR also requires access controls (Articles 15, 32). Today this works at the connection and storage level: storage credentials gate database access; the :branch config field (or SET datahike.branch over SQL via pg-datahike) pins a connection to a specific branch; commit-as-db (or SET datahike.commit_id) pins a specific snapshot for audit-only roles; read vs. write is per-connection.

Row-level access — “user X can query this branch but shouldn’t see rows where :department = "HR"” — applied consistently across current state, history, branches, and secondary-index queries is the next layer. EACL-style ReBAC is one direction we’re exploring.

Why immutable is still useful for compliance

The strong-form claim — “you can prove the data is gone” — doesn’t survive the backup and storage-history caveat. The narrower claim does, and it’s the one worth making.

In a mutable database, “which copies of this row ever existed?” has no manifest. You inspect WAL archives, replicas, backup catalogs, and page slack independently; each is a separate forensic exercise.

In a Datahike database, the copies are enumerable. History is explicit and branches are explicit. You can list — by walking the commit graph from every branch head — every snapshot that ever held the datom. Once enumerated, each can be addressed individually:

  • Live store — purge + cutoff-gc-storage rewrites the index path and sweeps the intermediate commits; commit-as-db lookups for swept UUIDs fail cleanly.
  • Secondary indices — covered by purge propagation on the konserve-backed ones; Scriptum needs the segment-merge / filesystem step.
  • Backups, S3 versioning, ZFS snapshots, git-backed stores — each is a separate destination addressed by policy.

The advantage isn’t “fewer copies than PostgreSQL” — usually it’s more copies, deliberately. The advantage is knowing what those copies are. “Show me the data is gone” becomes a checklist instead of an investigation.

What’s not solved yet

  • Multi-branch purge at scale is expensive. Hundreds of branches means hundreds of transactions, each rewriting its own tree path. Needs optimization work for agent-fanout workloads.
  • Scriptum filesystem GC. Lucene segments live outside konserve; d/gc-storage can’t reach them, and Lucene’s delete model is tombstones-until-merge. Filesystem cleanup is a separate operational step.
  • Row-level access control is the next layer. Connection / branch / commit-level access works today; fine-grained ReBAC across current state, history, branches, and secondary indices is the work in flight.
  • No retention-policy automation. There’s no built-in “delete everything older than 7 years.” :db.history.purge/before is the primitive; the policy loop is application code.

These are real limitations and we’re working on them.

Summary

Versioned, immutable data systems need to answer the deletion and access control questions directly. Datahike’s answer for the live store is explicit, structurally locatable deletion: purge rewrites the index path, cutoff-gc-storage sweeps the intermediate commits, secondary indices receive the retraction event, and the commit graph gives you a manifest of where the datom lived.

Where Datahike doesn’t differ from PostgreSQL is on backups, storage-layer versioning, and replicas — those still need a policy in either system. Where it does differ is in giving you a structured catalog of every copy on the live store: an enumerable checklist instead of an investigation. That’s the case for immutable data systems on compliance — not magic, but knowable.

Permalink

Advent of Code 2015 days 3-8 in Clojure

Yeah, I'm starting to think my initial comment about taking until the heat death of the universe to complete this wasn't far off the mark. I'm not really slow at finishing the tasks, it's just between working , the drudgery of everything else in life and possible undiagnosed ADHD, progress becomes a sporadic thing. Gone forever I think are the late-nighters of chugging caffeinated drinks...

But with that said, I have been slowly chugging away and gotten myself up to the heady heights of puzzle number 8.

I don't think I shall labour over each and every puzzle, instead I'll just gloss over the interesting parts.

Day 03

You're trying to determine how many points are visited if you're given a list of directions (N/S/E/W effectively). One pattern with this sort of thing I like to do is to store the position transformation as a map with the keys being the direction and the values being values to add to x and y coordinates.

(def moves {\^ [0  1]
            \v [0 -1]
            \> [1  0]
            \< [-1 0]})

The rest is just bookkeeping ;D

Day 04

Trying to mine a proprietary cryptocoin. I just used Java interop here - (java.security.MessageDigest/getInstance "MD5"). It is then a matter of looping through numbers to get the required number of leading zeros in the hexadecimal representation of the digest.

This was pretty brute-force and I don't expect to retire from crypto riches any time soon. It's quite slow too - clearly the hashing is going to be but I suspect we could possibly run multiple threads in parallel and/or optimise the way we're creating the byte-array to feed into Java each loop.

As it is, I was content just to wait for it to finish...

Day 05

String parsing really. I'm not a massive fan!

Was nice going back to see part of my original solution looks like this;

(defn nice?
  "is a word nice?"
  [word]
  (and (has-three-vowels? word)
       (has-repeated-letter? word)
       (no-forbidden-strings? word)))

Pretty legible. I guess you'd write something similar in most languages, but the convention of adding a question mark to tests is a nice touch.

Day 06

Mapping text instructions into turning a series of lights on/off/toggling in a given rectangle. Nothing to write about but I was interested to see if anyone had plotted the result in case there was a picture.

There isn't.

Day 07

Ah, we're getting interesting now, we had to simulate a basic logic circuit. We're given a input like;

123 -> x
456 -> y
x AND y -> d
x OR y -> e
x LSHIFT 2 -> f
y RSHIFT 2 -> g
NOT x -> h
NOT y -> i

I suspect there's a proper way of doing this in which we create a graph of connections and evaluate everything in the proper order. But what I did was just to brute force the whole thing; just keep running through the instructions until every instruction could be executed e.g. if you imagine in the example above that the numbers aren't applied to x and y until the last line, then we couldn't perform most of it until the second loop round the instructions.

Yes, as I edit this, I'm seeing a theme forming here where my solutions are just brute forced.

Day 08

String parsing again :D Trying to count the number of characters in a string and in the final representation. Not much to say again. Slogged through it.

Parser?

So far each problem's input is basic text, but I'd be interested to see if there's a tidy way of specifying the puzzle input. For example Day 2's input looks like;

1x2x3
1x3x6
...
...

So I'm thinking we almost want a function like;


(parse-input "day02.input" "<int>x<int>x<int>")

;; -> ((1 2 3) (1 3 6) ... ...)

Usually I'd either split or use regex for input, but it seems like what I'd like is to also be able to specify the basic type. Puzzle 8's example;

London to Dublin = 464
London to Belfast = 518
Dublin to Belfast = 141

Would be;

(parse-input "day08.input" "<string> to <string> = <int>")

;; -> (("London" "Dublin" 464) ...)

Or even better!

(parse-input "day08.input" "<string:from> to <string:to> = <int:distance>")
;; -->
({:from "London" :to "Dublin" :distance 464} ...)

I'd be interested if you know of anything like this feature.

End

I'll carry on with some more problems. I don't think I'll be entering any leaderboards soon ... unless there are some for slowest?

Permalink

Clojure Deref (May 26, 2026)

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem (feed: RSS).

Upcoming Events

Podcasts, videos, and media

Babashka Conf

Mike Møller Nielsen

Blogs, articles, and news

Libraries and Tools

Debut release

  • ring-node-adapter - Ring adapter for Node.js

  • neat - A small, language-agnostic nREPL client for Emacs

  • code-bubble - A Clojure dev-time UI inspired by Code Bubbles

  • clj-code-search - Per-function semantic index for Clojure/CLJS, exposed as a code-search CLI for AI coding agents.

  • phel-doom - Terminal DOOM-lite raycaster built in Phel (Lisp on PHP). 256-color ANSI, 5 procedural levels — a real-world Phel showcase app.

  • gojava - A Go reimplementation of selected parts of the Java standard library

  • clojure-android - The Clojure programming language for Android

  • clojure-android-demo - Clojure Android Demo

  • muschel - Bash interpreter with git-like memory and fine grained access control.

  • fulcro-tui - A JLine-base TUI wrapper, allowing you to write TUI projects in CLJ or Babashka via Fulcro

Updates

  • core.unify 0.7.3 - Unification library

  • tools.gitlibs 2.6.217 - API for retrieving, caching, and programatically accessing git libraries

  • glogi 1.4.177 - A ClojureScript logging library based on goog.log

  • openapi-v3-validator 0.2.7 - A pure-clojure library for validating ring requests & responses against OpenAPI v3 specifications.

  • mailman 0.4.0 - Simple event routing library

  • refactor-nrepl 3.13.0 - nREPL middleware to support refactorings in an editor agnostic way

  • fulcro-spec 3.2.10 - A library that wraps clojure.test for a better BDD testing experience.

  • drip 1.0.0-alpha41 - Drip is a transactional job queue for MariaDB, PostgreSQL, and SQLite

  • cursive 2026.2-eap1 - Cursive: The IDE for beautiful Clojure code

  • dtype-next 11.022 - A Clojure library designed to aid in the implementation of high performance algorithms and systems.

  • tech.ml.dataset 8.023 - A Clojure high performance data processing system

  • rephrase 1.0.2 - Rephrase exceptions to be more beginner-friendly

  • bareforge 0.7.0 - Companion visual builder for BareDOM web components. Drag components, declare reactive state, export fully interactive CLJS or JS project

  • statecharts 1.4.0-RC16 - A Statechart library for CLJ(S)

  • stratum 0.3.72 - Versioned, fast and scalable columnar database.

  • teensyp 0.5.4 - A small, zero-dependency Clojure TCP server that uses Java NIO

  • generate 1.0.57 - code generation for Clojure projects

  • r11y 1.0.7 - CLI tool for extracting URLs as Markdown

  • cli 0.9.68 - Turn Clojure functions into CLIs!

  • html 0.2.4 - Html generation library inspired by squint’s html tag

  • next-jdbc 1.3.1108 - A modern low-level Clojure wrapper for JDBC-based access to databases.

  • ring-jetty9-adapter 0.39.4 - An enhanced version of jetty adapter for ring, with additional features like websockets, http/2 and http/3

  • svar 0.5.9 - Type‑safe LLM output for Clojure. Works with any text‑only model.

  • neil 0.3.70 - A CLI to add common aliases and features to deps.edn-based projects

  • gloat 0.1.44 - Glojure AOT Tool

  • encore 3.161.0 - Core utils library for Clojure/Script

  • clj-kondo 2026.05.25 - Static analyzer and linter for Clojure code that sparks joy

  • phel-lang 0.40.0 - A functional, Lisp-inspired language that compiles to PHP. Inspired by Clojure, Phel brings macros, persistent data structures, and expressive functional idioms to the PHP ecosystem.

  • spindel 0.1.13 - Cross-platform FRP runtime with a git-like memory model.

  • memento 2.1.74 - Clojure Memoization project

  • baredom 3.3.0 - BareDOM: Lightweight CLJS UI components built on web standards (Custom Elements, Shadow DOM, ES modules). No framework, just the DOM

  • fulcro 3.10.0-RC1 - A library for development of single-page full-stack web applications in clj/cljs

Permalink

libwce: the entropy layer of a wavelet codec, on its own

Most image codecs you know about such as JPEG, JPEG 2000, JPEG XS, WebP are like layer cakes. You have transform sitting on top, entropy coding at the bottom, and rate control floats somewhere in the middle. And then there's a metadata layer wrapping it all up. The interesting bits are hidden under tons of framing code, profile parsers, and standards plumbing. If you just want to see how wavelet coefficients become bits, you have to dig deep into the guts of the codec.

I wrote libwce as a bare-bones implementation, consisting of only a single lib.rs file, weighing in at 500 lines. It just implements a patent-clean Bit-Plane Count (BPC)-style entropy layer in the spirit of JPEG XS, and nothing else. There is no boilerplate or dependencies with the library relying solely on stdlib.

A two-minute primer on BPC coding

A raw video stream is basically a grid of pixels, most of which share very similar color and brightness values with their immediate neighbors. Storing every pixel individually wastes a ton of bandwidth since there is a lot of repeated data in the stream. Codecs are used to compress this information by transforming the image from the spatial domain into the frequency domain. Rather than tracking individual pixels, a codec uses mathematical frequencies to describe color changes across the image. Older formats like standard JPEG end up chopping the image into squares and applying a discrete cosine transform, leading to the blocky artifacts we all know and love.

A wavelet is a newer approach that solves the problem by applying the transform process to the whole image at once, splitting the signal into low-frequency structural data and high-frequency detail data across multiple scales. After the wavelet transform, you end up with a 2D array of signed integer coefficients, most of which are near zero, with a long Laplacian tail. The purpose of the entropy layer is to compress this array down to a small number of significant bits.

BPC coding is done using groups of four coefficients at a time. For each group, you have to determine the smallest bpc such that every coefficient can be held. This is the bit-plane count representing the index above which all coefficient bits in the group are zero. In libwce, all the bpc values are written first into a single bitstream, then for each group the four coefficients are emitted coeff-major. These are the magnitude bits of each coefficient followed immediately by a single sign bit when that coefficient is nonzero. That takes care of all the data processing you need to do. Then, you get to the actual compression when you go to encode these bpc values. Neighboring groups tend to have similar sizes, so instead of writing each bpc as a raw 6-bit number, you can estimate it from its neighbors and, instead, write a small residual which tends to be tiny.

Here, libwce uses RUNNING (DPCM delta vs the previous group's bpc, zigzag-mapped and Rice-coded) and ZERO (unsigned residual against lossy_bits) predictors which can be optionally combined with a 1-bit-per-8-group sparse-block flag that short-circuits all-deadzone blocks. That leaves you with four predictor × flag combinations, and the encoder sweeps Rice-k across seven values inside each, picking the best per band via a single-pass cost search. All combinations give the same decoded result, but they produce different types of bitstreams. Each one works best for different pixel distribution such as textured regions, flat parts, or sub-bands which are mostly zeros.

What it looks like to use

Here's a complete decoder for one sub-band:

let mut coeffs = vec![0i32; N];
let lossy_bits = decode(buf, &mut coeffs).unwrap();
dequantize_optimal(&mut coeffs, lossy_bits, scale_b);

The library itself is stateless, and only works with whatever buffers you provide. It doesn't use I/O or hidden globals, and works purely through caller-owned buffers (a small BPC scratch buffer is allocated internally).

Compressing an image end-to-end

The repo has 3 demos. The most fun one is image_compress, which is a full codec built on top of libwce. It uses Haar wavelet in, libwce in the middle, and inverse Haar on the way out which run across four quality presets.

  preset          lossy_bits      payload     .wce file    ratio    PSNR
                  LL  HL  LH  HH    bytes        bytes
  near-lossless    2   4   4   5    146537       146597    1.52x   49.06 dB
  balanced         4   6   6   7     92631        92691    2.40x   37.54 dB
  aggressive       6   8   8   9     49516        49576    4.48x   28.79 dB
  very lossy       8  10  10  11     21923        21983   10.11x   21.62 dB

The whole process consisting of DWT, sub-band coding, quantization, and writing to a container takes under 500 lines of code. If you open the four reconstituted PGMs side by side and you'll see quality degrade as compression increases. At q1, the image will be indistinguishable from the original; q2 has minor smoothing in flat areas; q3 starts to show noticeable wavelet ringing around edges; and q4 is blocky in a recognizable wavelet way, looking eldritch but still legible.

The second demo, mode_shootout, runs a synthetic Laplacian sub-band through every predictor × flag combination and displays the winner.

  mode             total   ratio   ok
  --------------   -----  ------   --
  RUN, flag=off      658   12.45x   Y
  RUN, flag=on       666   12.30x   Y
  ZERO, flag=off     652   12.56x   Y
  ZERO, flag=on      660   12.41x   Y
  auto-pick          612   13.39x   Y

  best forced: ZERO, flag=off  (652 bytes)
  auto-pick beat best forced by 40 bytes (better rice_k).

This is precisely the kind of thing that's a pain to do within the confines of a full codec, where you’d have to fiddle with instrumenting internals, disable rate control, and then mock the framing layer. With libwce, mode comparison is just how the API works. You use the same sub-band through encode_with_options with each predictor × flag combination, then count the bytes and pick the winner, which is exactly what encode itself does internally.

The third demo, stream_surgery, does 256 random bitflips and 256 random byte scrambles across the encoded bitstream, 300 truncation points covering every 4-byte prefix, and a set of adversarial cases including all-ones “unary bombs” along with crafted bad headers.

  bit-flip (anywhere)           : 256/256 returned, avg 36 / max 1024 coeffs differ (of 1024)
  random byte (anywhere)        : 256/256 returned without crash
  truncation (every prefix)     : 300/300 prefix lengths returned
  adversarial (bombs + bad hdrs): 7 cases returned cleanly

The demo shows how every case gets successfully decoded without any hangups or a crash.

What it isn't

Finally, it's worth reiterating that I intentionally didn't write libwce to be a full codec implementation, which would necessitate adding a container format, rate control, and other plumbing. It's designed to illustrate how the most conceptually interesting layer of a mezzanine codec works and to make it easier to study and modify without the weight of the full codec around it. What you get is just the entropy layer that you can wire into your own pipeline.

The repo is at https://github.com/yogthos/libwce. Clone it and play with it. It's written with readability in mind.

Permalink

The Age of Accountable Agents: Building Trust in Your AI Automation

The Age of Accountable Agents: Building Trust in Your AI Automation\n\nThe air around AI feels different this "Long Hot A.I. Summer." Big tech is pouring billions into development-Elon Musk's legal battles, Meta reassigning 7,000 employees to focus on AI-it's a high-stakes, high-energy environment. But for us, building powerful AI agents on consumer hardware, this moment isn't just about raw computational power or complex models. It's about something more fundamental: trust.\n\nThe recent news cycle offers a stark reminder of the ethical considerations, user control challenges, and privacy implications that come with advanced automation. From a papal encyclical discussing AI's moral implications to significant settlements over hard-to-cancel subscriptions, and even debates around nationwide data collection, the narrative is clear: we're entering the Age of Accountable Agents. And as developers, especially those focused on local, user-centric AI, we have a unique opportunity to lead the charge.\n\n## Trust Through Transparency: The AI Encyclical's Echo\n\nWhen you hear about an Anthropic co-founder discussing AI ethics with the Pope, it's a signal that the impact of our work extends far beyond our terminals. AI agents, by their nature, automate decisions. For these agents to be truly valuable and accepted, they must be transparent.\n\nWhat does transparency mean for an agent running on your hardware? It means:\n\n* Clear Decision Paths: Can a user understand why their agent took a particular action? If your agent automatically categorizes emails, can it explain its reasoning?\n* Auditable Logic: Even if not a full "explanation," the underlying logic should be inspectable. This doesn't mean revealing proprietary secrets, but designing agents where state changes and rule applications are explicit.\n\nConsider an agent designed to manage your smart home devices. Instead of a black box, you could implement a simple logging mechanism:\n\n


python\nclass SmartHomeAgent:\n def __init__(self, name):\n self.name = name\n self.log = []\n\n def act_on_temperature(self, current_temp, desired_temp):\n if current_temp > desired_temp + 2:\n action = "Turning on AC"\n self.log_action(action, f"Current: {current_temp}°C, Desired: {desired_temp}°C")\n # ... actual AC control code\n elif current_temp < desired_temp - 2:\n action = "Turning on Heater"\n self.log_action(action, f"Current: {current_temp}°C, Desired: {desired_temp}°C")\n # ... actual Heater control code\n else:\n action = "No action needed"\n self.log_action(action, f"Current: {current_temp}°C, Desired: {desired_temp}°C")\n return action\n\n def log_action(self, action, details):\n self.log.append(f"[{self.name}] {datetime.now()}: {action} - {details}")\n\n# Usage\nagent = SmartHomeAgent("ClimateControl")\nagent.act_on_temperature(25, 22)\nprint(agent.log)\n

\n\nThis basic logging provides a human-readable trail, fostering trust by showing, not just doing.\n\n## User Autonomy, Not "Hard-to-Cancel": Learning from Shutterstock\n\nThe $35 million settlement Shutterstock faced over difficult subscription cancellations is a potent lesson: users demand control over automated systems. For AI agents, this translates directly to how we design interaction and management. Your agent shouldn't feel like a digital trap.\n\nKey design principles for user autonomy:\n\n* Explicit Opt-in/Opt-out: Clear consent for agent actions and data usage.\n* Easy Pause and Stop: Users must be able to halt or reconfigure an agent's operation immediately.\n* Understandable Configuration: Agent settings should be accessible and intuitive, not buried in obscure files.\n\nThink about how your agent's lifecycle is managed. Here's a conceptual AgentController:\n\n

python\n# pseudo-code for an AgentController\nclass AgentController:\n def __init__(self, agent):\n self.agent = agent\n self._running = False\n\n def start(self):\n if not self._running:\n print(f"Starting {self.agent.name}...")\n self._running = True\n # thread or process start logic for agent.run()\n self.agent.start_service()\n\n def pause(self):\n if self._running:\n print(f"Pausing {self.agent.name}...")\n self._running = False\n self.agent.pause_service()\n\n def stop(self):\n if self._running:\n print(f"Stopping {self.agent.name} permanently...")\n self._running = False\n self.agent.stop_service()\n # Clean up resources\n\n def configure(self, new_settings):\n print(f"Configuring {self.agent.name} with new settings.")\n self.agent.update_settings(new_settings)\n\n# When you're building your agents, consider how these controls are exposed to the user.\n

\n\nFor more effective agent management, especially concerning permissions and operational boundaries on local hardware, check out AgentGuard. It helps you build in these essential controls from the ground up.\n\n## Privacy by Design, Not by Accident: The FBI's Data Ambition\n\nThe FBI's desire for nationwide license plate reader access is a stark reminder of the sheer scale of data collection possible today. For local AI agents, privacy should be a default setting, not an afterthought.\n\nWhen designing your agents, prioritize:\n\n* Local-First Processing: Perform computations and store data on the user's device whenever possible.\n* Data Minimization: Only collect and process the data absolutely necessary for the agent's function.\n* Transparent Data Policies: Clearly communicate what data an agent uses, why, and whether it ever leaves the device.\n\nBuilding agents for consumer hardware gives us a distinct advantage here. We can champion local intelligence and ensure that user data stays private by default, not by policy fine print.\n\n## Architecting for Clarity: The Lisp Connection\n\nThe Lisp family of languages (Common Lisp, Racket, Clojure) are hyperpolyglots for a reason: their power in symbolic computation and metaprogramming encourages clarity in expressing complex logic. While you might not be writing your agent in Emacs Lisp, the principles of clear, inspectable, and modular design are paramount.\n\nAn agent with well-defined modules for perception, decision-making, and action is easier to debug, understand, and, crucially, to trust. Avoid monolithic codebases where an agent's reasoning is opaque.\n\n

python\n# Conceptual Agent Architecture\nclass AgentBrain:\n def __init__(self, perception_module, decision_module, action_module):\n self.perception = perception_module\n self.decision = decision_module\n self.action = action_module\n\n def run_cycle(self, environment_data):\n perceived_state = self.perception.process(environment_data)\n desired_action = self.decision.evaluate(perceived_state)\n self.action.execute(desired_action)\n return desired_action # For logging/traceability\n\n# Each module can have its own transparent logic, making the overall agent's behavior understandable.\n

\n\n## The Intentional Click: Feedback Loops and Refinement\n\nEven a seemingly simple site like clickclickclick.click can serve as a quirky reminder of direct user interaction. How do your agents confirm intent? How do they solicit feedback effectively? It's not about mindlessly automating every single interaction, but about designing clear, intentional communication channels between the user and the agent.\n\nConsider points where your agent might ask, "Did I do that correctly?" or "Is this what you intended?" rather than just assuming. This explicit feedback loop refines the agent's understanding and reinforces the user's sense of control.\n\n## Building for a Trustworthy AI Future\n\nThe AI revolution is here, and it's happening everywhere, from the largest data centers to the devices in our pockets. As developers crafting AI agents for consumer hardware, we stand at a critical juncture. We have the unique opportunity-and responsibility-to build agents that are not just intelligent and efficient, but also trustworthy, accountable, and respectful of user autonomy and privacy.\n\nThis summer's "AI gold rush" shouldn't just be about speed; it should be about quality, ethics, and user-centric design. By focusing on transparency, control, and privacy by default, we can ensure our AI agents truly empower, rather than overwhelm, the people who use them.\n\nTo help manage these critical aspects of your agent's lifecycle, from permissions to operational safety, explore AgentGuard. It's designed to support you in building the next generation of conscientious AI automation. Start building agents that earn trust, today.\n

Permalink

Preemptive commoditization

Note: I wrote this in 2018, while I was helping a young startup and convincing them to open source their core application. If I were to write it nowadays, in 2024, I’d elaborate a bit more, including James Governor’s evergreen remark about how “You can make money with open source, but it’s extremely hard to make money from open source.” (source). Leaving it as-is, however.*

I’ve been advising a startup on the data transformation space. As part of this, we re-wrote the core engine in Clojure. The new version is, at the worst case, 16 times as fast in the same hardware, and in some cases over 200 times faster. And it does it in a fraction of the lines of code.

We did this in under 3 months of part-time work. We couldn’t focus our entire attention on it, as we had other concerns as well - I was involved with general team and management tasks, and the second developer was helping on other internal projects as well. To further raise the bar: we had to keep it functionality-compatible with the current version, so I had to get acquainted with the existing feature set, and it was the other developer’s first Clojure project.

Clojure made our lives so much easier. But this is not a post about why Clojure is cool.

I’ve been arguing about why they should open source anything that is not enterprise-specific, including this layer. There’s many advantages, which I won’t go over right now, but there’s also a looming threat.

Layers are getting commoditized faster and faster. More and more, there is demand for people who are good at wiring things together (beyond the gem install hairball approach), or tools that help with that wiring.

I suspect that’s a big part of what’s driving how many companies like Seldon are going open source-first, or how Unreal opened their code as a way to compete with Unity.

“what’s driving how many… are going” is a doozy

Remember: This rewrite took about 3 man-months, with our attention pulled in multiple directions, while we strove to remain feature-compatible with the old engine. You have to assume anyone else who has the technical chops but doesn’t have that baggage can do it as well. Do you want to be disrupted by some motivated, random person who thought what you were doing was cool, but neither needed your entire feature set nor wanted to be shackled to your cloud version?

Better to commoditize yourself before someone else does it to you. You get to have a say in how it happens, use it as fuel to propel you somewhere new. And you get to tell your customers “if you think this thing we give away is cool, you should see the part we charge for”.

Permalink

The framework mirage

OK, this is going to be a tough one. Strap down, even if it pisses you off at first - I wouldn’t be writing it if I didn’t think it was helpful.

Stop me if you’ve heard this one before

A programmer has something he wants to build.

He finds a framework.

It looks easy enough to get into. There’s a few tutorials available, which show step by step how to build a basic application of just the right type.

He’s happy. He feels really productive. CRUD just “drops out” of the framework’s design. It’s like it was created with his problem in mind.

Then he’s done with the tutorials. He starts plugging in his own, specific requirements. Every so often he hits a snag, but that’s OK, he can just twist the framework’s arm a bit, right?

It all goes well for a while. Then he hits a big one. The framework doesn’t do quite work the way he wants it to.

He posts a question. Maybe he doesn’t get an answer. Maybe he does, and he’s told it’s not within the benevolent dictators’ design parameters.

Crap. Oh well. He can start working around the decisions that don’t match his application. This will add some development overhead. At some point it’ll be enough that he has to consider his options.

That’s OK. He has the source. He can fork it, and do his custom stuff - he knows it well enough by now. He’ll lose the upstream fixes, unless he puts in some real effort into integrating them, but at least he doesn’t have to ditch the codebase.

This will tide his team over until the re-write, which they’ll build upon a different framework. One that looks like it better suits their needs.

Status quo

That’s just the way things are, right?

If you take someone’s code, they have made some decisions for you. Sometimes the decisions work in your favor, sometimes they don’t.

There’s an alluring side to this. The main reason why a framework might feel solid at first is because someone made a lot of decisions for you - one might say such a framework is opinionated. Those decisions seem to make sense, and they get you from zero to CRUD in just a few steps.

If you happen to be building the exact type of application that the framework creators have in mind, if their way of thinking immediately clicks with you, that’ll take you a long way.

But when you try to move away just bit from the set of problems they decided to address, you realize that those choices accrete. On a solidly integrated framework, you don’t get to pick and choose. Pulling away even a single one requires considerable effort.

Most of the time, either you take it or you leave it.

The not-framework approach

Frameworks expect to have an answer to most of your fundamental questions. Can you really rely on them to do that?

Consider an alternate path.

At first glance, Clojure’s Luminus is not as solidly integrated as other alternatives. For every potential component, you get to pick and choose - and never mind PostgreSQL vs. MongoDB vs. Datomic.

  • It defaults to Migratus for database migrations… but you can just pull it out and use any alternative.
  • Ring provides a nice abstraction for HTTP, but which web server do you want? Immutant or HTTP Kit?
  • Do you want ClojureScript support? If so, do you want Reagent or Om? And what’s the difference between those two anyway?

Even once you’ve made a decision, chances are you won’t find any step-by-step zero-to-CRUD tutorials for your specific combination.

So… how does this help again?

Control

No, Luminus is not making any decisions for you. The very few areas where it provides a single choice, like Selmer for templating, are orthogonal to your application’s design.

Unlike the frameworks in fashion on other languages - or even some that are starting to trend for Clojure - it’s not opinionated. One could say it’s open minded.

Here’s what happens when you create a new Luminus application.

  1. You’ll have to tell it which options you want included. A list of the options is available on the site.
  2. Luminus will generate a new template from those options, every option potentially pulling in a few libraries.
  3. It will also include some reasonable defaults and glue code, to save you time.

While it’s happy to show you its conventions with the templates that it generates, you won’t find it making any choices for you in advance. But more importantly, at no point does it try to wrestle control of those decisions from you.

It lets you make up your own mind.

Decision independence

Each library Luminus includes does make some decisions internally, for obvious reasons. But these are mechanical in nature, and given that Luminus doesn’t want its basic components to be strongly fused together, you are free to swap out any particular part.

Think of it as the bevy of functions you have on Clojure. Every one of them provides a piece of independent, re-usable functionality. Yes, the internal decisions of how the functions were implemented are made for you in advance, but it’s up to you to decide how to put them together to build your application, or if even you’d rather use an external library.

Learning curve

Yes, this means there will be a slightly steeper learning curve. You’ll need to know what these choices mean, before you can make an informed decision.

That means that on day-1, you’ll feel less productive than how you would on an opinionated framework, because you don’t get to coast on decisions someone else has made for you.

If you are looking for day-1 productivity, it may not be for you.

I’m not. I’m looking at life time value.

After using this loosely-coupled approach on projects for a couple of years, I’ve realized two things.

I like choosing

I get to replace any single part of the system, if I find it doesn’t suit my needs.

I may not need to. Hopefully I made the right decision up-front. But if for whatever reason I didn’t, I know that no other choice is tightly integrated with it, so I can gradually pull it out and replace it - I don’t need to switch frameworks, or fork it, or live with it.

I get to change my mind.

I no longer dread updates

Up until a few months ago, YeSQL was the standard way to access SQL databases in Luminus. When YeSQL support slowed down, Dmitri Sotnikov decided to replace it with HugSQL, which follows the same approach but takes it further and is actively developed.

Want to guess how many existing Luminus projects it affected?

Zero.

How can I state that with certainty?

Because it couldn’t possibly have affected a single one. Your choices are only for which libraries is your initial template generated from. After that, you are free to decide which of these do you want to update at any point, depending on if it suits you or not.

Any changes to future Luminus choices will have zero impact on you.

Productivity trade-offs

This means initially you’ll be taking baby steps. You may feel like an idiot as you go about figuring things out.

We all hate that. I get it.

It’s part of what makes frameworks so appealing when facing a new platform: they promise you’ll hit the ground running. Nobody likes to crawl when they’re used to just zoom by basic tasks.

But when changing contexts, we have to take baby steps first and then go on from there, because every decision we make is a trade-off.

When choosing a loosely-coupled approach, you’re saying that day-1 productivity is less important than long-term maintenance and total application lifetime productivity.

Personally, I’d rather spend time up-front understanding the choices I’m making. If I don’t have fundamental decisions made for me in advance, I have to look at what the options are, and can decide to change some pieces.

This understanding increases transparency, which in turn enhances my comprehension of the very foundations that I’ll be building upon for months or years.

Given I spend more time on application support and extension than on the first few days, it’s a trade-off I’m happy to make.

Permalink

Alex Bedner proof read my Clojure Book

Hello Clojurians,

I’m happy to announce that the Clojure book has gotten a bit better! A few days ago, I received an email from someone named Alex Bedner. He read my Clojure book from start to finish and was kind enough to share some corrections. I’ve implemented them, and you can find the updated book here: https://clojure-book.gitlab.io/

I did try to upload the print copy to Amazon, but for some reason they require additional verification from my side — and I’m not entirely sure what that entails. I’m not great with bureaucracy, so getting the Amazon listing sorted will take some time. In the meantime, Alex Bedner’s proofread version of the Clojure book is available at the link above. I’m deeply grateful to him.

There’s still a lot to do, and I remain committed to Clojure. Here are the things I’d like to finish in the book:

Creating Your Own Libraries (https://clojure-book.gitlab.io/book.html#_creating_your_own_libraries) — this section currently has no accompanying video. I’ve been meaning to make one, but haven’t gotten around to it yet — I can’t quite explain why.

Debugging with Calva (https://clojure-book.gitlab.io/book.html#_debugging_with_calva) — I need to write this section, with plenty of screenshots so readers know how to debug with Calva. At the very least, I’d like to link to the Calva website, where the creator has put together some excellent videos.

I’d also like to add a section on Paredit at some point — we’ll see how that goes.

Finally, I’d love to use LLMs to help proofread and polish the book further. English isn’t my first language, and I’ve found that AI tools have been genuinely helpful in improving the quality of my writing lately.

Once again, thank you to everyone who has said kind words about the book, pointed out corrections, offered criticism, or praised it. All of it is valuable input, and I’ll make the most of every bit of it.

Permalink

Clojure Deref (May 19, 2026)

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem (feed: RSS).

Clojure Dev Call

The Clojure development team is pleased to announce a Clojure Dev Call on May 26 @ 17:00 UTC!

Join the Clojure dev team for an update on what we’ve been working on and what’s on our horizon. We’ll save time for a Q&A, so bring your questions.

Register here

Clojure/Conj 2026

Join us for the largest gathering of Clojure developers in the world! Meet new people and reconnect with old friends. Enjoy two full days of talks, a day of workshops, social events, and more.

September 30 – October 2, 2026
Charlotte Convention Center, Charlotte, NC

The CFP is open until June 14

Early Bird tickets are on sale now.

EuroClojure 2027

After a decade away, EuroClojure returns to mark twenty years of Clojure. Join us in Prague for three days of workshops, talks, and thoughtful conversation.

May 19-21, 2027
2027.euroclojure.org

Join the mailing list for early bird tickets and announcements. Share your ideas and content suggestions when you sign up.

Upcoming Events

Podcasts, videos, and media

Babashka Conf

Swish by Rod Schmidt

Mike Møller Nielsen

Blogs, articles, and news

Libraries and Tools

Debut release

  • let-go - Almost Clojure written in Go.

  • mino - A tiny, embeddable, REPL-friendly Lisp implemented in pure ANSI C.

  • clj-p4 - Read-only Perforce-to-Git bridge in Clojure, with stream and classic-depot support.

  • gloat-demo-fiber - A JSON REST API built with Fiber and Gloat, demonstrating Go/Clojure interoperability for web applications.

  • zulipdata - A Clojure library to fetch and analyse data from the Clojurians Zulip chat

  • rechentafel - A spreadsheet evaluator for Clojure/Script.

  • re-frame2 - The AI first-born of re-frame

  • ifgame - An interactive fiction game written in Clojure

  • drip - Drip is a transactional job queue for MariaDB, PostgreSQL, and SQLite

  • port - A minimalist Clojure interactive programming environment for Emacs, built on prepl

  • generate - code generation for Clojure projects

  • bara-lang - Clojure for nim language

  • gsheetplus - Low-level and high-level wrapper to work with Google Sheets. Reading, writing and sheet management.

  • winze - An agentic AI memory system (mcp server) with semantic search and GUI search/edit tool for Markdown knowledge bases

  • ghosttyfx - JavaFX terminal that uses libghostty

  • cljfx/ghosttyfx - Cljfx wrapper of GhosttyFX

  • beagle - a typed Lisp authoring surface for agent-written dynamic code.

  • ducktape - Connect tech.v3.dataset to DuckDB

  • phel-log - Data-driven logging library for Phel. Levels, namespace filtering, pluggable appenders, PSR-3 adapter, Monolog handler bridge. Inspired by Timbre + Monolog.

  • phel-symfony-demo - Symfony + Phel demo: Ring-style handlers over plain maps, no ORM

  • stube - Web framework inspired by Seaside and UnCommonWeb

  • clj-native-agent - Structural, semantic, and surgical skills for Clojure AI agents. Mimics expert practitioner workflows for code discovery, debugging, and editing using Babashka and rewrite-clj.

  • llmisp - JSON AST > Clojure

  • spindel - Cross-platform FRP runtime with a git-like memory model.

  • rephrase - Rephrase exceptions to be more beginner-friendly

Updates

  • clojure 1.12.5 - The Clojure programming language

  • clojure_cli 1.12.5.1645 - Clojure CLI

  • clojurescript 1.12.145 - Clojure to JS compiler

  • core.unify 0.7.2 - Unification library

  • clj-watson 6.1.0 - A Clojure tool that checks for vulnerable dependencies

  • aleph 0.9.8 - Asynchronous streaming communication for Clojure - web server, web client, and raw TCP/UDP

  • awesome-backseat-driver 1.0.13 - Plugin marketplace for Clojure AI context in GitHub Copilot: agents, skills, and workflows for REPL-first interactive programming with Calva Backseat Driver

  • logging4j2 1.0.8 - A Clojure wrapper for log4j2

  • core.async.flow-monitor 0.1.5 - A real-time monitoring and interaction tool for clojure.core.async.flow

  • martian-aleph 0.1.4 - Martian plugin to use the Aleph http client

  • thneed 1.1.9 - An eclectic set of Clojure utilities that I’ve found useful enough to keep around.

  • statecharts 1.4.0-RC13 - A Statechart library for CLJ(S)

  • joyride 0.0.75 - Making VS Code Hackable like Emacs since 2022

  • stratum 0.2.67 - Versioned, fast and scalable columnar database.

  • deps-new 0.12.1 - Create new projects for the Clojure CLI / deps.edn

  • bareforge 0.6.0 - Companion visual builder for BareDOM web components. Drag components, declare reactive state, export fully interactive CLJS or JS project

  • pg-datahike 0.1.43 - Postgres compatibility layer for Datahike.

  • de-dupe 0.3.0 - A ClojureScript library which "de-duplicates" Persistent Data Structures so they can be more efficiently serialised.

  • ClojureStorm 1.12.5 - A fork of the official Clojure compiler with extra code to make it a dev compiler

  • graal-build-time 1.0.6 - Initialize Clojure classes at build time with GraalVM native-image

  • tyrell 1.0.0-RC9 - Clojurescript WebComponents library

  • gloat 0.1.41 - Glojure AOT Tool

  • metamorph.ml 1.6.2 - Machine learning functions based on metamorph and machine learning pipelines

  • oksa 1.2.1 - Generate GraphQL queries using Clojure data structures.

  • pomegranate 1.3.27 - A sane Clojure API for Maven Artifact Resolver + dynamic runtime modification of the classpath

  • calva 2.0.588 - Clojure & ClojureScript Interactive Programming for VS Code

  • cli-tools 0.16.0-beta-7 - CLIs and subcommands for Clojure or Babashka

  • fulcro-spec 3.2.9 - A library that wraps clojure.test for a better BDD testing experience.

  • baredom 3.1.0 - BareDOM: Lightweight CLJS UI components built on web standards (Custom Elements, Shadow DOM, ES modules). No framework, just the DOM

  • o11ylite 2026.5.16-1112955 - Free, open-source OpenTelemetry backend powered by DuckDB 🦆 and SQLite

  • calva-backseat-driver 0.0.34 - VS Code AI Agent Interactive Programming. Tools for CoPIlot and other assistants. Can also be used as an MCP server.

  • plumcp 0.2.1 - Clojure/ClojureScript library for making MCP server and client

  • teensyp 0.5.0 - A small, zero-dependency Clojure TCP server that uses Java NIO

  • build-uber-log4j2-handler 2.26.0 - A conflict handler for log4j2 plugins cache files for the tools.build uber task.

  • phel-lang 0.39.0 - A functional, Lisp-inspired language that compiles to PHP. Inspired by Clojure, Phel brings macros, persistent data structures, and expressive functional idioms to the PHP ecosystem.

  • dexter 0.1-beta-2 - Dexter - Graphical Dependency Explorer

  • plotje 0.2.2 - simple and easy plotting

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.