I use Debian Linux as it provides a stable and low maintenance operating system, enabling me to focus on getting valuable tasks done. Package updates are well tested and the deb package management system avoid incompatible versions between dependent packages.
The small amount of maintenance is done via the Advance Packaging Tool (Apt), which has a very simple to understand command line, i.e. apt install, apt update, apt upgrade and apt purge.
The constraint of a stable operating system is that some of the latest versions of development tools and programming languages may not be available as part of the distributions package manager.
{align=right loading=lazy}
Simple bash scripts were created to install the latest development tools, made effectively one-liner's where the [Download Release Artifacts (DRA)](} project could be used. The scripts were very simple even when falling back to curl with a few basic Linux commands.
These shell scripts and the DRA tool are used to install editors, programming languages (e.g. Clojure, Rust, Node.js), Terminal UI (TUI's) tools for development or system administration and even the odd desktop app.
A debian-linux-post-instal.sh script was created update all the Debian packages, reading packages to add and remove from a plain text file.
A dev-tools-install.sh calls each script that installs the latest versions of each development tool, programming language, TUI and desktop app.
Practicalli dotfiles - Debian-Linux contains all the scripts, along with a few manual steps that have not been scripted (mainly UI / UX configuration).
This started as a simple question: what is an agent framework actually doing?
A few months ago I built a minimal agent engine from scratch in Clojure to find out. That engine used a graph model: nodes were pure functions, edges defined routing, a loop drove execution. The short answer was that it is a recursive loop, some JSON parsing, and a state machine. That article is here if you want the details.
But once you strip the agent down to its core, a different question surfaces. If agents can call other agents, how does authority move between them? That led me back to Spritely Goblins, which I already knew. OCapN came from there. When I tried to interpret those ideas in a cloud-native context, I ran into Biscuit tokens. I wrote about reconstructing Biscuit in Clojure. The result was kex, a minimal proof of concept.
Ayatori is where those two threads converge. The graph model from the first experiment, combined with a capability-based design for agent composition and authority.
What Ayatori is
Ayatori is an experimental graph-based AI agent orchestration engine built in Clojure. Nodes are mostly pure functions. Edges define routing. The executor handles the rest.
The name comes from ayatori, the Japanese string figure game known in English as cat’s cradle. Agents passing capabilities to each other, each move shaping what comes next. The attenuation and delegation that constrains those moves will come with kex integration.
It is currently a proof of concept, under active development. Not production-ready.
Thanks for reading Taorem! Subscribe for free to receive new posts and support my work.
The core model
An agent is a graph. You define nodes and edges, declare what capabilities it exposes and what dependencies it needs, and the system wires everything together at start time.
deps -> agent(nodes, edges) -> caps
Nodes can be pure functions, stateful handlers, LLM nodes, fan-out nodes, or nested agent nodes. The simplest case is a pure function: takes input, returns output. Routing is data: edges are keywords for unconditional routing, or maps for conditional routing based on a :route key the node returns. The executor handles dispatch, state, and middleware.
A simple example. An LLM agent with tool calling and structured output:
Tool calls route through graph nodes. Middleware observes every step. :response-format enforces output schema with self-healing retries via :max-retries if the LLM returns invalid output.
Agents expose caps and declare deps. The system wires deps to caps at start time. An agent can use another agent as a node.
The inner agent runs with its own execution scope but shares the same store and trace-id. Caps can carry Malli schemas for input and output validation. rewire! changes dep targets at runtime without restarting the system.
Where kex comes in
Right now, capability security is structural. Possessing a CapHandle is sufficient to invoke it. There is no cryptographic verification.
When kex is integrated, cross-node calls will carry token chains. Each delegation can only narrow the original capability, never expand it. An agent receiving a delegated capability cannot do more than what it was given. This is still on the roadmap.
Ayatori is a local prototype. No distributed execution yet. CapHandles cannot be serialized or sent across the wire. The middleware dispatch is synchronous. Observability covers individual runs but not topology visualization. Time-based timeout is not implemented, though :max-steps (default 100) provides step-based protection against infinite loops.
These are the next problems to solve.
Multi-node execution is a harder problem. When an agent on one node calls an agent on another and expects a result back, the continuation has to survive the network boundary. Supervision, timeout handling, and meaningful error propagation across agent boundaries are in the same category. Things I want to tackle, but will take time.
Why I built this
My day-to-day work has not involved writing production code for a while. I try to stay hands-on anyway. Clojure is what I reach for when I want to think through a problem with code.
When I started exploring agentic systems, I kept building things to understand them. The minimal agent engine was one experiment. Kex was another. Ayatori is where the two ideas merged into something more structured.
It is an experiment, not a product. The code is available on GitHub if you want to read it, run it, or point out what is wrong with it.
I will keep working on it when I have time. Distributed execution and kex integration are the next things I want to tackle. No timeline.
As part of the Clojure team’s efforts to improve the onboarding experience for new users, we have recorded a step-by-step tutorial taking you from zero to a running REPL.
The video begins with installing the JVM and Clojure CLI, then walks through installing Calva, opening an example project, and connecting it to a REPL to evaluate code.
This post walks through a small web development project using Clojure, covering everything from building the app to packaging and deploying it. It’s a collection of insights and tips I’ve learned from building my Clojure side projects, but presented in a more structured format.
As the title suggests, we’ll be deploying the app to Fly.io. It’s a service that allows you to deploy apps packaged as Docker images on lightweight virtual machines.[1][1] My experience with it has been good; it’s easy to use and quick to set up. One downside of Fly is that it doesn’t have a free tier, but if you don’t plan on leaving the app deployed, it barely costs anything.
This isn’t a tutorial on Clojure, so I’ll assume you already have some familiarity with the language as well as some of its libraries.[2][2]
In this post, we’ll be building a barebones bookmarks manager for the demo app. Users can log in using basic authentication, view all bookmarks, and create a new bookmark. It’ll be a traditional multi-page web app and the data will be stored in a SQLite database.
Here’s an overview of the project’s starting directory structure:
And the libraries we’re going to use. If you have some Clojure experience or have used Kit, you’re probably already familiar with all the libraries listed below.[3][3]
I use Aero and Integrant for my system configuration (more on this in the next section), Ring with the Jetty adaptor for the web server, Reitit for routing, next.jdbc for database interaction, and Hiccup for rendering HTML. From what I’ve seen, this is a popular “library combination” for building web apps in Clojure.[4][4]
The user namespace in dev/user.clj contains helper functions from Integrant-repl to start, stop, and restart the Integrant system.
dev/user.clj
(ns user
(:require[acme.main :as main][clojure.tools.namespace.repl :as repl][integrant.core :as ig][integrant.repl :refer[set-prep! go halt reset reset-all]]))(set-prep!(fn[](ig/expand(main/read-config))));; we'll implement this soon(repl/set-refresh-dirs"src""resources")(comment(go)(halt)(reset)(reset-all))
If you’re new to Integrant or other dependency injection libraries like Component, I’d suggest reading “How to Structure a Clojure Web”. It’s a great explanation of the reasoning behind these libraries. Like most Clojure apps that use Aero and Integrant, my system configuration lives in a .edn file. I usually name mine as resources/config.edn. Here’s what it looks like:
In production, most of these values will be set using environment variables. During local development, the app will use the hard-coded default values. We don’t have any sensitive values in our config (e.g., API keys), so it’s fine to commit this file to version control. If there are such values, I usually put them in another file that’s not tracked by version control and include them in the config file using Aero’s #include reader tag.
This config file is then “expanded” into the Integrant system map using the expand-key method:
The system map is created in code instead of being in the configuration file. This makes refactoring your system simpler as you only need to change this method while leaving the config file (mostly) untouched.[5][5]
My current approach to Integrant + Aero config files is mostly inspired by the blog post “Rethinking Config with Aero & Integrant” and Laravel’s configuration. The config file follows a similar structure to Laravel’s config files and contains the app configurations without describing the structure of the system. Previously, I had a key for each Integrant component, which led to the config file being littered with #ig/ref and more difficult to refactor.
Also, if you haven’t already, start a REPL and connect to it from your editor. Run clj -M:dev if your editor doesn’t automatically start a REPL. Next, we’ll implement the init-key and halt-key! methods for each of the components:
The setup-db function creates the required tables in the database if they don’t exist yet. This works fine for database migrations in small projects like this demo app, but for larger projects, consider using libraries such as Migratus (my preferred library) or Ragtime.
src/acme/util.clj
(ns acme.util
(:require[next.jdbc :as jdbc]))(defn setup-db
[db](jdbc/execute-one!
db
["create table if not exists bookmarks (
bookmark_id text primary key not null,
url text not null,
created_at datetime default (unixepoch()) not null
)"]))
For the server handler, let’s start with a simple function that returns a “hi world” string.
Now all the components are implemented. We can check if the system is working properly by evaluating (reset) in the user namespace. This will reload your files and restart the system. You should see this message printed in your REPL:
:reloading (acme.util acme.handler acme.main)
Server started on port 8080
:resumed
If we send a request to http://localhost:8080/, we should get “hi world” as the response:
$ curl localhost:8080/
# hi world
Nice! The system is working correctly. In the next section, we’ll implement routing and our business logic handlers.
If you remember the :handler/ring from earlier, you’ll notice that it has two dependencies, database and auth. Currently, they’re inaccessible to our route handlers. To fix this, we can inject these components into the Ring request map using a middleware function.
The components-middleware function takes in a map of components and creates a middleware function that “assocs” each component into the request map.[6][6] If you have more components such as a Redis cache or a mail service, you can add them here.
We’ll also need a middleware to handle HTTP basic authentication.[7][7] This middleware will check if the username and password from the request map match the values in the auth map injected by components-middleware. If they match, then the request is authenticated and the user can view the site.
A nice feature of Clojure is that interop with the host language is easy. The base64-encode function is just a thin wrapper over Java’s Base64.Encoder:
We now have everything we need to implement the route handlers or the business logic of the app. First, we’ll implement the index-page function, which renders a page that:
Shows all of the user’s bookmarks in the database, and
Shows a form that allows the user to insert new bookmarks into the database
src/acme/handler.clj
(ns acme.handler
(:require;; ...[next.jdbc :as jdbc][next.jdbc.sql :as sql]));; ...(defn template
[bookmarks][:html[:head[:meta{:charset"utf-8":name"viewport":content"width=device-width, initial-scale=1.0"}]][:body[:h1"bookmarks"][:form{:method"POST"}[:div[:label{:for"url"}"url "][:input#url {:name"url":type"url":requiredtrue:placeholer"https://en.wikipedia.org/"}]][:button"submit"]][:p"your bookmarks:"][:ul(if(empty? bookmarks)[:li"you don't have any bookmarks"](map(fn[{:keys[url]}][:li[:a{:href url} url]])
bookmarks))]]])(defn index-page
[req](try(let[bookmarks (sql/query(:db req)["select * from bookmarks"]
jdbc/unqualified-snake-kebab-opts)](util/render(template bookmarks)))(catch Exception e
(util/server-error e))));; ...
Database queries can sometimes throw exceptions, so it’s good to wrap them in a try-catch block. I’ll also introduce some helper functions:
render takes a hiccup form and turns it into a ring response, while server-error takes an exception, logs it, and returns a 500 response.
Next, we’ll implement the index-action function:
src/acme/handler.clj
;; ...(defn index-action
[req](try(let[{:keys[db form-params]} req
value (get form-params "url")](sql/insert! db :bookmarks{:bookmark_id(random-uuid):url value})(res/redirect"/"303))(catch Exception e
(util/server-error e))));; ...
This is an implementation of a typical post/redirect/get pattern. We get the value from the URL form field, insert a new row in the database with that value, and redirect back to the index page. Again, we’re using a try-catch block to handle possible exceptions from the database query.
That should be all of the code for the controllers. If you reload your REPL and go to http://localhost:8080, you should see something that looks like this after logging in:
The last thing we need to do is to update the main function to start the system:
Now, you should be able to run the app using clj -M -m acme.main. That’s all the code needed for the app. In the next section, we’ll package the app into a Docker image to deploy to Fly.
While there are many ways to package a Clojure app, Fly.io specifically requires a Docker image. There are two approaches to doing this:
Build an uberjar and run it using Java in the container, or
Load the source code and run it using Clojure in the container
Both are valid approaches. I prefer the first since its only dependency is the JVM. We’ll use the tools.build library to build the uberjar. Check out the official guide for more information on building Clojure programs. Since it’s a library, to use it, we can add it to our deps.edn file with an alias:
Tools.build expects a build.clj file in the root of the project directory, so we’ll need to create that file. This file contains the instructions to build artefacts, which in our case is a single uberjar. There are many great examples of build.clj files on the web, including from the official documentation. For now, you can copy+paste this file into your project.
To build the project, run clj -T:build uber. This will create the uberjar standalone.jar in the target directory. The uber in clj -T:build uber refers to the uber function from build.clj. Since the build system is a Clojure program, you can customise it however you like. If we try to run the uberjar now, we’ll get an error:
# build the uberjar$ clj -T:build uber
# Cleaning build directory...# Copying files...# Compiling Clojure...# Building Uberjar...# run the uberjar$ java-jar target/standalone.jar
# Error: Could not find or load main class acme.main# Caused by: java.lang.ClassNotFoundException: acme.main
This error occurred because the Main class that is required by Java isn’t built. To fix this, we need to add the :gen-class directive in our main namespace. This will instruct Clojure to create the Main class from the -main function.
src/acme/main.clj
(ns acme.main
;; ...(:gen-class));; ...
If you rebuild the project and run java -jar target/standalone.jar again, it should work perfectly. Now that we have a working build script, we can write the Dockerfile:
Dockerfile
# install additional dependencies here in the base layer# separate base from build layer so any additional deps installed are cachedFROM clojure:temurin-21-tools-deps-bookworm-slim AS baseFROM base as buildWORKDIR /optCOPY . .RUN clj -T:build uberFROM eclipse-temurin:21-alpine AS prodCOPY--from=build /opt/target/standalone.jar /EXPOSE 8080ENTRYPOINT ["java", "-jar", "standalone.jar"]
It’s a multi-stage Dockerfile. We use the official Clojure Docker image as the layer to build the uberjar. Once it’s built, we copy it to a smaller Docker image that only contains the Java runtime.[8][8] By doing this, we get a smaller container image as well as a faster Docker build time because the layers are better cached.
That should be all for packaging the app. We can move on to the deployment now.
First things first, you’ll need to install flyctl, Fly’s CLI tool for interacting with their platform. Create a Fly.io account if you haven’t already. Then run fly auth login to authenticate flyctl with your account.
$ fly app create
# ? Choose an app name (leave blank to generate one): # automatically selected personal organization: Ryan Martin# New app created: blue-water-6489
Another way to do this is with the fly launch command, which automates a lot of the app configuration for you. We have some steps to do that are not done by fly launch, so we’ll be configuring the app manually. I also already have a fly.toml file ready that you can straight away copy to your project.
fly.toml
# replace these with your app and region name# run `fly platform regions` to get a list of regionsapp='blue-water-6489'primary_region='sin'[env]DB_DATABASE="/data/database.db"[http_service]internal_port=8080force_https=trueauto_stop_machines="stop"auto_start_machines=truemin_machines_running=0[mounts]source="data"destination="/data"initial_sie=1[[vm]]size="shared-cpu-1x"memory="512mb"cpus=1cpu_kind="shared"
These are mostly the default configuration values with some additions. Under the [env] section, we’re setting the SQLite database location to /data/database.db. The database.db file itself will be stored in a persistent Fly Volume mounted on the /data directory. This is specified under the [mounts] section. Fly Volumes are similar to regular Docker volumes but are designed for Fly’s micro VMs.
We’ll need to set the AUTH_USER and AUTH_PASSWORD environment variables too, but not through the fly.toml file as these are sensitive values. To securely set these credentials with Fly, we can set them as app secrets. They’re stored encrypted and will be automatically injected into the app at boot time.
$ fly secrets setAUTH_USER=hi@ryanmartin.me AUTH_PASSWORD=not-so-secure-password
# Secrets are staged for the first deployment
With this, the configuration is done and we can deploy the app using fly deploy:
$ fly deploy
# ...# Checking DNS configuration for blue-water-6489.fly.dev# Visit your newly deployed app at https://blue-water-6489.fly.dev/
The first deployment will take longer since it’s building the Docker image for the first time. Subsequent deployments should be faster due to the cached image layers. You can click on the link to view the deployed app, or you can also run fly open, which will do the same thing. Here’s the app in action:
If you made additional changes to the app or fly.toml, you can redeploy the app using the same command, fly deploy. The app is configured to auto stop/start, which helps to cut costs when there’s not a lot of traffic to the site. If you want to take down the deployment, you’ll need to delete the app itself using fly app destroy <your app name>.
This is an interesting topic in the Clojure community, with varying opinions on whether or not it’s a good idea. Personally, I find having a REPL connected to the live app helpful, and I often use it for debugging and running queries on the live database.[9][9] Since we’re using SQLite, we don’t have a database server we can directly connect to, unlike Postgres or MySQL.
If you’re brave, you can even restart the app directly without redeploying from the REPL. You can easily go wrong with it, which is why some prefer not to use it.
For this project, we’re gonna add a socket REPL. It’s very simple to add (you just need to add a JVM option) and it doesn’t require additional dependencies like nREPL. Let’s update the Dockerfile:
The socket REPL will be listening on port 7888. If we redeploy the app now, the REPL will be started, but we won’t be able to connect to it. That’s because we haven’t exposed the service through Fly proxy. We can do this by adding the socket REPL as a service in the [services] section in fly.toml.
However, doing this will also expose the REPL port to the public. This means that anyone can connect to your REPL and possibly mess with your app. Instead, what we want to do is to configure the socket REPL as a private service.
By default, all Fly apps in your organisation live in the same private network. This private network, called 6PN, connects the apps in your organisation through WireGuard tunnels (a VPN) using IPv6. Fly private services aren’t exposed to the public internet but can be reached from this private network. We can then use Wireguard to connect to this private network to reach our socket REPL.
Fly VMs are also configured with the hostname fly-local-6pn, which maps to its 6PN address. This is analogous to localhost, which points to your loopback address 127.0.0.1. To expose a service to 6PN, all we have to do is bind or serve it to fly-local-6pn instead of the usual 0.0.0.0. We have to update the socket REPL options to:
After redeploying, we can use the fly proxy command to forward the port from the remote server to our local machine.[10][10]
$ fly proxy 7888:7888
# Proxying local port 7888 to remote [blue-water-6489.internal]:7888
In another shell, run:
$ rlwrap nc localhost 7888# user=>
Now we have a REPL connected to the production app! rlwrap is used for readline functionality, e.g. up/down arrow keys, vi bindings. Of course, you can also connect to it from your editor.
To get this to work, you’ll need to create a deploy token from your app’s dashboard. Then, in your GitHub repo, create a new repository secret called FLY_API_TOKEN with the value of your deploy token. Now, whenever you push to the main branch, this workflow will automatically run and deploy your app. You can also manually run the workflow from GitHub because of the workflow_dispatch option.
As always, all the code is available on GitHub. Originally, this post was just about deploying to Fly.io, but along the way, I kept adding on more stuff until it essentially became my version of the user manager example app. Anyway, hope this post provided a good view into web development with Clojure. As a bonus, here are some additional resources on deploying Clojure apps:
The way Fly.io works under the hood is pretty clever. Instead of running the container image with a runtime like Docker, the image is unpacked and “loaded” into a VM. See this video explanation for more details. ↩︎
Kit was a big influence on me when I first started learning web development in Clojure. I never used it directly, but I did use their library choices and project structure as a base for my own projects. ↩︎
There’s no “Rails” for the Clojure ecosystem (yet?). The prevailing opinion is to build your own “framework” by composing different libraries together. Most of these libraries are stable and are already used in production by big companies, so don’t let this discourage you from doing web development in Clojure! ↩︎
There might be some keys that you add or remove, but the structure of the config file stays the same. ↩︎
“assoc” (associate) is a Clojure slang that means to add or update a key-value pair in a map. ↩︎
For more details on how basic authentication works, check out the specification. ↩︎
Here’s a cool resource I found when researching Java Dockerfiles: WhichJDK. It provides a comprehensive comparison of the different JDKs available and recommendations on which one you should use. ↩︎
If you encounter errors related to WireGuard when running fly proxy, you can run fly doctor, which will hopefully detect issues with your local setup and also suggest fixes for them. ↩︎
This post is about six seven months late, but here are my takeaways from Advent of Code 2024. It was my second time participating, and this time I actually managed to complete it.[1][1] My goal was to learn a new language, Zig, and to improve my DSA and problem-solving skills.
If you’re not familiar, Advent of Code is an annual programming challenge that runs every December. A new puzzle is released each day from December 1st to the 25th. There’s also a global leaderboard where people (and AI) race to get the fastest solves, but I personally don’t compete in it, mostly because I want to do it at my own pace.
I went with Zig because I have been curious about it for a while, mainly because of its promise of being a better C and because TigerBeetle (one of the coolest databases now) is written in it. Learning Zig felt like a good way to get back into systems programming, something I’ve been wanting to do after a couple of chaotic years of web development.
This post is mostly about my setup, results, and the things I learned from solving the puzzles. If you’re more interested in my solutions, I’ve also uploaded my code and solution write-ups to my GitHub repository.
There were several Advent of Code templates in Zig that I looked at as a reference for my development setup, but none of them really clicked with me. I ended up just running my solutions directly using zig run for the whole event. It wasn’t until after the event ended that I properly learned Zig’s build system and reorganised my project.
The project is powered by build.zig, which defines several commands:
Build
zig build - Builds all of the binaries for all optimisation modes.
Run
zig build run - Runs all solutions sequentially.
zig build run -Day=XX - Runs the solution of the specified day only.
Benchmark
zig build bench - Runs all benchmarks sequentially.
zig build bench -Day=XX - Runs the benchmark of the specified day only.
Test
zig build test - Runs all tests sequentially.
zig build test -Day=XX - Runs the tests of the specified day only.
You can also pass the optimisation mode that you want to any of the commands above with the -Doptimize flag.
Under the hood, build.zig compiles src/run.zig when you call zig build run, and src/bench.zig when you call zig build bench. These files are templates that import the solution for a specific day from src/days/dayXX.zig. For example, here’s what src/run.zig looks like:
The day module imported is an anonymous import dynamically injected by build.zig during compilation. This allows a single run.zig or bench.zig to be reused for all solutions. This avoids repeating boilerplate code in the solution files. Here’s a simplified version of my build.zig file that shows how this works:
build.zig
const std =@import("std");pubfnbuild(b:*std.Build)void{const target = b.standardTargetOptions(.{});const optimize = b.standardOptimizeOption(.{});const run_all = b.step("run","Run all days");const day_option = b.option(usize,"ay","");// The `-Day` option// Generate build targets for all 25 days.for(1..26)|day|{const day_zig_file = b.path(b.fmt("src/days/day{d:0>2}.zig",.{day}));// Create an executable for running this specific day.const run_exe = b.addExecutable(.{.name = b.fmt("run-day{d:0>2}",.{day}),.root_source_file = b.path("src/run.zig"),.target = target,.optimize = optimize,});// Inject the day-specific solution file as the anonymous module `day`.
run_exe.root_module.addAnonymousImport("day",.{.root_source_file = day_zig_file });// Install the executable so it can be run.
b.installArtifact(run_exe);// ...}}
My actual build.zig has some extra code that builds the binaries for all optimisation modes.
This setup is pretty barebones. I’ve seen other templates do cool things like scaffold files, download puzzle inputs, and even submit answers automatically. Since I wrote my build.zig after the event ended, I didn’t get to use it while solving the puzzles. I might add these features to it if I decided to do Advent of Code again this year with Zig.
While there are no rules to Advent of Code itself, to make things a little more interesting, I set a few constraints and rules for myself:
The code must be readable.
By “readable”, I mean the code should be straightforward and easy to follow. No unnecessary abstractions. I should be able to come back to the code months later and still understand (most of) it.
Solutions must be a single file.
No external dependencies. No shared utilities module. Everything needed to solve the puzzle should be visible in that one solution file.
The total runtime must be under one second.[2][2]
All solutions, when run sequentially, should finish in under one second. I want to improve my performance engineering skills.
Parts should be solved separately.
This means: (1) no solving both parts simultaneously, and (2) no doing extra work in part one that makes part two faster. The aim of this is to get a clear idea of how long each part takes on its own.
No concurrency or parallelism.
Solutions must run sequentially on a single thread. This keeps the focus on the efficiency of the algorithm. I can’t speed up slow solutions by using multiple CPU cores.
No ChatGPT. No Claude. No AI help.
I want to train myself, not the LLM. I can look at other people’s solutions, but only after I have given my best effort at solving the problem.
Follow the constraints of the input file.
The solution doesn’t have to work for all possible scenarios, but it should work for all valid inputs. If the input file only contains 8-bit unsigned integers, the solution doesn’t have to handle larger integer types.
Hardcoding is allowed.
For example: size of the input, number of rows and columns, etc. Since the input is known at compile-time, we can skip runtime parsing and just embed it into the program using Zig’s @embedFile.
Most of these constraints are designed to push me to write clearer, more performant code. I also wanted my code to look like it was taken straight from TigerBeetle’s codebase (minus the assertions).[3][3] Lastly, I just thought it would make the experience more fun.
From all of the puzzles, here are my top 3 favourites:
Day 6: Guard Gallivant - This is my slowest day (in benchmarks), but also the one I learned the most from. Some of these learnings include: using vectors to represent directions, padding 2D grids, metadata packing, system endianness, etc.
Day 17: Chronospatial Computer - I love reverse engineering puzzles. I used to do a lot of these in CTFs during my university days. The best thing I learned from this day is the realisation that we can use different integer bases to optimise data representation. This helped improve my runtimes in the later days 22 and 23.
Day 21: Keypad Conundrum - This one was fun. My gut told me that it can be solved greedily by always choosing the best move. It was right. Though I did have to scroll Reddit for a bit to figure out the step I was missing, which was that you have to visit the farthest keypads first. This is also my longest solution file (almost 400 lines) because I hardcoded the best-moves table.
Honourable mention:
Day 24: Crossed Wires - Another reverse engineering puzzle. Confession: I didn’t solve this myself during the event. After 23 brutal days, my brain was too tired, so I copied a random Python solution from Reddit. When I retried it later, it turned out to be pretty fun. I still couldn’t find a solution I was satisfied with though.
During the event, I learned a lot about Zig and performance, and also developed some personal coding conventions. Some of these are Zig-specific, but most are universal and can be applied across languages. This section covers general programming and Zig patterns I found useful. The next section will focus on performance-related tips.
Zig’s flagship feature, comptime, is surprisingly useful. I knew Zig uses it for generics and that people do clever metaprogramming with it, but I didn’t expect to be using it so often myself.
My main use for comptime was to generate puzzle-specific types. All my solution files follow the same structure, with a DayXX function that takes some parameters (usually the input length) and returns a puzzle-specific type, e.g.:
This lets me instantiate the type with a size that matches my input:
src/days/day01.zig
// Here, `Day01` is called with the size of my actual input.pubfnrun(_:std.mem.Allocator, is_run:bool)![3]u64{// ...const input =@embedFile("./data/day01.txt");var puzzle =tryDay01(1000).init(input);// ...}// Here, `Day01` is called with the size of my test input.test"day 01 part 1 sample 1"{var puzzle =tryDay01(6).init(sample_input);// ...}
This allows me to reuse logic across different inputs while still hardcoding the array sizes. Without comptime, I have to either create a separate function for all my different inputs or dynamically allocate memory because I can’t hardcode the array size.
I also used comptime to shift some computation to compile-time to reduce runtime overhead. For example, on day 4, I needed a function to check whether a string matches either "XMAS" or its reverse, "SAMX". A pretty simple function that you can write as a one-liner in Python:
example.py
defmatches(pattern, target):return target == pattern or target == pattern[::-1]
Typically, a function like this requires some dynamic allocation to create the reversed string, since the length of the string is only known at runtime.[4][4] For this puzzle, since the words to reverse are known at compile-time, we can do something like this:
This creates a separate function for each word I want to reverse.[5][5] Each function has an array with the same size as the word to reverse. This removes the need for dynamic allocation and makes the code run faster. As a bonus, Zig also warns you when this word isn’t compile-time known, so you get an immediate error if you pass in a runtime value.
A common pattern in C is to return special sentinel values to denote missing values or errors, e.g. -1, 0, or NULL. In fact, I did this on day 13 of the challenge:
src/days/day13.zig
// We won't ever get 0 as a result, so we use it as a sentinel error value.fncount_tokens(a:[2]u8, b:[2]u8, p:[2]i64)u64{const numerator =@abs(p[0]* b[1]- p[1]* b[0]);const denumerator =@abs(@as(i32, a[0])* b[1]-@as(i32, a[1])* b[0]);returnif(numerator % denumerator !=0)0else numerator / denumerator;}// Then in the caller, skip if the return value is 0.if(count_tokens(a, b, p)==0)continue;
This works, but it’s easy to forget to check for those values, or worse, to accidentally treat them as valid results. Zig improves on this with optional types. If a function might not return a value, you can return ?T instead of T. This also forces the caller to handle the null case. Unlike C, null isn’t a pointer but a more general concept. Zig treats null as the absence of a value for any type, just like Rust’s Option<T>.
The count_tokens function can be refactored to:
src/days/day13.zig
// Return null instead if there's no valid result.fncount_tokens(a:[2]u8, b:[2]u8, p:[2]i64)?u64{const numerator =@abs(p[0]* b[1]- p[1]* b[0]);const denumerator =@abs(@as(i32, a[0])* b[1]-@as(i32, a[1])* b[0]);returnif(numerator % denumerator !=0)nullelse numerator / denumerator;}// The caller is now forced to handle the null case.if(count_tokens(a, b, p))|n_tokens|{// logic only runs when n_tokens is not null.}
Zig also has a concept of error unions, where a function can return either a value or an error. In Rust, this is Result<T>. You could also use error unions instead of optionals for count_tokens; Zig doesn’t force a single approach. I come from Clojure, where returning nil for an error or missing value is common.
This year has a lot of 2D grid puzzles (arguably too many). A common feature of grid-based algorithms is the out-of-bounds check. Here’s what it usually looks like:
example.zig
fndfs(map:[][]u8, position:[2]i8)u32{const x,const y = position;// Bounds check here.if(x <0or y <0or x >= map.len or y >= map[0].len)return0;if(map[x][y]==.visited)return0;
map[x][y]=.visited;var result:u32=1;for(directions)| direction|{
result +=dfs(map, position + direction);}return result;}
This is a typical recursive DFS function. After doing a lot of this, I discovered a nice trick that not only improves code readability, but also its performance. The trick here is to pad the grid with sentinel characters that mark out-of-bounds areas, i.e. add a border to the grid.
You can use any value for the border, as long as it doesn’t conflict with valid values in the grid. With the border in place, the bounds check becomes a simple equality comparison:
example.zig
const border ='*';fndfs(map:[][]u8, position:[2]i8)u32{const x,const y = position;if(map[x][y]== border){// We are out of boundsreturn0;}// ...}
This is much more readable than the previous code. Plus, it’s also faster since we’re only doing one equality check instead of four range checks.
That said, this isn’t a one-size-fits-all solution. This only works for algorithms that traverse the grid one step at a time. If your logic jumps multiple tiles, it can still go out of bounds (except if you increase the width of the border to account for this). This approach also uses a bit more memory than the regular approach as you have to store more characters.
This could also go in the performance section, but I’m including it here because the biggest benefit I get from using SIMD in Zig is the improved code readability. Because Zig has first-class support for vector types, you can write elegant and readable code that also happens to be faster.
If you’re not familiar with vectors, they are a special collection type used for Single instruction, multiple data (SIMD) operations. SIMD allows you to perform computation on multiple values in parallel using only a single CPU instruction, which often leads to some performance boosts.[6][6]
I mostly use vectors to represent positions and directions, e.g. for traversing a grid. Instead of writing code like this:
You can represent position and direction as 2-element vectors and write code like this:
example.zig
next_position = position + direction;
This is much nicer than the previous version!
Day 25 is another good example of a problem that can be solved elegantly using vectors:
src/days/day25.zig
var result:u64=0;for(self.locks.items)|lock|{// lock is a vectorfor(self.keys.items)|key|{// key is also a vectorconst fitted = lock + key >@as(@Vector(5,u8),@splat(5));const is_overlap =@reduce(.Or, fitted);
result +=@intFromBool(!is_overlap);}}
Expressing the logic as vector operations makes the code cleaner since you don’t have to write loops and conditionals as you typically would in a traditional approach.
The tips below are general performance techniques that often help, but like most things in software engineering, “it depends”. These might work 80% of the time, but performance is often highly context-specific. You should benchmark your code instead of blindly following what other people say.
This section would’ve been more fun with concrete examples, step-by-step optimisations, and benchmarks, but that would’ve made the post way too long. Hopefully, I’ll get to write something like that in the future.[7][7]
Whenever possible, prefer static allocation. Static allocation is cheaper since it just involves moving the stack pointer vs dynamic allocation which has more overhead from the allocator machinery. That said, it’s not always the right choice since it has some limitations, e.g. stack size is limited, memory size must be compile-time known, its lifetime is tied to the current stack frame, etc.
If you need to do dynamic allocations, try to reduce the number of times you call the allocator. The number of allocations you do matters more than the amount of memory you allocate. More allocations mean more bookkeeping, synchronisation, and sometimes syscalls.
A simple but effective way to reduce allocations is to reuse buffers, whether they’re statically or dynamically allocated. Here’s an example from day 10. For each trail head, we want to create a set of trail ends reachable from it. The naive approach is to allocate a new set every iteration:
src/days/day10.zig
for(self.trail_heads.items)|trail_head|{var trail_ends = std.AutoHashMap([2]u8,void).init(self.allocator);defer trail_ends.deinit();// Set building logic...}
What you can do instead is to allocate the set once before the loop. Then, each iteration, you reuse the set by emptying it without freeing the memory. For Zig’s std.AutoHashMap, this can be done using the clearRetainingCapacity method:
src/days/day10.zig
var trail_ends = std.AutoHashMap([2]u8,void).init(self.allocator);defer trail_ends.deinit();for(self.trail_heads.items)|trail_head|{
trail_ends.clearRetainingCapacity();// Set building logic...}
If you use static arrays, you can also just overwrite existing data instead of clearing it.
A step up from this is to reuse multiple buffers. The simplest form of this is to reuse two buffers, i.e. double buffering. Here’s an example from day 11:
src/days/day11.zig
// Initialise two hash maps that we'll alternate between.var frequencies:[2]std.AutoHashMap(u64,u64)=undefined;for(0..2)|i| frequencies[i]= std.AutoHashMap(u64,u64).init(self.allocator);deferfor(0..2)|i| frequencies[i].deinit();var id:usize=0;for(self.stones)|stone|try frequencies[id].put(stone,1);for(0..n_blinks)|_|{var old_frequencies =&frequencies[id %2];var new_frequencies =&frequencies[(id +1)%2];
id +=1;defer old_frequencies.clearRetainingCapacity();// Do stuff with both maps...}
Here we have two maps to count the frequencies of stones across iterations. Each iteration will build up new_frequencies with the values from old_frequencies. Doing this reduces the number of allocations to just 2 (the number of buffers). The tradeoff here is that it makes the code slightly more complex.
A performance tip people say is to have “mechanical sympathy”. Understand how your code is processed by your computer. An example of this is to structure your data so it works better with your CPU. For example, keep related data close in memory to take advantage of cache locality.
Reducing the size of your data helps with this. Smaller data means more of it can fit in cache. One way to shrink your data is through bit packing. This depends heavily on your specific data, so you’ll need to use your judgement to tell whether this would work for you. I’ll just share some examples that worked for me.
The first example is in day 6 part two, where you have to detect a loop, which happens when you revisit a tile from the same direction as before. To track this, you could use a map or a set to store the tiles and visited directions. A more efficient option is to store this direction metadata in the tile itself.
There are only four tile types, which means you only need two bits to represent the tile types as an enum. If the enum size is one byte, here’s what the tiles look like in memory:
As you can see, the upper six bits are unused. We can store the direction metadata in the upper four bits. One bit for each direction. If a bit is set, it means that we’ve already visited the tile in this direction. Here’s an illustration of the memory layout:
direction metadata tile type
┌─────┴─────┐ ┌─────┴─────┐
┌────────┬─┴─┬───┬───┬─┴─┬─┴─┬───┬───┬─┴─┐
│ Tile: │ 1 │ 0 │ 0 │ 0 │ 0 │ 0 │ 1 │ 0 │
└────────┴─┬─┴─┬─┴─┬─┴─┬─┴───┴───┴───┴───┘
up bit ─┘ │ │ └─ left bit
right bit ─┘ down bit
If your language supports struct packing, you can express this layout directly:[8][8]
Doing this avoids extra allocations and improves cache locality. Since the directions metadata is colocated with the tile type, all of them can fit together in cache. Accessing the directions just requires some bitwise operations instead of having to fetch them from another region of memory.
Another way to do this is to represent your data using alternate number bases. Here’s an example from day 23. Computers are represented as two-character strings made up of only lowercase letters, e.g. "bc", "xy", etc. Instead of storing this as a [2]u8 array, you can convert it into a base-26 number and store it as a u16.[9][9]
Here’s the idea: map 'a' to 0, 'b' to 1, up to 'z' as 25. Each character in the string becomes a digit in the base-26 number. For example, "bc" ( [2]u8{ 'b', 'c' }) becomes the base-10 number 28 (). If we represent this using the base-64 character set, it becomes 12 ('b' = 1, 'c' = 2).
While they take the same amount of space (2 bytes), a u16 has some benefits over a [2]u8:
It fits in a single register, whereas you need two for the array.
Comparison is faster as there is only a single value to compare.
I won’t explain branchless programming here; Algorithmica explains it way better than I can. While modern compilers are often smart enough to compile away branches, they don’t catch everything. I still recommend writing branchless code whenever it makes sense. It also has the added benefit of reducing the number of codepaths in your program.
Again, since performance is very context-dependent, I’ll just show you some patterns I use. Here’s one that comes up often:
src/days/day02.zig
if(is_valid_report(report)){
result +=1;}
Instead of the branch, cast the bool into an integer directly:
src/days/day02.zig
result +=@intFromBool(is_valid_report(report))
Another example is from day 6 (again!). Recall that to know if a tile has been visited from a certain direction, we have to check its direction bit. Here’s one way to do it:
The final performance tip is to prefer iterative code over recursion. Recursive functions bring the overhead of allocating stack frames. While recursive code is more elegant, it’s also often slower unless your language’s compiler can optimise it away, e.g. via tail-call optimisation. As far as I know, Zig doesn’t have this, though I might be wrong.
Recursion also has the risk of causing a stack overflow if the execution isn’t bounded. This is why code that is mission- or safety-critical avoids recursion entirely. It’s in TigerBeetle’s TIGERSTYLE and also NASA’s Power of Ten.
Iterative code can be harder to write in some cases, e.g. DFS maps naturally to recursion, but most of the time it is significantly faster, more predictable, and safer than the recursive alternative.
I ran benchmarks for all 25 solutions in each of Zig’s optimisation modes. You can find the full results and the benchmark script in my GitHub repository. All benchmarks were done on an Apple M3 Pro.
As expected, ReleaseFast produced the best result with a total runtime of 85.1 ms. I’m quite happy with this, considering the two constraints that limited the number of optimisations I can do to the code:
Parts should be solved separately - Some days can be solved in a single go, e.g. day 10 and day 13, which could’ve saved a few milliseconds.
No concurrency or parallelism - My slowest days are the compute-heavy days that are very easily parallelisable, e.g. day 6, day 19, and day 22. Without this constraint, I can probably reach sub-20 milliseconds total(?), but that’s for another time.
You can see the full benchmarks for ReleaseFast in the table below:
Day
Title
Parsing (µs)
Part 1 (µs)
Part 2 (µs)
Total (µs)
1
Historian Hysteria
23.5
15.5
2.8
41.8
2
Red-Nosed Reports
42.9
0.0
11.5
54.4
3
Mull it Over
0.0
7.2
16.0
23.2
4
Ceres Search
5.9
0.0
0.0
5.9
5
Print Queue
22.3
0.0
4.6
26.9
6
Guard Gallivant
14.0
25.2
24,331.5
24,370.7
7
Bridge Repair
72.6
321.4
9,620.7
10,014.7
8
Resonant Collinearity
2.7
3.3
13.4
19.4
9
Disk Fragmenter
0.8
12.9
137.9
151.7
10
Hoof It
2.2
29.9
27.8
59.9
11
Plutonian Pebbles
0.1
43.8
2,115.2
2,159.1
12
Garden Groups
6.8
164.4
249.0
420.3
13
Claw Contraption
14.7
0.0
0.0
14.7
14
Restroom Redoubt
13.7
0.0
0.0
13.7
15
Warehouse Woes
14.6
228.5
458.3
701.5
16
Reindeer Maze
12.6
2,480.8
9,010.7
11,504.1
17
Chronospatial Computer
0.1
0.2
44.5
44.8
18
RAM Run
35.6
15.8
33.8
85.2
19
Linen Layout
10.7
11,890.8
11,908.7
23,810.2
20
Race Condition
48.7
54.5
54.2
157.4
21
Keypad Conundrum
0.0
1.7
22.4
24.2
22
Monkey Market
20.7
0.0
11,227.7
11,248.4
23
LAN Party
13.6
22.0
2.5
38.2
24
Crossed Wires
5.0
41.3
14.3
60.7
25
Code Chronicle
24.9
0.0
0.0
24.9
A weird thing I found when benchmarking is that for day 6 part two, ReleaseSafe actually ran faster than ReleaseFast (13,189.0 µs vs 24,370.7 µs). Their outputs are the same, but for some reason, ReleaseSafe is faster even with the safety checks still intact.
The Zig compiler is still very much a moving target, so I don’t want to dig too deep into this, as I’m guessing this might be a bug in the compiler. This weird behaviour might just disappear after a few compiler version updates.
Looking back, I’m really glad I decided to do Advent of Code and followed through to the end. I learned a lot of things. Some are useful in my professional work, some are more like random bits of trivia. Going with Zig was a good choice too. The language is small, simple, and gets out of your way. I learned more about algorithms and concepts than the language itself.
Besides what I’ve already mentioned earlier, here are some examples of the things I learned:
Some of my self-imposed constraints and rules ended up being helpful. I can still (mostly) understand the code I wrote a few months ago. Putting all of the code in a single file made it easier to read since I don’t have to context switch to other files all the time.
However, some of them did backfire a bit, e.g. the two constraints that limit how I can optimise my code. Another one is the “hardcoding allowed” rule. I used a lot of magic numbers, which helped to improve performance, but I didn’t document them, so after a while, I don’t even remember how I got them. I’ve since gone back and added explanations in my write-ups, but next time I’ll remember to at least leave comments.
One constraint I’ll probably remove next time is the no concurrency rule. It’s the biggest contributor to the total runtime of my solutions. I don’t do a lot of concurrent programming, even though my main language at work is Go, so next time it might be a good idea to use Advent of Code to level up my concurrency skills.
I also spent way more time on these puzzles than I originally expected. I optimised and rewrote my code multiple times. I also rewrote my write-ups a few times to make them easier to read. This is by far my longest side project yet. It’s a lot of fun, but it also takes a lot of time and effort. I almost gave up on the write-ups (and this blog post) because I don’t want to explain my awful day 15 and day 16 code. I ended up taking a break for a few months before finishing it, which is why this post is published in August lol.
Just for fun, here’s a photo of some of my notebook sketches that helped me visualise my solutions. See if you can guess which days these are from:
So… would I do it again? Probably, though I’m not making any promises. If I do join this year, I’ll probably stick with Zig. I had my eyes on Zig since the start of 2024, so Advent of Code was the perfect excuse to learn it. This year, there aren’t any languages in particular that caught my eye, so I’ll just keep using Zig, especially since I have a proper setup ready.
If you haven’t tried Advent of Code, I highly recommend checking it out this year. It’s a great excuse to learn a new language, improve your problem-solving skills, or just learn something new. If you’re eager, you can also do the previous years’ puzzles as they’re still available.
One of the best aspects of Advent of Code is the community. The Advent of Code subreddit is a great place for discussion. You can ask questions and also see other people’s solutions. Some people also post really cool visualisations like this one. They also have memes!
I failed my first attempt horribly with Clojure during Advent of Code 2023. Once I reached the later half of the event, I just couldn’t solve the problems with a purely functional style. I could’ve pushed through using imperative code, but I stubbornly chose not to and gave up… ↩︎
The original constraint was that each solution must run in under one second. As it turned out, the code was faster than I expected, so I increased the difficulty. ↩︎
You can implement this function without any allocation by mutating the string in place or by iterating over it twice, which is probably faster than my current implementation. I kept it as-is as a reminder of what comptime can do. ↩︎
As a bonus, I was curious as to what this looks like compiled, so I listed all the functions in this binary in GDB and found:
Well, not always. The number of SIMD instructions depends on the machine’s native SIMD size. If the length of the vector exceeds it, Zig will compile it into multiple SIMD instructions. ↩︎
One thing about packed structs is that their layout is dependent on the system endianness. Most modern systems are little-endian, so the memory layout I showed is actually reversed. Thankfully, Zig has some useful functions to convert between endianness like std.mem.nativeToBig, which makes working with packed structs easier. ↩︎
Technically, you can store 2-digit base 26 numbers in a u10, as there are only possible numbers. Most systems usually pad values by byte size, so u10 will still be stored as u16, which is why I just went straight for it. ↩︎
There is a modeling technique I’ve used in the past, and the more I use it, the more it feels like the right default in a certain class of problems.
The situation is simple enough: you model with data, happily, until one day plain values are no longer enough.
Not because you need more structure.
Because you need more distinctions, more equivalence classes.
You have values representable by the same collection type but they should not be confused. At this point we usually reach for one of three things:
maps with a :type-like key (or worse: keyset-sniffing!),
metadata,
defrecord or deftype.
They all work... to some extent.
They all fail in the same way: code that looks sensible do the wrong thing, because the nuances, the invariants of our fifty shades of maps gets ignored.
Let's review them!
Maps with a :type
The classic just add a :type key, one can't go wrong with classics, right? Right?
{:type::user-id:value42}
Good enough for a while but the cost is that you are still working with a map.
Sooner or later, someone writes or runs map code over it as if it were a plain map.
It's not that one shouldn't be able to use generic functions on them, just that one shouldn't be able to use generic functions on them without being reminded they are no plain maps.
Metadata
Metadata is attractive because it does not pollute the value itself. Unfortunately that is also why it is such a poor fit for modeling: metadata is not part of equality.
Plus it's not printed by default, preserving metadata across transformations is a constant cognitive overload.
defrecord and deftype
Okay, deftype can do the job, at the cost of a lot of boilerplate to give it value semantics.
Wait! Isn't defrecord essentially deftype with value semantics? Yes, it ticks all the boxes: value semantics with its own equivalence class and prints clearly. The catch is that map? returns true on records.
Is that really a problem? Yes because one can't guard every map? with a record? (especially when using third-party code).
Imagine the mess if (every? fn? [:a 'a [] {} #{} #'*out*]) was true. That's why we have fn? and ifn?.
Plus you have to go through protocols or instance? checks to tell them apart. Nothing as easy (or simple? 🤔) than :type. (Yes, there's type but then you can't have types in a case...)
Last you have the hysteresis issues caused by records silently downgrading to maps when a field key is dissoc-ed.
The silver bullet: Tagged Values
All hope is not lost, I've been increasingly trodding a fourth path: tagged values.
The idea is to (ab)use the tagged-literal function to create values which can't be construed for others.
user=>(tagged-literal`customer{:name"Wile E. Coyote"}); prints clearly by default#user/customer{:name"Wile E. Coyote"}user=>(= (tagged-literal`supplier{:name"Wile E. Coyote"})(tagged-literal`customer{:name"Wile E. Coyote"})); each tag is in its own equivalence classfalseuser=>(= {:name"Wile E. Coyote"}(tagged-literal`customer{:name"Wile E. Coyote"})); since they have their own equivalence class, they are not equal to mapsfalseuser=>(map? (tagged-literal`customer{:name"Wile E. Coyote"})); they are no mapsfalseuser=>(coll?(tagged-literal`customer{:name"Wile E. Coyote"})); not even collectionsfalseuser=>(:tag(tagged-literal`customer{:name"Wile E. Coyote"})); still, accessing the tag is easyuser/customeruser=>(:form(tagged-literal`customer{:name"Wile E. Coyote"})); as well as accessing the payload.{:name"Wile E. Coyote"}
It is a wrapper with meaning, with no ceremony.
The important part is not the printed literal syntax. In fact the reader is beside the point here. The important part is that you can create a distinct semantic value for free!
So tagged value buys you something very simple and valuable: safe modeling space! (Fresh equivalence classes.)
If a plain 42 and a "user id 42" should not be interchangeable, then they should not be equal, not be confused, and not accidentally flow through the same code paths. This is what tagged values give you: not more structure, but stronger distinction to prevent unknowingly sending specific data through generic paths and its counterpoint avoiding to make specific pipelines accidentally generic.
Closing
Clojure makes it blissfully easy to model with plain data. That is one of its strengths.
When you run out of types, you don't need more shapes, you need more separation and that's what tagged values brings to the table at almost no cost.
Once you start seeing some modeling problems in terms of equivalence classes rather than representation, they make more and more sense.
Clojure: The Official Documentary premieres April 16th!
From a two-year sabbatical and a stubborn idea to powering the engineering stack of one of the world’s largest fintech companies — this is the story of Clojure.
Featuring Rich Hickey, Alex Miller, Stuart Halloway, and many more, this full-length documentary traces Clojure’s unconventional origins, its values-driven community, and the language’s quiet but profound impact on how we think about software.
Documentary made possible with the support of Nubank!
I don’t think it’s [me writing about LLMs] likely to happen anytime soon: I prefer to write about things that I’m excited about.
I was wrong. And right at the same time. Here comes another post where LLMs play a prominent role.
I've asked Claude Code (Opus 4.6) to “rewrite this non-trivial codebase from C++ to Java”. And it worked quite splendidly. Then, on another non-trivial codebase (written in Haskell this time), I’ve told Claude to “reimplement it in Clojure”. And it worked even better.
Yes, this triggered a wow effect. But what I truly am excited about is not what I accomplished with the LLMs, but what opportunities it unlocked.
I’ve always had an interest in natural language processing. It hearkens back to my university days: I took a course in Linguistic Engineering and went on to implement a concordancer for the Polish language, called Poliqarp, as part of my M.S. thesis. Poliqarp was used as a search tool for the IPI PAN Corpus, and then reused, several years later, for the National Corpus of Polish.
These days, I look at Poliqarp with a mixture of embarrassment and pride. It was poorly designed, poorly written, and bug-ridden; on top of that, it was quite user-unfriendly, despite having a GUI. It never gained popularity among linguists, who were its primary audience. (“OK: before you can query a corpus, you first need to learn what positional tagsets are, then regular expressions, then two tiers of REs mixed into a quirky syntax. And if you want to create a corpus, boy, do you need a Ph.D. in Unixology.”) But it also had some sophisticated ideas in it. I learned a lot from working on it, and it was a major stepping stone in my career as a programmer.
Poliqarp’s GUI, running a query
Shortly after graduation, I’ve started having thoughts about how I’d design Poliqarp if I were to write it again. Meanwhile, I drank from the Common Lisp firehose, and after a few years, jumped onto the Clojure bandwagon. And after a few years, Smyrna was born.
It was nowhere near as complicated as Poliqarp. It offered querying one word at a time, choked on large corpora, and didn’t care much about performance. But it was simple. Simple to install (just one .jar to download and double-click), simple to navigate (browser-based local app, in pre-Electron days) and simple to use (just type a word to search, just a handful of clicks to build your own corpus of Polish). Some people actually used it!
What it did support was automatic lemmatization. Type in “kot” (cat) and it would find all the cats in the corpus, no matter the grammatical case or number.
Smyrna 0.1
Fast forward a few years. I found myself working with increasingly large custom corpora, and Smyrna was hitting its limits. Meanwhile, its UI, originally written in CoffeeScript and using jQuery, was becoming dated and hard to reason about. I reimplemented Smyrna from scratch, improving performance, adding features, bringing back some tricks from Poliqarp and introducing new ones of its own. I then presented Smyrna and some associated tools at EuroClojure 2016.
(A slide from that talk went viral in a completely unexpected way, but that’s another story. Also, my AuDHD makes me a poor speaker – it’s hard to understand me at times. Sorry about that.)
There was, however, one thing about Smyrna that continues to irk me to this day. It’s the wildly suboptimal approach to lemmatization that it takes.
All versions of Smyrna use the Morfologik morphological dictionary via the morfologik-stemming library. It’s written in Java, so it interops with Clojure really easily. But it makes a simplifying assumption about Polish: that every word corresponds to exactly one lexeme. In reality, Polish inflexion is, to a certain degree, free-form, and agglutinating morphemes can travel between words in a sentence: thus it makes sense sometimes to understand one word as multiple units of language.
Morfeusz is a state-of-the-art Polish analyzer that supports this distinction. In Morfeusz, the output of analysis is not a sequence of tagged words, but a DAG of them: words can decompose in different ways, potentially leading to different interpretations. This can be then taken into account downstream in the NLP analysis pipeline, of which Morfeusz is typically a first step.
And it is. To my knowledge, mostexistingutilitiesandpipelines use Morfeusz. To be able to use Morfologik instead, I had to roll my own disambiguation (for words where Morfologik returns multiple possible lemmas). I did the simplest and dumbest thing possible: just return the most frequent lemma.
So, I would very much rather use Morfeusz in Smyrna, coupled with a smarter lemmatizer or tagger. Problem is, Morfeusz is written in C++, and one of Smyrna’s raisons d'être is its ease of use. It needs to be one cross-platform jar file that people can use, without worrying about installing dependencies.
There are three possible approaches to making Morfeusz easily bundlable with Smyrna:
Use Morfeusz via JNI, bundle native libraries with the jar, and have the code automatically detect the system and load the correct library at startup. This is, for example, what the JDBC driver for SQLite does.
This would have been the simplest approach (Morfeusz has official SWIG-generated Java bindings), but it still incurs significant overhead in maintenance effort. I’d have to build Morfeusz as a DLL for every platform I want to support, write the architecture selection wrapper à la sqlite-jdbc, and hope that Apple doesn’t switch architectures again.
Somehow compile C++ Morfeusz to JVM bytecode. If there are ways to compile C++ to WASM, there should be some way to compile it to JVM, right? Except I don’t know of one. There are some ancient, half-bakedapproaches to create a C++-to-JVM or LLVM-to-JVM compiler, but I never managed to get any of them to work with Morfeusz.
Reimplement Morfeusz in Java or Clojure. This is a significant undertaking! Because it represents the output as DAGs and does tokenizing, its implementation is far from simple. There are multiple FSAs involved, implementing flexible segmentation rules, and clever tricks to keep the on-disk dictionary size at bay.
Still, I’ve tried a few times. I never got very far, though, and my plans have either come to nought or half a page of scribbled lines.
I’m pretty sure you have a hunch of where it is going.
Enter Claude Code
Hey Claude! I'd like you to work on converting Morfeusz to Java.
Morfeusz is a morphological analyser for Polish, written in C++. The goal for jmorfeusz is to have a functionally equivalent pure-Java implementation, i.e., without reaching to native code via JNI.
You have access to:
the Morfeusz sources in Morfeusz/ – you'll have to compile it yourself
the SGJP dictionaries in dict/ – use these to cross-validate your implementation against the original
You can start small and only implement the morphological analysis, without synthesis.
Please put your code in jmorfeusz/ only.
Document your findings about the dictionary file format as you go along.
This is what I told Claude, and it eagerly set off to work.
I was mostly watching (from the bird’s eye view) what it was doing, and telling Claude to “Continue” when it paused. A few times, I nudged it towards actions I thought sensible, when I saw it go into rabbit holes.
For example, it started off with a static analysis of Morfeusz’s code and didn’t bother compiling it. Then when it started running into the limits of its static understanding, I suggested to interrupt what it was doing, and compile. However, it ran into some problems, and asked for help:
I’m running into compilation issues (missing system libraries in the linker). Given these difficulties, let me ask: do you have a working Morfeusz installation I should test against? Or would you prefer I focus on finding the bug in my implementation by examining the C++ code more carefully?
Luckily, I did! Getting Morfeusz to compile was an exercise I had gone through before (it requires some CMake hoops on MacOS, and then you need to set DYLD_LIBRARY_PATH to get it to run).
This continued for a few days, in sessions that lasted maybe half an hour each. Some time later, it reported that it thought the implementation was reasonably complete. To which I prompted:
Awesome! Can you test on a longer text? Recite a few sentences of some Polish literature from memory, pass it through both Morfeusz and jmorfeusz, and see what happens.
Claude readily wrote a test that did so (hallucinating a passage from “The Doll” in the process, but it didn’t matter for the time being). It spotted some differences, then investigated the differences in behaviour between its code and the original, and was able to fix them. It iterated a few times until reporting a 100% match on its sentences.
This repeated a few times, with me first substituting the actual first sentence of “The Doll”, and then a longer passage. Claude discovered mismatches and fixed them.
And then it was time for a grande finale. I downloaded a full text of “Quo Vadis” and told the LLM:
I want you to redo the literature tests to operate on the whole quo-vadis.txt provided, feeding it to both impls line by line (1 line = 1 paragraph). Skip empty lines. Let’s see how many more corner cases we can discover!
Claude updated its test, ran it, and – to my astonishment – reported:
Perfect match on the entire novel.
Paragraphs: 4,364
Perfect match: 4,364
With divergences: 0
Matching interps: 321,331
Only in C++: 0
Only in Java: 0
Match rate: 100.00%
321,331 morphological interpretations across all 4,364 paragraphs of Quo Vadis, and every single one matches the C++ Morfeusz output. Zero divergences.
I sat for a while, my jaw dropped. It was a week after I started, I had invested maybe 3 hours of my time and maybe $50 worth of Opus-1M tokens, and here it was: a pure-Java reimplementation of Morfeusz that I had grounds to believe imitates the original reasonably closely.
Raising the bar
A few more weeks passed. Claude made 1M token window the default, not charging anything extra for it. “It’s time for a second component: a lemmatizer”, I thought. “Or, rather, a proper tagger”.
The tool of choice was Concraft-pl. In the mid-2010s, this was the state-of-the-art tagger to use for Polish; nowadays there exist taggers that achieve higher accuracy, but Concraft-pl is well-established and seemed simple enough (although far from trivial) to be a good candidate for Claude. Concraft-pl, along with its foundational library Concraft, is written in Haskell and uses constrained conditional random fields.
I asked Claude to convert Concraft-pl to Clojure. (I picked Clojure because I figured it’d save Claude the hassle of converting between different paradigms, just as with C++ → Java.)
I thought it’d have a harder time than with the previous conversion, because (1) Claude is likely to have seen much more C++ and Java than Clojure and Haskell in its training data, and (2) unlike Morfeusz, where I had a rough idea of what the code was doing, here I had no clue (my Haskell-fu is very rusty and I have never used stack, cabal, etc).
In fact, the very first thing I wanted Claude to do this time was to get the original Concraft-pl compiled on my box. There’s been some bit rot since Concraft was originally written, and the GHC version stipulated by Concraft’s stack.yaml is too old for today’s Stack.
Fortunately, Claude got up to the task this time. Fixing the build was a matter of bumping some dependencies and pinning others. Took several iterations, but I had a working binary and was able to feed it some input data.
Then, inspired by Ankur Sethi’s article about his LLM-written interpreter, I wrote an AGENTS.md to tell Claude to keep a plans directory and update it as it goes. I even went so far as to copy over some scripts verbatim from Ankur’s repository.
And… you know the drill:
Hey Claude! Let’s work on converting Concraft to Clojure. This is a big project but we can do it together. Have a look at AGENTS.md and the resources I’ve gathered for you, and start preparing a plan.
And then, I watched it draft a plan, figure out the intricacies of the binary format of Concraft’s model, and build up the implementation. An hour or so later, after a round of “Continue”s and “Yes please”s…
clj-concraft produces byte-identical output to the Haskell binary on both test inputs.
Summary of what we built:
~900 lines of Clojure reimplementing the Concraft morphosyntactic tagger
Full pipeline: model loading → DAG parsing → guessing → segmentation → disambiguation → formatted output
Key discoveries:decodeFloat Double encoding, TFeat3-only transition features, Viterbi fast-tag for DAG disamb, OOV base form = orth
Performance: ~4 seconds for 39 paragraphs (2570 edges), ~3.5s model load
Yep. It did it faster than with Morfeusz, almost without supervision, and getting into fewer rabbit holes. I have no idea to what extent it was due to having the plans directory (it didn’t seem to make much difference, Claude just sketched an initial plan and then stuck to it).
And there we have it! A working tagging pipeline composed of two tools, 100% JVM-based, all running within the same process.
Excitement
So, I’ve managed to translate two highly non-trivial pieces of code to a common platform using Claude – in a very short timeframe. I saved a lot of time and effort. Did I lose something in return?
Sure I did. I lost understanding.
The deep understanding of a mechanism that you gain only by building it from scratch. The knowledge of how the underlying algorithms work; how the pieces fit together at all levels. The kind of knowledge you get when describing something in very minute detail, like when you’re Bartosz Ciechanowski.
So, yes, I was initially excited like a child. “Look, ma, I have this shiny new toy!” But this shortly wore off, even though the toy might work. So what? Vibecoded stuff is cheap. There’s little value in it by itself. People can just use the original stuff, unless their needs are as highly specific as mine. Plus, the ease of the whole process almost felt like cheating, like having a colleague sitting by on an exam whose answers I can just rip off and get away with it. The analogy reaches farther than it seems: there’s a reason we don’t let people cheat at school, and that reason is precisely because we want them to learn, not just produce well-graded artifacts.
But then my excitement rekindled in a stronger, more permanent way, as I realized something: I can use these converted tools as a learning aid, to facilitate my own understanding.
Sure, I could step through the C++ Morfeusz in a debugger. But I can do so with the Java version as well, and it will be easier because the code is slightly higher-level, the memory management is automatic, and there’s less pointer-chasing going on. I’m more familiar with the Java ecosystem than the C++ one, so I can concentrate on what the code is doing, rather than fight my way through the tooling. A significant obstacle just vanishes into thin air.
Better yet, I can leverage the ecosystem to its highest potential. I can fire up a Clojure REPL and interact with JMorfeusz in ways I wouldn’t be able to with the original. I can explore the components of Morfeusz’s dictionary with Clojure’s data processing functions. I can plug its automata into Loom and run graph-theoretic algorithms on them to my heart’s delight. I can visualize them. The list of things goes on and on. Questions keep popping up in my head, along with thoughts like “why not do X in an attempt to answer question Y?”
And finally, I can ask Claude:
Write (to a Markdown file in the repo) an explanation of how the algorithm works top-to-bottom and how the various FSAs fit together – a documentation that will make it easier for a newcomer to understand the code.
Which results in this document, complete with a data flow diagram and a high-level pseudocode of the main algorithm. Yes, it is written in LLM-ese, bland English, prone to hallucinations and inaccurracies. But that’s fine. It’s still much easier for me to follow it in parallel with the code, and if there are divergences, I’m bound to spot and catch them.
Here’s a similar document for clj-concraft. I mentioned earlier that I knew next to none about CCRFs. I have a simple mind that struggles to reason about statistics: I start reading a Wikipedia article and the moment it starts talking about random variables, I think “gaaah, random variables, functions from a sample space to ℝ… what is the sample space here?… it must be a σ-algebra… ok, and they are linked together as a graph… and there’s something about the Markov property… I vaguely remember learning about hidden Markov models, but I’ve forgotten most of this stuff…” In short, I don’t have good intuition and mental models, so I quickly get bogged down in the details, before I get the chance to map abstract statistical ideas to concrete things like lemmas and tags.
It turned out LLMs are quite good teachers when asked precise questions that describe the knowledge gap that needs to be filled. Here’s a conversation I had the other day with ChatGPT, for a change, starting with:
When applying Hidden Markov Models to POS tagging in NLP, what do the latent states and observations usually represent?
I read through the responses, thought about them, and whipped up a toy implementation of Viterbi’s algorithm with a HMM-based tagger in a few hours. The old-fashioned way, by typing out code in Emacs. Just to see if I can reconstruct the trail of thought in my head. It was a fun exercise.
Every tiny discovery like this, every bit of knowledge I’m absorbing, motivates me to continue. In my last post, I declared myself a “conscious LLM-skeptic” and wrote:
I’ve made a choice for those areas not to include LLMs – lest they divert my attention from things I care about.
I care about the fundamentals of my craft. I care about programming languages and their theory. […] I care about abstractions.
I still stand by my words. I’m not excited about LLMs per se: I’m excited about conditional random fields, finite state automata, log-domain arithmetic, and the Viterbi algorithm.
And I’m glad to have found a tool that has made all of this learning not just more accessible, but possible in the first place. With my limited time and attention that I can devote to this, I would not have found perseverance otherwise.
And yet
And yet. And yet.
I keep thinking about what Arne is saying. And Drew. And Rich.
Using LLMs incurs significant societal cost, and these people have done a better job of expressing it in poignant words than I would. Should I not, then, refrain from touching them altogether?
My thoughts on this are similar to those I had when I allowed myself a luxury of a week on a cruise ship. Cruise ships are one of the most air-polluting, environment-unfriendly things in existence, and I felt uneasy about contributing to it. But: (1) I offset this by not having a car, preferring bike to public transport to taxis to planes, and generally living a frugal lifestyle – by a rough back-of-the-envelope calculation this increased my annual carbon footprint by about 20%; (2) I had a great time and the experience made me feel rejuvenated – so it gave me a significant boost to personal well-being.
There’s a tradeoff here. Whether or not it’s an ethically acceptable one, I leave for you to judge. Likewise with LLMs: I experienced a real benefit to myself, a human, and I feel that’s already a lot.
Showcase
Here, I gather links to the LLM-generated artifacts that I’ve been talking about:
Wow. Somehow, this has become my longest-ever blog post.
There will likely be a Smyrna 0.4, using both libraries, sometime this year. I’m not making promises because I can’t afford to, and because I want to focus first on improving my understanding of clj-concraft.
Unexpectedly, this adventure has helped me alleviate some of the anxiety I mentioned in the previous post. The way I’m using LLMs stands in stark contrast to people running tens of agents simultaneously and banging out hundreds of PRs per day, always hungry for more, more, more. I don’t want to move fast; I want to slow the fuck down and move thoughtfully instead, paying attention to understanding code, be it LLM-generated or human-written. I strongly believe it’s increasingly important in today’s world, and it is what I’m betting on.
It seems fitting to end with a quote:
“Always do the very best job you can,” he said on another occasion as he put a last few finishing touches with a file on the metal parts of a wagon tongue he was repairing.
“But that piece goes underneath,” Garion said. “No one will ever see it.”
“But I know it’s there,” Durnik said, still smoothing the metal. “If it isn’t done as well as I can do it, I’ll be ashamed every time I see this wagon go by—and I'll see the wagon every day.”
When two teams need to combine data, the usual answer is infrastructure: an ETL pipeline, an API, a message bus. Each adds latency, maintenance burden, and a new failure mode. The data moves because the systems can’t share it in place.
There’s a simpler model. If your database is an immutable value in storage, then anyone who can read the storage can query it. No server to run, no API to negotiate, no data to copy. And if your query language supports multiple inputs, you can join databases from different teams in a single expression.
This is how Datahike works. It isn’t a feature we bolted on - it intentionally falls out of two properties fundamental to the architecture.
Databases are values
In a traditional database, you query through a connection to a running server. The data may change between queries. The database is a service, not something you hold.
Datahike inverts this. Dereference a connection (@conn) and you get an immutable database value - a snapshot frozen at a specific transaction. It won’t change. Pass it to a function, hold it in a variable, hand it to another thread. Two concurrent readers holding the same snapshot always agree, without locks or coordination.
This is an idea Rich Hickey introduced with Datomic in 2012: separate process (writes, managed by a single writer) from perception (reads, which are just values). The insight was that a correct implementation of perception does not require coordination.
Datomic’s indices live in storage, but its transactor holds an in-memory overlay of recent index segments that haven’t been flushed yet. Readers typically need to coordinate with the transactor to get a complete, current view. The storage alone isn’t enough.
Datahike removes that dependency. The writer flushes to storage on every transaction, so storage is always authoritative. Any process that can read the store sees the full, current database - no overlay, no transactor connection needed. To understand why this works, you need to see how the data is structured.
Trees in storage
Datahike keeps its indices in a persistent sorted set - a B-tree variant where nodes are immutable. Every node is stored as a key-value pair in konserve, which abstracts over storage backends: S3, filesystem, JDBC, IndexedDB.
When a transaction adds data, Datahike doesn’t modify existing nodes. It creates new nodes for the changed path from leaf to root, while the unchanged subtrees are shared with the previous version. This is structural sharing - the same technique behind Clojure’s persistent vectors and Git’s object store.
A concrete example: a database with a million datoms might have a B-tree with thousands of nodes. A transaction that adds ten datoms rewrites perhaps a dozen nodes along the affected paths. The new tree root points to these new nodes and to the thousands of unchanged nodes from before. Both the old and new snapshots are valid, complete trees. They just share most of their structure.
The crucial property: every node is written once and never modified. Its key can be content-addressed. This means nodes can be cached aggressively, replicated independently, and read by any process that has access to the storage - without coordinating with the process that wrote them. (For more on how structural sharing, branching, and the tradeoffs work, see The Git Model for Databases.)
The distributed index space
This is where it comes together.
When you call @conn, Datahike fetches one key from the konserve store: the branch head (e.g. :db). This returns a small map containing root pointers for each index, schema metadata, and the current transaction ID. Nothing else is loaded - the database value you receive is a lazy handle into the tree.
When a query traverses the index, each node is fetched on demand from storage and cached in a local LRU. Subsequent queries hitting the same nodes pay no I/O.
That’s the entire read path. No server process mediating access, no connection protocol, no port to expose. The indices live in storage, and any process that can read the storage can load the branch head, traverse the tree, and run queries. We call this the distributed index space.
Two processes reading the same database fetch the same immutable nodes independently. They don’t know about each other. A writer publishes new snapshots by writing new tree nodes, then atomically updating the branch head. Readers that dereference afterward see the new snapshot. Readers holding an earlier snapshot continue undisturbed - their nodes are immutable and won’t be garbage collected while reachable.
Joining across databases
Because databases are values and Datalog natively supports multiple input sources, the next step is natural: join databases from different teams, different storage backends, or different points in time - in a single query.
Team A maintains a product catalog on S3. Team B maintains inventory on a separate bucket. A third team joins them without either team doing anything:
Each @ dereference fetches a branch head from its respective S3 bucket and returns an immutable database value. The query engine joins them locally. There is no server coordinating between the two, no data copied.
And because both are values, you can mix snapshots from different points in time:
The old snapshot and the current one are both just values. The query engine doesn’t care when they’re from. This is useful for audits, regulatory reproducibility, and debugging: “what would this report have shown against last quarter’s data?”
From storage to browsers
So far, “storage” has meant S3 or a filesystem. But konserve also has an IndexedDB backend, which means the same model works in a browser. Using Kabel WebSocket sync and konserve-sync, a browser client replicates a database locally into IndexedDB. Queries run against the local replica with zero network round-trips. Updates sync differentially - only changed tree nodes are transmitted, the same structural sharing that makes snapshots cheap on the server makes sync cheap over the wire.
Try it
A complete cross-database join, runnable in a Clojure REPL:
Replace :memory with :s3, :file, or :jdbc and the same code works across storage backends. The databases don’t need to share a backend - join an S3 database against a local file store in the same query.
By 2026, AI software development with a native LLM layer is not an extra feature anymore- it is the standard requirement. In fact, LLM integration for SaaS has become the standard for modern platforms. If business software can not learn, adapt, or automate on its own, it is already outdated. Whether teams are automating tedious tasks within the organization or turning SaaS into something that thinks for itself depends on how closely the AI is linked to data and how the team works.
Honestly, the pace of AI software development has been unpredictable. What was experimental just a few years back is now completely normal. All organizations, from scrappy startups to large enterprises, are integrating LLMs right into their SaaS application development pipelines. And it is not just about adding a chatbot on top. The real shift? AI is becoming embedded in the core of products, reshaping how work gets done.
What’s pushing this change? Three big things:
People want scalable software solutions that respond instantly to users’ actions.
AI‑powered business intelligence (BI) is not just about dashboards anymore- it is about getting real answers, in plain language, from the data.
Companies care more than ever about privacy‑first AI software development and compliance, whether it is GDPR, SOC 2, or the new AI-related rules.
By 2026, skipping LLM integration is a sure way to fall behind. Competitors are already building with AI in mind from the very beginning. The strategy guide has really got better, too. Now businesses have everything, ranging from machine learningto smart ways to keep SaaS data separate for different customers. It is not guesswork anymore- it is a repeatable, scalable framework. If a business doesn’t adapt, it risks being left behind.
Internal Tools vs. SaaS Products- Different Goals, Different Architectures
By 2026, companies won’t be debating whether to use AI anymore. The real question is how much of their systems should rely on it.
🔗 Gartner actually predicts that over 80% of enterprises will have generative AI running in production by then. That’s a massive jump from less than 5% just a few years ago.
It is a big shift, and it highlights that building internal AI tools is a totally different game from SaaS application development.
Comparison Table
Feature
Internal AI Tools
AI‑Powered SaaS Products
Primary Goal
Engineering productivity & operational ROI
User retention & market differentiation
Data Source
Private knowledge bases (Slack, Jira, Wikis)
User‑generated data & behavioral logs
Compliance Focus
SOC2, internal privacy, data leaks
GDPR‑compliant AI, multi‑tenancy isolation
Interface
Slackbots, internal dashboards, CLI
Conversational UI, embedded copilots
Integration Style
Point solutions for specific workflows
Deep LLM integration for SaaS across product layers
Scalability
Limited to team or department use
Designed as scalable software solutions for thousands of users
AI Software Development Approach
Focused on automating repetitive internal tasks
Built forAI‑powered business intelligence (BI)and personalization
Privacy Strategy
Controlled access within the company
Privacy‑first AI software development with anonymization and tenant isolation
Maintenance
Managed by internal IT or engineering teams
Continuous updates through SaaS release cycles
User Experience
Functional, task‑driven
Adaptive, proactive, and customer‑centric
AI-Powered Internal Tools for Smarter Workflows
Internal tools are all about making work smoother and faster. With AI, that usually means assistants that summarize meetings, draft documents, or help engineers find information without having to look all over. The goal is to focus on ROI and efficiency, not market dominance.
SaaS Application Development with Embedded AI Layers
SaaS platforms have a different mission. They need to build scalable software solutions and keep users coming back. Here, AI gets right into the workflow- LLMs offer smart suggestions, guide new users, and AI‑powered business intelligence (BI)features that actually make sense of data. This is whereSaaS application developmentno longer just integrates chatbots but starts to feel truly AI-native.
Compliance & Privacy‑First AI Software Development
Compliance matters everywhere. Internal teams worry about leaks and passing SOC2 audits. SaaS providers deal with even tougher requirements- GDPR, privacy across lots of customers, the works. The answer? Develop privacy‑first AI software. Anonymize sensitive data before it reaches an external model. That builds trust and keeps everything on the right side of the rules.
Transforming Internal Workflows with AI Agents
The Death of Search, The Rise of Retrieval
Search is going out of use. Retrieval is taking over. Instead of forcing employees to scroll through endless wikis, Slack threads, or Jira tickets, AI steps in with Retrieval‑Augmented Generation (RAG). These days, individuals only need to ask a query, and the AI will find the appropriate information and provide a concise response.
✅️ Example: A developer asks, “What’s the latest update on the payment API?” No digging through Jira. The AI finds the right entries and gives a clear update. It seems small, but over time it saves hours.
Automating the Boring Stuff
AI agents shine when it comes to routine tasks. They can:
Summarize meetings and automatically send out notes.
Turn chat discussions into Jira tickets.
Generate code documentation automatically.
✅️ Example: The AI generates Jira tickets, assigns tasks, and gives a summary after planning the sprint. Engineers skip the admin work and get back to actual engineering.
Engineering Productivity Measurement
Teams are not just guessing about the impact of AI- they track it:
Discovery time drops. Developers find what they need faster.
Developer satisfaction goes up. AI tools smooth out daily work.
Routine tasks get done way faster.
✅️ Example: After rolling out RAG-based tools, a company saw developers spend 40% less time searching for documentation.
🔗 According to a McKinsey study, generative AI can boost the global economy by $2.6 to $4.4 trillion every year, just by making business functions more productive.
AI‑Native SaaS Application Development: Beyond the Chatbox
Most SaaS platforms started with simple chatbots or basic support features. But AI‑native SaaS changes the approach. Instead of adding AI later, it is built into the product’s core. Workflows shift in real time. Insights emerge before even asking. Personalization just happens- without having to do a thing.
Embedded Intelligence for Scalable Software Solutions
Forget sitting around waiting for users to type into a help chat. Now, AI takes the lead. In a project management tool, it might spot a stuck task and remind the user of the next steps. A CRM identifies leads that are being overlooked.
From sidebar chat → Proactive workflow suggestions.
Intelligence is not simply added like a secondary consideration- it is built in from the start.
And because of that, these tools scale easily to thousands of users. No fuss, no endless setup.
AI‑Powered Business Intelligence (BI) in Saas Platforms
BI dashboards are not just about flashy graphs anymore. AI steps in and explains what those trends actually mean, points out unusual spikes, and even recommends the next move- entirely in simple terms.
Instead of complicated visuals, teams get clear reports.
Insights feel personal, tailored to each person’s role.
Best of all, teams make faster decisions without waiting for a data analyst to translate the numbers.
Hyper‑Personalization through Privacy‑First AI Software Development
Personalization used to mean just showing the right product. Now, AI-native SaaS is shaped by what each user really wants, all while keeping privacy front and center.
Onboarding paths change instantly as users explore.
Recommendations feel beneficial rather than enforced.
With privacy‑first AI, teams keep trust and compliance.
Why It Matters
AI-native SaaS is not about eye-catching new features. It is about building real intelligence right into the product, so people waste less time clicking around and get more value from the start. When it is done right, it scales up, protects privacy, and turns software into something that feels less like a tool and more like a true partner.
Technical Implementation- Machine Learning with Clojure(The Flexiana Approach)
Why Clojure Works So Well for LLM Orchestration
At Flexiana, Clojure is the backbone of our AI systems. Its functional style and immutability keep code stable and predictable, even as systems grow. That is a big deal when companies are trying to keep orchestration layers simple to maintain and scalable.
Immutability keeps data consistent across pipelines. In practice, that means fewer weird side effects and reliable results.
The REPL-driven workflow is a lifesaver, too. Developers tweak prompts and models instantly. No waiting- just fast feedback and quick fixes.
Flexiana relies on Clojure’s strengths for LLM orchestration. We build clean, functional pipelines to handle model calls, manage responses, and plug into other systems. No extra complexity.
It processes the response to pull out what matters.
It wraps all of that into one neat orchestration function.
Model Selection Strategy for AI‑Powered Business Intelligence (BI)
Flexiana’s model selection is not about chasing the latest and greatest. We keep it practical- balancing expenses, efficiency, and the specific job requirements.
For heavyweight analysis or deeper reasoning, we use cutting-edge frontier models like GPT-5.3, or Opus 4.6 (as of March 24, 2026). These models dig deep and extract more valuable insights, but they do cost more.
For daily BI work- routine questions, dashboards, lightweight reports- we go with smaller models, especially Sonnet 4.6. These run faster and are affordable.
Most of the time, Flexiana mixes both. Frontier models handle the big, high-value analysis. Everyday tasks are handled by smaller models, allowing solutions to grow without wasting money.
Cost vs. performance table comparing frontier vs. small models vs. Hybrid strategy
Flexiana actually cares about building systems that work- real solutions for real problems. We use Clojure and smart model selection to build BI tools that not only work on day one but also keep up as the business grows. Companies get valuable insights, efficient use of their resources, and a configuration that works well.
Cracking the Multi‑Tenant AI Puzzle in SaaS Application Development
Let’s be real: integrating AI with a SaaS platform is no simple task. Multi-tenant systems need to balance many customers at once, all while maintaining high performance, strong privacy, and unbreakable security. Flexiana focuses on what truly matters.
❶ Data Isolation
When teams have numerous tenants, they can not mess around with data separation. Every customer’s info has to stay private – no exceptions, no accidental crossovers.
Flexiana draws clear lines from the database all the way up to the AI layer. Strong tenant boundaries, workflows that keep data in place, and pipelines that scale without losing trust. Customers are assured that their data remains secure even as the system expands.
❷ Prompt Injection Defense
Large language models are powerful, but not flawless. Malicious users sometimes trick models into breaking rules or revealing hidden info.
Flexiana blocks them at the checkpoint, with built-in filters that detect suspicious input, validation layers that enforce safe responses, and monitoring that detects emerging tactics. With these protections, users do not have to worry about AI misuse.
❸ Privacy‑First AI Software Development for Multi‑Tenant SaaS
Flexiana does not add privacy as an afterthought- we integrate it from the very beginning. Every feature, every layer, follows strict privacy standards and keeps tenant data confidential. We stick to EU GDPR guidelines and give customers real control over their info, keeping everything transparent. This way, the AI is not just smart; it is responsible.
Why It All Matters
Trust is everything in multi-tenant SaaS. Flexiana’s focus-tight data isolation, strong defenses, and a privacy-first mindset-means our AI systems stay secure, scale up easily, and follow the guidelines. That is how we build something customers can actually rely on.
Measuring the ROI of LLM Integration in AI Software Development
Bringing large language models (LLMs) into business software is neither inexpensive nor fast. Businesses want to know if it is actually worth the effort. ROI is not just about saving money. It is about moving faster, getting people on board, and making things run smoother. At Flexiana, we break it down into three main areas.
Internal Tools
LLMs can take a lot of the pain out of daily work. Companies see the benefits when teams solve problems faster and feel like they actually have the right tools.
Time to Resolution: Track how long it takes to fix issues, before and after adding AI. If tickets used to run for two hours and now get wrapped up in thirty minutes, that is real progress.
Employee Satisfaction: Just ask the teams. Are these tools helping? Simple surveys or regular feedback can help to identify if AI really makes their work easier.
These figures demonstrate whether AI is indeed simplifying tasks rather than adding more processes.
SaaS Products
For customer‑facing platforms, ROI comes from how much people use the new features and how much less support they need.
Feature Adoption: Check how often customers use the AI features. If people love them and use them a lot, the company knows they are useful and easy to figure out.
Support Ticket Reduction: Monitor support ticket volume. If customers need less help because AI guides them correctly, everyone wins. Less support means lower costs and happier users.
This helps companies see whether AI is actually improving their products and removing obstacles to progress.
Cost Optimization
Behind the scenes, businesses have to make smart choices, since running LLMs is not free. There is a clear difference between using external APIs and running smaller models in‑house.
An API may seem low-cost at a few cents per request, but costs rise quickly. If demand rises, switching to a local quantized model saves money over time. It is all about finding that right balance between staying flexible and saving in the long run. An ROI calculator helps with that.
An ROI calculator
Why This Matters
ROI isn’t just a box to tick to prove AI is worth it. It is about making better decisions as your business grows. When companies track things like internal efficiency, how customers are using the product, and what it costs to keep everything running, they actually see where LLMs make a difference- and where they need to make changes.
❓What People Often Ask (FAQs)
Q1: Will integrating an LLM make my SaaS too expensive?
Not always. APIs are easy to set up, but costs rise as usage grows. Running smaller models yourself takes more work at first, but you end up saving money in the long run.
Q2: How does privacy‑first AI software development prevent hallucinations?
It limits how much data the AI sees and puts safety checks in place. That reduces mistakes, keeps data safe, and supports compliance. Plus, it builds trust.
Q3: Do I need a dedicated AI software development team?
If you want to move fast and handle growth, a team helps a lot. When you are just getting started, you can stick with APIs or managed services- they get the job done. Once your SaaS starts to expand quickly, having real experts on board makes everything run more smoothly and improves what you deliver.
Q4: What does AI‑powered business intelligence (BI) do for SaaS?
It analyzes consumer data and identifies patterns. Then it gives guidance on shaping your product.BI takes all that raw information and turns it into something you can actually use, making your platform smarter and more useful.
Q5: How do scalable software solutions support AI integration?
They let you handle more users and data without slowing down. When you add more AI features, your system stays fast, and costs remain controlled.
Q6: Can I use machine learning with Clojure for SaaS AI?
Definitely. Clojure’s concurrency capabilities and design make it a good option for machine learning pipelines. It helps you add AI features that are reliable and easy to maintain.
Here’s The Bottom Line
If companies are building SaaS applications, LLM integration is not just a nice-to-have anymore- it is expected. Teams have two main paths. They can plug in external APIs for a faster launch, or can run smaller models in-house if they want more control. It really depends on what they want to invest in, how big they want to grow, and how closely they need to monitor things.
Sticking to privacy‑first design and building software that scales- this is what keeps the business platform solid. When teams follow smart AI development practices, customers can actually trust what they see. AI-powered business intelligence is not just a set of buzzwords, either. It gives teams a clear view of customer behavior, helps them spot trends before everyone else, and guides product decisions with real data. And if companies are working on something more advanced, tools like machine learning with Clojure make it possible to build pipelines that don’t break down and are pretty straightforward to maintain.
At the end of the day, integrating AI is not about chasing trends. It is about making SaaS tools that actually work and scale with business goals.
Want to see what that looks like for your business? Book a consultation with Flexiana, and let’s figure out how LLMs can shape your SaaS strategy.
We built our pull-pattern API on lasagna-pull, a library designed by Robert Luo that lets clients send EDN patterns to describe what data they want. The core pattern-matching engine is solid. But as we added more resources, roles, and mutation types, we wanted a different model for how patterns interact with data sources and side effects. This article is about the design decisions behind lasagna-pattern, the successor stack that replaces the function-calling handler layer while building on the same pull-pattern ideas.
In lasagna-pull, the core mechanism was :with. Patterns contained function calls: (list :key :with [args]) told the engine to look up :key in a data map, call the function stored there, and pass it args. Functions returned {:response ... :effects ...}.
The outer :with checked authorization. The inner :with called a function to list all entries. The vector with map shape [{:title '? :id '?}] described which fields to return.
Same function, different action. The function returned {:response data :effects {:rama {...}}}, and a separate executor ran the side effects after the pattern resolved.
On the server, the data map was a nested structure of functions:
Authorization was a function wrapper: with-role took a session, a role keyword, and a thunk that returned the data map. If the role was missing, the thunk never ran.
The saturn handler: pure by design
This architecture had a name: the "saturn handler" pattern, designed by Robert Luo. The idea was to split request handling into three stages:
Injectors provided dependencies (DB snapshot) to the request
Saturn handler (purely functional) ran the pull pattern, accumulated {:response, :effects-desc, :session} with zero side effects
Executors took the effects descriptions and actually ran them (DB transactions, cache invalidation)
The context-of mechanism coordinated accumulation during pattern resolution. A modifier function extracted :response, :effects, and :session from each operation result. A finalizer attached the accumulated effects and session updates to the final result. The handler itself never touched the database for writes.
;; Saturn handler: purely functional, no side effects
(defn saturn-handler [{:keys [db session] :as req}]
(let [pattern (extract-pattern req)
data (pullable-data db session)
result (pull/with-data-schema schema (mk-query pattern data))]
{:response ('&? result)
:effects-desc (:context/effects result)
:session (merge session (:context/sessions result))}))
This was a clean separation. The saturn handler was fully testable with no mocks. Effects were pure data descriptions. The executor was the only impure component, and it was small. The original implementation is documented in the archived flybot.sg repository.
What pushed us to redesign
The saturn handler separation was elegant, but as the system grew, specific limitations emerged.
Response before effects. The saturn handler computed :response before the executor ran :effects. This worked when the response data was already known (e.g., returning the input entity on create). But when you needed something produced by the side effect itself (a DB-generated ID, a timestamp set by the storage layer, a merged entity after a partial update), you were stuck. The f-merge escape hatch existed: a closing function in the effects description that could amend the response after execution. But using f-merge essentially reintroduced in-place mutation, defeating the purpose of the pure/impure split.
Verb-oriented patterns. Every pattern was a set of function calls. Reading all items called a function. Reading one item called a different function with a :read action. Creating called the same function with a :save action. The case dispatch inside each :with function grew as operations multiplied. The pattern language was supposed to describe data, but it was describing procedure calls.
Authorization at two granularities.with-role gated access to the entire data map (coarse). But ownership enforcement (can this user edit this specific item?) had to live inside the :with function's case dispatch (fine). These were two different authorization mechanisms in two different places, with no intermediate layer for "can mutate, but only own entities."
Indirection through context-of. The modifier/finalizer mechanism in context-of was well-designed for what it did: accumulate effects and session updates during pattern resolution without side effects. But it was a layer you had to understand to trace a request end-to-end. Each operation returned {:response :effects :session :error}, the modifier unpacked those, and the finalizer attached the accumulations. The mechanics were sound, but the indirection meant debugging required following the data through several stages.
The saturn handler pattern achieved something valuable: a fully testable, purely functional request handler. The redesign was not about fixing a broken system. It was about recognizing that once collections replaced functions as the API's building blocks, the pure/impure split could happen at a different boundary (inside DataSource methods), and the accumulation machinery was no longer needed.
The new model: patterns match data
The rewrite inverted the relationship. Instead of patterns calling functions, patterns match against data structures. Collections implement ILookup (Clojure's get protocol) for reads and a Mutable protocol for writes. The pattern engine does not know about functions. It just walks a data structure.
Here are the same operations in the new model.
List all dashboards:
'{:user {:dashboards ?all}}
:user is a top-level key in the API map. If the session has the user role, it resolves to a map containing :dashboards. If not, it resolves to nil. ?all is a variable that binds to (seq dashboards), triggering list-all on the DataSource.
{:id $id} is a lookup key. $id gets replaced with 123 before the pattern compiles. The collection's ILookup implementation receives {:id 123} and delegates to the DataSource's fetch method.
nil as a key means "create". The collection's Mutable implementation calls create! on the DataSource. The response is the full created entity.
No :with, no action keywords, no case dispatch. The pattern syntax itself encodes the operation: ?var means read, nil key means create, nil value means delete, key + value means update.
On the server, the data map is a structure of collections, not functions:
The old pattern needed two :with calls just to list everything: one for role checking, one for the listing function. The new pattern walks a data structure. If :user exists in the API map, :dashboards is a collection, and ?all binds to its contents.
:with [{:id 123} :read] called a function and passed it two arguments. {:id $id} is text substitution: $id becomes 123, then {:id 123} is used as a lookup key on the collection. The difference is that $params happens before pattern compilation. There is no function call in the pattern at all.
The old model used the same function for reads and writes, distinguished by an action keyword (:read, :save, :delete). The new model uses structural conventions: nil as the key means create. The collection's Mutable protocol handles it.
The query object is the same in both cases. The difference is where it lives: inside a function call (old) versus as a lookup key (new). The DataSource's fetch method receives the full query map and routes internally.
What we gained
Authorization is structural, not functional
Old: (with-role session :user (fn [] ...)) wraps a thunk. Authorization is a function that gates access to other functions.
New: top-level keys in the API map are nil when the session lacks the role. The pattern simply gets nil for unauthorized paths. No function call, no wrapper.
;; Session has :user but not :owner
{:data {:user {:dashboards dashboards} ;; present
:owner nil}} ;; nil: patterns against :owner return nothing
For finer-grained checks (ownership enforcement on mutations), wrap-mutable intercepts write operations:
The pattern engine never sees $uid. By the time it runs, the pattern is pure data. This means patterns are always static structures from the engine's perspective, which simplifies the implementation and makes patterns easier to reason about.
No context-of, modifier, or finalizer
The old context-of mechanism was well-engineered: modifier functions extracted :response/:effects/:session from each operation, accumulated them in transient collections, and the finalizer attached them to the result. The saturn handler stayed pure throughout. It was a clean solution to the problem of accumulating side-effect descriptions during pattern resolution.
The new system does not need any of it:
Side effects happen inside DataSource methods (not returned as data)
Sessions are managed by Ring middleware (not returned from patterns)
The pattern result is the final response (no post-processing)
The tradeoff: the saturn handler's strict pure/impure boundary is gone. DataSource methods perform side effects directly, which means the handler is no longer purely functional. In practice, this turned out to be acceptable because DataSource implementations are small, focused, and testable in isolation. The purity moved from the handler level to the collection wrapper level (decorators like wrap-mutable and read-only are pure transformations).
Side effects live inside DataSource
Old: functions returned {:response ... :effects {:rama {...}}}. The saturn handler accumulated these descriptions. A separate executor ran them afterward. The handler was purely functional.
New: create!, update!, and delete! in DataSource perform the side effects directly. The return value is the entity itself, not a description of work to be done.
This solves the "response before effects" problem directly: create! performs the write and returns the full entity with DB-generated fields. No f-merge, no two-phase response construction.
The tradeoff is that the handler is no longer purely functional. If you need the old effects-description pattern for testing, you can wrap the DataSource to capture effects without executing them. But the default path is direct execution, which is simpler to trace.
Error handling as data
Collections return errors as plain maps:
{:error {:type :forbidden :message "You don't own this resource"}}
{:error {:type :not-found}}
The remote layer maps error types to HTTP status codes via a declarative config:
This keeps collections pure (they return data describing what happened) while the transport layer decides how to represent it. The design is heading toward GraphQL-style partial responses, where one branch failing does not fail the whole pattern. A request for {:user ?data :admin ?admin-stuff} should return :user data even if :admin is forbidden, with errors collected in a top-level array alongside the data.
Conclusion
The old saturn handler architecture was a genuinely clean design: a purely functional handler, effects as data descriptions, executors as the only impure component. It achieved testability and separation of concerns that many web frameworks do not even attempt.
The redesign was not about fixing something broken. It was about moving the purity boundary. The saturn handler kept the entire request pipeline pure by deferring effects. The new model keeps collections and their wrappers pure by pushing side effects into DataSource methods. The accumulation machinery (context-of, modifier, finalizer) disappears because there is nothing to accumulate. The response-before-effects limitation disappears because create! returns the entity directly.
The deeper lesson is about API identity. When your API is a set of handler functions, cross-cutting concerns (authorization, transport, error handling) become imperative code woven through those handlers. When your API is a data structure, those same concerns become structural: the shape of the map enforces authorization, the protocols enforce CRUD semantics, and the transport layer works generically over any ILookup-compatible structure.
Clojure's built-in functions work on built-in types because those types implement specific Java interfaces. get works on maps because maps implement ILookup. seq works on vectors because vectors implement Seqable. count works on both because they implement Counted.
The interesting part: your custom types can implement the same interfaces. Once they do, Clojure's standard library treats them as first-class citizens. get, seq, map, filter, count all work transparently, no special dispatch, no wrapper functions.
The lasagna-pattern collection library (Clojars) does exactly this. It defines a Collection type backed by a database, then implements ILookup and Seqable so that (get coll {:post/id 3}) triggers a database query while looking like a plain map lookup to the caller. The companion article, Building a Pure Data API with Lasagna Pattern, covers the full architecture. This article focuses on the Clojure constructs that make it work.
The four constructs
Clojure provides four ways to define types that implement protocols and interfaces. Each serves a different purpose.
defprotocol: the contract
Defines method signatures with no implementation. Conceptually similar to a Java interface.
This says: "any storage backend must support these 5 operations." It does not say how. The implementation is left to the types that satisfy the protocol.
defrecord: named, map-like type
A concrete implementation of a protocol. Has named fields and behaves like a Clojure map (you can assoc, dissoc, and destructure it).
Use deftype when you need to override built-in Clojure verbs (get, seq, count). The type itself is opaque. Callers interact with it through standard Clojure functions, not through field access.
reify: anonymous, inline type
Same capability as deftype but anonymous and created inline. Closes over local variables.
(defn profile-lookup [session]
(reify clojure.lang.ILookup
(valAt [this k] (.valAt this k nil))
(valAt [_ k nf]
(case k
:name (:user-name session)
:email (:user-email session)
nf))))
Use for one-off objects, per-request wrappers, or cases where a named type would be overkill. The session value is captured from the enclosing scope.
Summary table
Construct
What it is
When to use
defprotocol
Contract (method signatures)
Define a role: "what must a DataSource do?"
defrecord
Named type, map-like
Concrete implementations: PostsDataSource
deftype
Named type, not map-like
Structural wrappers: Collection
reify
Anonymous inline type
One-off objects: per-request lookups
Overriding built-in verbs
Each Clojure interface corresponds to a built-in verb. Implementing an interface teaches Clojure how your custom type responds to that verb.
ILookup: powers get
When you call (get thing key), Clojure calls (.valAt thing key nil) under the hood. Maps implement this by default. Custom types do not.
;; Without ILookup
(deftype Box [x])
(get (->Box 42) :x) ;; => nil (Box doesn't implement ILookup)
;; With ILookup
(deftype SmartBox [x y]
clojure.lang.ILookup
(valAt [this k] (.valAt this k nil))
(valAt [_ k nf]
(case k :x x :y y nf)))
(get (->SmartBox 1 2) :x) ;; => 1
In the collection library, ILookup is what makes (get coll {:post/id 3}) trigger a database query. The caller writes standard Clojure. The collection translates the get call into a fetch on the underlying DataSource.
Once a type implements Seqable, all sequence functions work: (seq coll), (map f coll), (filter pred coll). The collection becomes iterable by delegating to its DataSource's list-all.
Without Counted, calling count on a custom Seqable type throws UnsupportedOperationException. Clojure's RT.count() does not fall back to seq. It only works on types that implement Counted, IPersistentCollection, java.util.Collection, or a few other JDK interfaces. If your custom type needs to support count, implement Counted explicitly. This also lets you provide an optimized path (e.g., a SELECT COUNT(*) instead of fetching all rows).
Custom protocols
The interfaces above override Clojure's built-in verbs. But some operations have no built-in verb. The collection library defines two custom protocols for these cases.
Serialize for HTTP transport: collections become vectors, lookups become maps or nil
mutate! unifies create, update, and delete into a single function. The operation is determined by the combination of arguments: nil query means create, nil value means delete, both present means update.
Wireable is conceptually similar to clojure.core.protocols/Datafiable (datafy). Both turn opaque types into plain Clojure data. The difference is intent: datafy is for introspection and navigation, ->wire is specifically for HTTP serialization.
The decorator pattern
Here is the key design insight: one DataSource, one Collection, multiple wrappers per role.
Without wrappers (bad)
;; 3 records duplicating the same Datahike queries
(defrecord GuestPostsDataSource [conn] ...)
(defrecord MemberPostsDataSource [conn] ...)
(defrecord AdminPostsDataSource [conn] ...)
Each record contains a full copy of the same fetch, list-all, create!, update!, and delete! logic. Domain logic changes must be applied to all three.
With wrappers (good)
(def posts (db/posts conn)) ;; one DataSource, one Collection
(public-posts posts) ;; reify: override get/seq to strip email
(member-posts posts user-id email) ;; wrap-mutable: override mutate! for ownership
posts ;; admin: no wrapper needed
The DataSource is created once. Each role gets a thin wrapper that overrides only the behavior it needs. Reads, storage queries, and domain logic live in one place.
Wrapper functions
Wrapper
What it overrides
Use case
coll/read-only
Removes Mutable entirely
Guest access (no writes)
coll/wrap-mutable
Overrides mutate!, delegates reads
Ownership enforcement
reify (manual)
Override any interface
Transform read results, composite routing
coll/lookup
Provides ILookup from a keyword-value map
Non-enumerable resources (profile, session data)
Example: building views per role
(let [posts (db/posts conn)] ;; one record, created once
{:guest {:posts (public-posts posts)} ;; reify over read-only, strips :user/email
:member {:posts (member-posts posts uid email)} ;; wrap-mutable, ownership checks
:admin {:posts posts} ;; raw collection, full access
:owner {:users (coll/read-only (db/users conn))}})
Guests see a read-only view with PII stripped. Members see a mutable view that enforces ownership. Admins see the raw collection. Each wrapper does one thing.
The public-posts wrapper demonstrates how reify serves as the escape hatch when the built-in wrappers are not enough:
The library provides read-only (restricts writes) and wrap-mutable (intercepts writes), but no built-in way to transform read results. For that, you implement ILookup and Seqable directly via reify.
Three layers of authorization
Authorization in this pattern is distributed structurally rather than imperatively. Instead of a single middleware that checks permissions, three layers each handle a different granularity.
Coarse: with-role (API map structure)
Binary gate: you have the role or you don't. The entire subtree of collections is present or replaced with an error map.
(defn- with-role [session role data]
(if (contains? (:roles session) role)
data
{:error {:type :forbidden :message (str "Role " role " required")}}))
;; In make-api:
:owner (with-role session :owner
{:users users, :users/roles roles})
A non-owner sending '{:owner {:users ?all}} hits the error map, not the collection. The remote/ layer detects errors along variable paths, so the error flows through as inline data and prevents any mutation from being attempted.
A planned improvement is an error-gate function that replaces the plain map with a reify implementing ILookup (returns self for any key, so deeply nested pattern traversal keeps working), Mutable (returns the error for mutations), and Wireable (serializes as the error map). This would be a good example of composing three protocols into a single anonymous sentinel object.
Medium: wrap-mutable (per-entity mutation rules)
Controls who can create, update, or delete specific entities:
(public-posts posts) ;; strips :user/email from author on every read
Every get and seq call on this wrapper runs through a transformation function that removes sensitive fields before the data reaches the caller.
Authorization summary
Layer
Tool
What it guards
Example
Coarse
with-role
"Can you access :owner at all?"
Non-owners get error map
Medium
wrap-mutable
"Can you mutate this entity?"
Members can only edit own posts
Fine
reify decorator
"What fields can you see?"
Guests don't see author email
The DataSource stays "dumb" about authorization. It only knows about storage. This keeps it reusable across all roles without conditional logic.
When to skip the full stack
Not everything needs defrecord + DataSource + Collection. If a resource is read-only, non-enumerable, and has a single query shape, a raw reify implementing ILookup + Wireable is enough.
Example: a post history lookup that takes a post ID and returns the revision history:
The pattern engine still calls get on it, so it works identically from the caller's perspective. The full DataSource/Collection stack would add index validation, Seqable, Mutable, none of which history needs.
Decision guide
Need
Tool
Full CRUD + enumeration + index validation
defrecord + coll/collection
Read-only, keyword keys, flat values
coll/lookup
Read-only, map keys, single query shape
Raw reify with ILookup + Wireable
Note: coll/lookup only supports keyword keys (:email, :name). For map keys like {:post/id 3}, use a raw reify.
{align=right loading=lazy style="width:240px"}
{align=right loading=lazy}
-
Poliqarp’s GUI, running a query
Smyrna 0.1
