Babashka tasks with automatic help and completions

{:tasks
 {dev   {:doc "Start the dev system"
         :task (clojure "-M:dev")}

  test  {:doc "Run the tests"
         :task (clojure "-M:test")}

  build {:doc "Build an uberjar"
         :depends [test]
         :task (clojure "-T:build uber")}}}

$ bb tasks
The following tasks are available:

dev    Start the dev system
test   Run the tests
build  Build an uberjar

{:paths ["bb"]
 :tasks {dev {:doc "Start the dev system"
              :task (exec &apostasks/dev)}}}

(ns tasks
  (:require [babashka.tasks :refer [clojure]]))

(defn dev
  {:org.babashka/cli {:spec {:port {:coerce :int
                                    :default 8080}}}}
  [opts]
  (clojure "-X:dev" opts))

$ bb dev
dev system on port 8080

$ bb dev --port 3000
dev system on port 3000

$ bb -x tasks/dev
dev system on port 8080

$ bb -x tasks/dev --port 3000
dev system on port 3000

{:paths ["bb"]
 :tasks {dev {:exec-fn tasks/dev}}}

(ns tasks
  (:require [babashka.tasks :refer [clojure]]))

(defn dev
  "Start the dev system"
  {:org.babashka/cli {:spec {:port {:coerce :int
                                    :default 8080
                                    :desc "HTTP port"}}}}
  [opts]
  (clojure "-X:dev" opts))

$ bb dev --help
Usage: bb dev [options]

Start the dev system

Options:
      --port  HTTP port (default: 8080)
  -h, --help  Show this help

$ bb tasks
The following tasks are available:

dev  Start the dev system

$ bb tasks
The following tasks are available:

dev                              Start the dev system
dev:with-transactor              Start the dev system with a transactor
dev:with-transactor:with-nrepl   Start the dev system with a transactor and nREPL

(defn dev
  "Start the dev system"
  {:org.babashka/cli {:spec {:port       {:coerce :int
                                          :default 8080
                                          :desc "HTTP port"}
                             :transactor {:coerce :boolean
                                          :desc "Start a transactor"}
                             :nrepl      {:coerce :boolean
                                          :desc "Start an nREPL server"}}}}
  [opts]
  (clojure "-X:dev" opts))

$ bb dev --help
Usage: bb dev [options]

Start the dev system

Options:
      --port        HTTP port (default: 8080)
      --transactor  Start a transactor
      --nrepl       Start an nREPL server
  -h, --help        Show this help

$ bb dev --transactor
dev system on port 8080 transactor=true nrepl=false

source <(bb org.babashka.cli/completions snippet --shell zsh)

$ bb <TAB>
dev  -- Start the dev system

$ bb dev <TAB>
--help  -h  -- Show this help
--port      -- HTTP port

$ bb org.babashka.cli/completions complete --shell zsh -- dev &apos&apos
--port	HTTP port
--help	Show this help
-h	Show this help

(defn dev
  "Start the dev system"
  {:org.babashka/cli {:spec {:port {:coerce :int
                                    :default 8080
                                    :desc "HTTP port"}
                             :env  {:desc "Environment"
                                    :enum ["dev" "staging" "prod"]
                                    :default "dev"}}}}
  [opts]
  (clojure "-X:dev" opts))

$ bb dev --help
Usage: bb dev [options]

Start the dev system

Options:
      --port  HTTP port (default: 8080)
      --env   Environment (one of: dev, staging, prod) (default: dev)
  -h, --help  Show this help

$ bb dev --env qa
Error: Invalid value for option --env: qa. Expected one of: dev, staging, prod

Usage: bb dev [options]

Run "bb dev --help" for more information.

$ bb dev --env <TAB>
dev  prod  staging

{:paths ["bb"]
 :tasks {dev {:exec-fn tasks/dev}

         db  {:doc "Manage the database"
              :cmd {"migrate" {:exec-fn tasks/db-migrate}
                    "seed"    {:exec-fn tasks/db-seed}}}}}

(defn db-migrate
  "Run pending migrations"
  {:org.babashka/cli {:spec {:env {:desc "Environment"
                                   :enum envs
                                   :default "dev"}}}}
  [{:keys [env]}]
  (println "migrating" env))

(defn db-seed
  "Seed the database with fixtures"
  {:org.babashka/cli {:spec {:env {:desc "Environment"
                                   :enum envs
                                   :default "dev"}}}}
  [{:keys [env]}]
  (println "seeding" env))

$ bb db
No command given.

$ bb db --help
Usage: bb db [options] <command>

Manage the database

Commands:
  migrate  Run pending migrations
  seed     Seed the database with fixtures

Options:
  -h, --help  Show this help

Run "bb db <command> --help" for more information on a command.

$ bb db <TAB>
migrate  -- Run pending migrations
seed     -- Seed the database with fixtures

$ bb db migrate --env <TAB>
dev  prod  staging

{:paths ["bb"]
 :tasks {dev {:exec-fn tasks/dev}

         db  {:doc "Manage the database"
              :cli {:epilog "Migration code is in resources/migrations."}
              :cmd {"migrate" {:exec-fn tasks/db-migrate}
                    "seed"    {:exec-fn tasks/db-seed}}}}}

$ bb db --help
Usage: bb db [options] <command>

Manage the database

Commands:
  migrate  Run pending migrations
  seed     Seed the database with fixtures

Options:
  -h, --help  Show this help

Run "bb db <command> --help" for more information on a command.

Migration code is in resources/migrations.

{:paths ["bb"]
 :tasks {dev {:exec-fn tasks/dev}

         db  {:doc "Manage the database"
              :cli tasks/db-cli
              :cmd {"migrate" {:exec-fn tasks/db-migrate}
                    "seed"    {:exec-fn tasks/db-seed}}}}}

(defn- report-error
  [{:keys [msg]}]
  (binding [*out* *err*]
    (println "db:" msg))
  (System/exit 1))

(def db-cli
  {:epilog "Migration code is in resources/migrations."
   :error-fn report-error})

$ bb db migrate --env qa
db: Invalid value for option --env: qa. Expected one of: dev, staging, prod

{:paths ["bb"]
 :tasks {:cli tasks/db-cli
         ...}}

db {:doc ["Manage the database"
          "Migrations are applied in order and are idempotent."]
    :cmd {"migrate" {:exec-fn tasks/db-migrate}
          "seed"    {:exec-fn tasks/db-seed}}}

$ bb db --help
Usage: bb db [options] <command>

Manage the database
Migrations are applied in order and are idempotent.

Commands:
  migrate  Run pending migrations
  seed     Seed the database with fixtures

Options:
  -h, --help  Show this help

Run "bb db <command> --help" for more information on a command.

$ bb tasks
The following tasks are available:

dev  Start the dev system
db   Manage the database

org.babashka/cli {:mvn/version "0.12.85"}

Permalink

Closing the Find-Usages Gap in CIDER

Next up in the series on the notable changes in CIDER 2.0: cross-references. Or, as most people call the feature, “find usages” - for years the most commonly cited reason to run clojure-lsp alongside (or instead of) CIDER. Let’s talk about why that gap existed and how we finally closed it.

Why runtime xref wasn’t enough

CIDER has had runtime cross-referencing for a while: the cider/fn-refs op walks the loaded vars in your REPL and reports which functions reference the one at point. It’s a genuinely cool trick - the REPL literally knows your program - but as a “find usages” answer it has three structural problems:

• It only sees loaded code. Namespaces you haven’t required yet - often most of the codebase - are invisible.
• It reports functions, not occurrences. Each hit points at the caller’s definition, not the exact call site, and a function that calls yours three times shows up once.
• It’s JVM-only, so ClojureScript users got nothing.

clojure-lsp, by contrast, builds a static index of your whole project with clj-kondo’s analyzer and answers instantly, loaded or not. For occurrence-oriented questions (“show me every place this is used, so I can change all of them”), static analysis is simply the right tool. No amount of runtime cleverness fixes “the code isn’t loaded”.

The fix: search the source

So CIDER 2.0 does the obvious thing we should have done years ago: xref-find-references (M-?) now finds references by searching the project’s source files on disk. Unloaded code, cljs files, commented-out drafts - if the name occurs in the project, you’ll see the exact occurrence, in the standard xref UI you already use for everything else in Emacs. To borrow the franchise that has been handing programmers debugging metaphors for over two decades now: the runtime is the Matrix, a tidy compiled illusion of your program, and to see every place a thing is really used you sometimes have to unplug and look at the source itself.¹

Here’s the difference in one picture - the same query on orchard.misc/require-and-resolve, first in runtime mode, then in source mode:

The runtime knows about three callers - and points you at each caller’s definition. The source scan turns up all ten actual occurrences, across five files, including the ones written as misc/require-and-resolve in namespaces the REPL never loaded. That’s the gap, in one screenshot.

Now, “search the source” makes it sound like a grep, and I want to be clear that it isn’t a dumb one. Say you’re chasing orchard.misc/require-and-resolve. The search runs in three stages:

1. First CIDER asks the REPL to resolve the symbol at point to its fully qualified name, orchard.misc/require-and-resolve. The REPL is right there on the other end of the wire, so why guess when you can ask?
2. A fast first pass (ripgrep, via Emacs’ own xref-matches-in-files) finds every file that so much as mentions require-and-resolve, purely to narrow the field.
3. Then the Clojure-aware part. For each of those files, CIDER reads its (ns ...) form to learn how that file pulls in orchard.misc - aliased as [orchard.misc :as misc], brought in with :refer [require-and-resolve], or is this orchard.misc itself? - and builds a regexp that matches only the forms that file could legitimately use: the qualified orchard.misc/require-and-resolve, the aliased misc/require-and-resolve, or a bare require-and-resolve where the namespace declaration makes that valid. The requires inside the ns form are excluded, so the import line doesn’t show up as a “usage”.²

Is any of this as smart as clj-kondo’s full analysis? No - it’s ultimately a syntactic search, so an identically named var in another namespace can still sneak through as a false positive. But because it’s ns-aware rather than a blind text match, a bare require-and-resolve in some file that never requires orchard.misc (and has a require-and-resolve of its own) won’t be mistaken for yours. For the daily “where is this used?” question it turns out to be remarkably close to the real thing in practice, it requires zero extra infrastructure, and it composes with what only CIDER has: the running REPL.

That composition is configurable via cider-xref-references-mode:

• source (the default) - occurrences from the project’s files.
• runtime - the historical loaded-vars behavior.
• both - source occurrences first, plus the runtime hits the scan can’t see. And there are such hits: references generated by macro expansion leave no textual trace in your source, but the runtime knows about them. Static analysis can’t ever tell you those; your REPL can.

(There’s also cider-xref-fn-refs-in-source, C-c C-? s, when you want the source search explicitly, and outside a project the source mode gracefully falls back to the runtime search.)

Beyond find usages: the who-* family

While closing the gap, we went further and built out a whole family of SLIME-inspired cross-referencing commands under C-c C-w, most of them rendered as expandable trees:

• cider-who-calls / cider-who-is-called - the call graph, upward and downward. Expand a caller to see its callers; spelunk as deep as you like:

  

• cider-who-implements - a protocol’s implementing types (inline defrecord/deftype implementations included) or a multimethod’s dispatch values, each jumping to the implementation’s source. Multimethods are a nice case study in hybrid thinking: the method functions carry no source metadata at runtime, so CIDER locates the defmethod forms by - you guessed it - searching the source.
• cider-type-protocols / cider-protocols-with-method - the reverse lookups: what does this type implement, and which protocols declare this method?
• cider-who-macroexpands - a macro’s use sites, found via source search, because macro invocations are expanded away at compile time and the runtime literally cannot see them.

Much of this is powered by new ops in cider-nrepl (and Orchard underneath), and much of the inspiration came straight from SLIME and swank-clojure, which offered who-calls back when Clojure itself was barely out of the crib. And here’s the part I love: the whole “go read the source instead of trusting the runtime” instinct was already there in swank-clojure’s implementation. Its who-calls would find candidate callers among the loaded vars, sure, but then it went and read the actual source form of each one off disk and walked the parsed code looking for your symbol, rather than believing whatever the compiled runtime claimed. It was still anchored to loaded code - it never scanned the whole project the way CIDER 2.0 does - but the core idea, that the source on disk is the ground truth and the runtime is just a convenient approximation, predates this release by about fifteen years. Good ideas don’t expire. Sometimes nothing beats revisiting the classics.³

So do you still need clojure-lsp?

If you were running clojure-lsp primarily for find-usages - the most common answer I heard when I asked - then CIDER now covers you out of the box. If you use it for project-wide renames, unused-var linting, or editing without a REPL, carry on; those are real strengths of static analysis and CIDER doesn’t try to replicate them. The two continue to work fine side by side, and the new async eldoc even yields politely so LSP-provided docs can compose with CIDER’s.

My goal was never to “beat” clojure-lsp - it was to make a freshly installed CIDER answer the questions every Clojure programmer asks a dozen times a day, with no extra moving parts, and with the one advantage nobody else has: a live runtime on the other end of the wire.

The full story is in the navigation docs. Keep hacking!

Permalink

Clojure vars - notes

Intro

• named references (symbol) to values of fn
• they live in a namespace
• mutable reference to an immutable value

Var is separate from the value so you can redefine it

Note: symbol is just name

Dig Deeper

Functions are actually stored in vars

(defn greet [name] (str "hello" name))
; is equal to 
(def greet (fn [name] (str "hello" name)))
; because defn is macro

Dynamic vars

Allows temporary thread-local bindings

(def ^:dynamic *debug* false)

; temporarily change it
(binding [*debug* true]
 (prinlnt *debug*))

Metadata

(def ^{:doc "A counter"} counter 0)

Symbol vs Var

Symbol

Pice of code that names something

(type 'x)
;; => clojure.lang.Symbol

Writen as 'x
Exists in code and data

Var

Runtime object that point to immutable value
Written as #'x or (var x)
Exists at runtime

Symbol ──▶ Var ──▶ Value
x ──▶ #'user/x ──▶ 42

Example explained

(def x 43)
x
; 43

In the code

1. find the symbol
2. resolve it to var #'user/x
3. dereference var
4. produce 42

Dig Deeper

Note symbol can refere to local binding (not a Var) or local variable

; local binding
(let [x 10]
  x)

; local variable
(fn [x] x) 

Immutability

Vars are mutable
The values they point are (usually immutable)

The var can change what it points to the data/vale cannot change

Imagine a program without vars
we have immutable values but no way to give them names

Vars and namespaces

Namespace is mapping from symbols to vars
Namespace is object that contains vars

(ns math)
; internaly
; pi --> var
; inc --> var

Referring to another namespace

(ns animals)
(def dog "Labrador")

(ns zoo
  (:require [animals]))

animals/dog ;; 


(ns zoo-2
  (:require [animals :as a])) ; aliases

a/dog ; shorthand for animals/dog


(ns zoo-2
  (:require [animals :refer [dog]])) ; refering vars

dog ; shorthland for a/dog

Dereferencing

Var fully qualified name includes namespace: math/pi or user/x

Example

(def x 42)

When evaluating steps

1. find var
2. deref var
3. 42

you can also do dereferencing by:

@#'x 
; or
(deref #'x)

Evaluation pipeline simplify

Examples

(double-it x)

1. Soruce code contains symbols (double-it x)
   Symbols is name ('x, it has type clojure.lang.Symbol)

2. Compiler resolve namespace
   What does the symbol x refer to in the current namespace
   So we get namespace-name/x var
   Note: namespace dosen't store values it stores vars

3. Var is runtime object so it dereferences it to get values

4. Invocation (of the function or the value)

Why we have vars, what not directly have values, because with vars we have

• redefintion (REPL)
• metadata
• dynaming binding

Dig Deeper

Special Symbols

' — Quote

Dont evaluate this treat it as data

Example

(def x 42)

This means

1. Resolve symbol
2. Find the var
3. Return it value

'x
; => x

This means

1. Return symbol

#' — Var Quote

Resolve this symbol to its Var

Example

(def x 42)
#'x
;; => #'user/x

The result is not 42
the result is the Var object
To get a value you can dereference it using @

@#'x
;; => 42
; or

(deref #'x)

When to use it

• Metadata is attached to the Var
• Testing and mocking
• Implementing macros
• Dynamic Vars and advanced runtime features

Permalink

AI is ruining self-learning

Permalink

Modernizing CIDER’s Completion

CIDER’s code completion has quietly gotten quite good over the years, and I don’t think it gets enough credit. It’s all built on Emacs’s standard completion-at-point, so it works with whatever completion UI you prefer - the built-in one, Corfu, company - without any special setup. Under the hood compliment does the heavy lifting for Clojure (and clj-suitable for ClojureScript), which means smart, backend-driven matching: mai completes to map-indexed, cji to clojure.java.io, and an unimported BiFun to java.util.function.BiFunction. The candidates come back ranked by the backend and are context-aware - it knows when you’re inside a -> or completing a deftype field.

Lately I’ve been giving the Emacs side of things some attention, to bring it in line with the modern completion stack so many of us now use - Vertico, Corfu, Consult, Marginalia and friends. Two changes are worth calling out.

The first is about the symbol prompts. A number of CIDER commands ask you for a Clojure symbol when there’s nothing at point - cider-doc, cider-find-var and the like. Historically those prompts used the older completion machinery, so your completing-read UI didn’t kick in and you were left with TAB and a *Completions* buffer. There’s now cider-use-completing-read-for-symbol (off by default for now); turn it on and those prompts go through completing-read over a collection that queries the running REPL lazily as you type. Vertico, Ivy, Helm - whatever you drive completing-read with - just works, and the candidates carry their type and namespace.

The second is smaller, but I like it a lot: annotations now line up in a proper column instead of trailing raggedly after each candidate.

This comes from an affixation-function, the richer successor to the old annotation-function, so every frontend that understands it - the built-in *Completions*, Corfu, Vertico - renders the aligned version. company keeps showing its own trailing annotations, same as before.

Both changes are in the latest CIDER MELPA build and will ship in the next stable release. As always, I’d love to hear how they work out for you.

That’s all I have for you today. Keep hacking!

P.S. If you use Embark, I wrote up a fun way to act on Clojure symbols with it - documentation, jump-to-definition, inspect and so on - over on Emacs Redux.

Permalink

AI agents, Haskell, and the stubbornness of developers

Recently, I found an article about a member of the Haskell Foundation – who was moving his company away from Haskell. The company built a hardware product with Haskell, and they were moving away reluctantly.

I’m not going to repeat the whole thing here (you can read it yourself, but the TL;DR; is: the Haskell compiler and tooling ecosystem was too slow, and that became a bottleneck now that the LLM agent writing the code is so fast.

Now, let’s be clear: I don’t want to start a holy war about “static versus dynamic typing”, so I’m not going to do that. I’m not going to discuss if the merits of static typing are overrated or not, and I’m not going to discuss if dynamic languages are more or less flexible. This was discussed over and over again – and nobody knows the result. And for me, that in-determination is also a result.

What I am going to talk about is the stubbornness of software developers. It’s something that’s been bothering me for a couple of years now, and it still bothers me today. Looking at this article was no different.

The bottleneck is the compiler

So, again, the big TL;DR; of the article: the Haskell tooling is too slow, and the agents are really fast to write code.

If an agent produces working code in, say, 20 seconds, and the Haskell compiler takes 5, 10, 15 minutes to compile… then the compiler becomes a huge bottleneck in the loop where the agent writes code, tests it, sees if it works, and rewrites it to iterate. The agent is done, and the Haskell compiler still didn’t finish.

And here’s the part that got my attention: the author moved from Haskell to Python.

This seems like a weird move to me, because they’re moving from a static language to a dynamic one. But the thing is – it also matches my own experience (I also started with some static typed language, and moved to Ruby/Clojure). And here’s the stubbornness part: this is something that people who like dynamic languages (like me) have been saying for literal years – He experienced something that we iterate, and explain, and prove, again and again and people are still fighting to believe us.

It’s not the LLM

So what’s different right now? It’s not the LLM.

Even supposing the LLM could write excellent code – or even good code (which, to be honest, has not been my experience – It’s mostly code that works, but that I would not write: full of defensive coding, and sometimes even duplications), the real difference between an LLM and a human is: a human programmer is usually proficient in 3, 4, 5 languages, and plenty of programmers are proficient in one or two at most.

The agent, on the other hand, is proficient in multiple languages, and it writes with the same code quality in all of them. Be it good or bad, it’s the same. And that is the interesting part – you now have a mythical “person” (a mythical robot, maybe) – that can produce, with the same ease and the same quality, code in both Haskell and Python: which means you can finally compare the two while holding the developer constant.

And what they found is that the number of defects in the Python code, over a given timeline, was the same as the number of defects in the Haskell code over that same timeline. (Probably fewer in Python, actually – otherwise why would they be moving away from Haskell? But I’m not going to guess). Uhnnn… who could have predicted, except the academic study that studied that in 2016, and that ANOTHER study in 2019 found the same conclusion?

What about quality?

So here’s what makes a good quality software: it’s the developer. And in this case, the machine is the same developer producing the same quality code in two languages. Which means they can measure how much the compiler was actually influencing the time it takes to write something – and what they found is that the compiler wasn’t helping them as much as you’d expect.

I am not going to be that guy and say that the compiler wasn’t helping at all – maybe it was – but the actual time to produce code of quality X was the same in both a compiled, static language like Haskell and a dynamic, interpreted one like Python.

And this, again, matches my experience.

The number of hours that I saw people spend producing correct, and quality code in C# and Java was the same – if not higher – than the time I saw people doing quality code in Ruby. Yes, sure: if you did a very bad Ruby implementation, it was very, very hard to read. But I also saw very, very hard to read C# and Java code. The type system helped me understand what the code was doing – but it did not help me write the fixed version because, for that, I usually had to change the type of some function… and changing that type would cascaded compilation errors over and over the codebase, forcing me either to write “adapter”, or “facades” or other techniques to fix the issue (again, structural typing might help but not by much – you can have the proliferation of very small, specific types that you need to merge and split, essentially making the whole thing a duck-typed code anyway)

Also, if I am writing new types, cascading it over the code, can I actually guarantee the same code quality? After all, am I not changing the shape of the data I’m sending, essentially invalidating the whole “this data have this structure and we’re sure of it”?

Dehumanization, again

You know what bothers me? I met people that worked in Haskell. Most of them would not accept, at all, any argument for a language with “weaker typing” (I wrote how that term is meaningless in the past, so I’m also using it in a meaningless way, just to be 100% sure) than Haskell – some would argue that we need more types and that it’s impossible to guarantee any quality in languages Clojure (because it’s dynamic) and even worse in Ruby or Python (because they are imperative). Few people actually wanted to listen to a different approach, to listen to the “Clojure guy” (usually that was me, before I quit most of the groups because they ended up discussing into theorems, category algebra, etc and less about software) about some different approach to solve a problem.

But… apparently, a machine is ok.

And the worst part? It’s the honeymoon phase. They are doing what LLMs are very good at – capturing a code that already exists, that have very well-define semantics, and porting that to another language. The LLM will probably be able to solve some issues on that migration, and probably will offer a good path for the future.

And that will be attributed to the machine – not the humans that wrote the first Haskell code, well-written, in the first place.

Exactly like they listened to the machine saying that “A dynamic language can produce code faster, and we can iterate sooner on that result” than us, developers of said language saying the exact same thing over the years.

Permalink

Things I wish Datomic had: Map values

(def prog
  {:program/name    "Apache Maven"
   :program/url     "https://maven.apache.org"
   :program/version {:version/major 3
                     :version/minor 9
                     :version/patch 16}})

(def schema
  [{:db/ident :program/name
    :db/valueType :db.type/string
    :db/unique :db.unique/identity
    :db/cardinality :db.cardinality/one}
   {:db/ident :program/url
    :db/valueType :db.type/uri
    :db/cardinality :db.cardinality/one}
   {:db/ident :version/major
    :db/valueType :db.type/long
    :db/cardinality :db.cardinality/one}
   {:db/ident :version/minor
    :db/valueType :db.type/long
    :db/cardinality :db.cardinality/one}
   {:db/ident :version/patch
    :db/valueType :db.type/long
    :db/cardinality :db.cardinality/one}
   {:db/ident :program/version
    :db/valueType :db.type/ref
    :db/isComponent true
    :db/cardinality :db.cardinality/one}])

@(d/transact conn schema)
@(d/transact conn [prog])

(let [db (d/db conn)
      mvn (d/entity db [:program/name "Apache Maven"])]
  (:program/version (d/touch mvn)))
;=> {:db/id 17592186045419, :version/major 3, :version/minor 9, :version/patch 16}

@(d/transact conn
             [[[:program/name "Apache Maven"]
               :program/version
               #:version{:major 3, :minor 9, :patch 17}]])

(let [db (d/db conn)
      mvn (d/entity db [:program/name "Apache Maven"])]
  @(d/transact conn [[(-> mvn :program/version :db/id) :version/major 4]]))

Permalink

Database adapters in Biff 2

I've released two new Biff libraries, both database adapters: biff.sqlite and biff.xtdb. Both of them implement some interfaces used by various other Biff libraries, and they also both implement additional functionality that can be useful to Clojure apps even if they aren't using Biff.

The interfaces

So far, modifying a Biff app to use a different database than the default has been kind of inconvenient, as a result of philosophy about modularity for Biff 1. The ability to swap out defaults was more of an escape hatch so that starting out with Biff doesn't mean your locked into all its choices forever. So that meant if you wanted to swap out the database, you had to, for example, copy and paste all of Biff's sign-in-via-email code and rewrite the queries.

With Biff 2, modularity is becoming more of a first-class citizen. Hence these new "interfaces." And the main interface here addresses the question of, you know, how do you put things in a database. And get them back out. For Biff 2, I wanted be able to package up shared application functionality that needs persistence (such as the authentication module) in a database-agnostic way.

So Biff 2 now defines a key-value store interface which can be implemented by database adapter libraries like the two I've just released:

• :biff.core/kv-set
• :biff.core/kv-get
• :biff.core/kv-list

And my two implementations: sqlite and xtdb. These KV-store functions are exposed via a biff.core module, for example.

In general I try to avoid adding layers to things unnecessarily, so it took a little thinking to arrive at this solution. My main thought was that I'm primarily interested in database-agnosticism for libraries, not applications. Migrating a single application's database is a different use case than wanting to provide functionality for multiple applications using different databases, and it's the latter use case that I'm trying to address.

So I didn't want to introduce some sort of "Biff query/transaction language" that you'd use whenever writing a Biff app; that would be overkill. The database-agnostic libraries I wanted to write have only basic persistence needs, so a key-value interface is sufficient. And that's an easy interface for database adapters to implement.

The second biggest interface-type-thing is that both adapters come with make-resolvers functions (sqlite / xtdb) which generate a set of biff.graph resolvers for the tables in your application schema.

The features

Besides the key-value functions and a couple other doodads, the remaining functionality in these database libraries is just whatever stuff I thought would be nice to have when writing a Clojure app using the respective databases. From biff.sqlite's README:

• Sane defaults like WAL mode, STRICT tables, etc.
• Backup/restore via Litestream.
• Migrations via sqldef.
• Rich schema types: define columns as e.g. booleans, instants, nested maps, etc; and biff.sqlite converts them to/from SQLite's supported types (ints, blobs, etc).
• Validate transactions based on centralized authorization rules you define (helps to keep LLM code secure).

And for biff.xtdb:

• Start up an in-process node with high-level config defaults.
• Custom :biff/upsert and :biff/assert-unique transaction operations.
• Optionally enforce Malli schemas on write.
• Define centralized authorization rules for validating transactions.

biff.sqlite is a chonkier library due to all the logic needed for supporting rich schema types.

Write your own database adapter

Finally, I have written a database adapter guide which lists all the interfaces that adapters should implement and suggests additional features that may or may not be relevant for any given database. Between that guide and the two sqlite/xtdb reference implementations, I'm hoping it should be straightforward to implement an adapter for whatever database you want to use. I will probably only maintain the SQLite and XTDB adapters, but I might publish some as-is code for other databases which could be picked up and maintained anyone who so chooses.

Plug: my team is hiring for a senior software engineer, writing ClojureScript and Python. We make optimization software for clean energy projects.

Permalink

Python Fundamentals for a JavaScript Developer

I'll guide you through Python fundamentals by comparing concepts with JavaScript. Let's start!

1. Hello World & Basic Syntax

JavaScript

console.log("Hello World");
let x = 5;

Python

print("Hello World")
x = 5  # No semicolon, no let/const

Key Differences:

• No semicolons in Python
• Indentation matters (replaces curly braces)
• Comments use # instead of //

2. Variables & Data Types

JavaScript

let name = "Alice";  // string
let age = 30;        // number
let isStudent = true; // boolean
let scores = [95, 87, 91]; // array
let person = {        // object
    name: "Bob",
    age: 25
};
let nothing = null;
let notDefined = undefined;

Python

name = "Alice"        # str
age = 30              # int (or float for decimals)
is_student = True     # bool (capital T/F)
scores = [95, 87, 91] # list (mutable)
person = {            # dict (dictionary)
    "name": "Bob",
    "age": 25
}
nothing = None        # Python's null/undefined

Key Differences:

• Python uses snake_case (not camelCase)
• True/False capitalized
• None instead of null/undefined
• Lists ≈ Arrays, Dicts ≈ Objects

3. Control Flow

JavaScript

// If-else
if (age >= 18) {
    console.log("Adult");
} else if (age >= 13) {
    console.log("Teen");
} else {
    console.log("Child");
}

// For loop
for (let i = 0; i < 5; i++) {
    console.log(i);
}

// While loop
let count = 0;
while (count < 5) {
    console.log(count);
    count++;
}

Python

# If-else (indentation instead of braces)
if age >= 18:
    print("Adult")
elif age >= 13:  # NOT else if
    print("Teen")
else:
    print("Child")

# For loop (more like for...of in JS)
for i in range(5):  # range(5) = [0, 1, 2, 3, 4]
    print(i)

# Iterate over list (like for...of)
for score in scores:
    print(score)

# While loop
count = 0
while count < 5:
    print(count)
    count += 1  # No ++ operator in Python

4. Functions

JavaScript

// Function declaration
function add(a, b) {
    return a + b;
}

// Arrow function
const multiply = (a, b) => a * b;

// Default parameters
function greet(name = "Guest") {
    return `Hello ${name}`;
}

Python

# Function definition (def instead of function)
def add(a, b):
    return a + b  # Indented body

# Lambda functions ≈ Arrow functions
multiply = lambda a, b: a * b

# Default parameters
def greet(name="Guest"):
    return f"Hello {name}"  # f-strings like template literals

# Multiple return values (tuples)
def get_coordinates():
    return 10, 20  # Returns a tuple (10, 20)

x, y = get_coordinates()  # Destructuring assignment

5. Data Structures Comparison

Arrays/Lists

// JavaScript Arrays
let arr = [1, 2, 3];
arr.push(4);          // [1, 2, 3, 4]
arr.pop();            // [1, 2, 3]
let sliced = arr.slice(0, 2);  // [1, 2]

# Python Lists
arr = [1, 2, 3]
arr.append(4)         # [1, 2, 3, 4]
arr.pop()             # [1, 2, 3] (removes last)
sliced = arr[0:2]     # [1, 2] (slicing syntax)
arr.insert(1, 99)     # [1, 99, 2, 3]

# List comprehension (unique to Python)
squares = [x**2 for x in range(5)]  # [0, 1, 4, 9, 16]

Objects/Dictionaries

// JavaScript Objects
let person = {
    name: "Alice",
    age: 30,
    greet() {
        return `Hello, I'm ${this.name}`;
    }
};
console.log(person.name);
console.log(person["age"]);

# Python Dictionaries
person = {
    "name": "Alice",
    "age": 30,
    "greet": lambda self: f"Hello, I'm {self['name']}"
}
print(person["name"])  # Access with brackets
print(person.get("age"))  # Safer access

# Methods don't naturally have 'this' context
# Usually you'd use classes for that (see below)

6. Classes & OOP

JavaScript (ES6+)

class Person {
    constructor(name, age) {
        this.name = name;
        this.age = age;
    }

    greet() {
        return `Hello, I'm ${this.name}`;
    }

    static species = "Human";
}

const alice = new Person("Alice", 30);

Python

class Person:
    species = "Human"  # Class attribute (static)

    def __init__(self, name, age):  # Constructor
        self.name = name  # Instance attribute
        self.age = age

    def greet(self):  # Methods always have self parameter
        return f"Hello, I'm {self.name}"

    @staticmethod
    def static_method():
        return "This is static"

alice = Person("Alice", 30)
print(alice.greet())  # No parentheses needed for self when calling

7. Modules & Imports

JavaScript (ES6 Modules)

// math.js
export const PI = 3.14159;
export function add(a, b) { return a + b; }

// main.js
import { PI, add } from './math.js';
import * as math from './math.js';

Python

# math.py
PI = 3.14159
def add(a, b):
    return a + b

# main.py
from math import PI, add
import math  # Then use math.PI, math.add
import math as m  # Alias

8. Error Handling

JavaScript

try {
    throw new Error("Something went wrong");
} catch (error) {
    console.error(error.message);
} finally {
    console.log("Cleanup");
}

Python

try:
    raise Exception("Something went wrong")
except Exception as e:  # 'as' instead of variable declaration
    print(f"Error: {e}")
finally:
    print("Cleanup")

9. Async Programming

JavaScript (Promises/Async-Await)

// Promise
fetch('https://api.example.com/data')
    .then(response => response.json())
    .then(data => console.log(data));

// Async/await
async function getData() {
    const response = await fetch(url);
    return await response.json();
}

Python (Async/Await)

import asyncio
import aiohttp  # External library for HTTP

async def fetch_data(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.json()

# Run async function
asyncio.run(fetch_data('https://api.example.com/data'))

10. Common Patterns & Tips

1. Type Checking (Python is dynamically typed but has type hints)

def add(a: int, b: int) -> int:  # Type hints (optional)
    return a + b

2. String Formatting (multiple ways)

name = "Alice"
# f-strings (Python 3.6+, like template literals)
print(f"Hello {name}")

# .format() method
print("Hello {}".format(name))

# % formatting (older style)
print("Hello %s" % name)

3. Tuple vs List

# List - mutable
my_list = [1, 2, 3]
my_list[0] = 99  # OK

# Tuple - immutable
my_tuple = (1, 2, 3)
# my_tuple[0] = 99  # ERROR!

4. Sets (unique unordered collection)

my_set = {1, 2, 3, 3, 2}  # {1, 2, 3} (duplicates removed)
another_set = set([1, 2, 3, 4])  # Alternative creation

Quick Reference Table

Practice Exercise

Convert this JavaScript code to Python:

function filterEvenSquares(numbers) {
    return numbers
        .filter(n => n % 2 === 0)
        .map(n => n ** 2);
}

const result = filterEvenSquares([1, 2, 3, 4, 5]);
console.log(result); // [4, 16]

Python solution:

def filter_even_squares(numbers):
    return [n**2 for n in numbers if n % 2 == 0]

result = filter_even_squares([1, 2, 3, 4, 5])
print(result)  # [4, 16]

Next Steps

1. Install Python and a good IDE (VS Code with Python extension works well)
2. Practice by rewriting your JS projects in Python
3. Explore Python-specific features: decorators, generators, context managers
4. Learn popular libraries:
   • Web: Flask/Django (Express equivalents)
   • Data: NumPy, Pandas
   • AI/ML: TensorFlow, PyTorch

The main mindset shift: Python emphasizes readability and simplicity over cleverness. You'll write less code to accomplish the same tasks!

Permalink

Security incident disclosure — July 2026

Permalink

Looking for work

Hey, Niki here. This is a bit unusual. My sabbatical is coming to an end, and I am looking for a new opportunity. Full-time or contract, startup or research, remote or Berlin, individual contributor, ideally—tight team, ambitious product.

I am a software engineer first and foremost with 20+ years of experience. I work on technically challenging products, foundational technology, dev tools. I’ve been doing Clojure and web recently, but I'm also very excited to explore closer-to-the-metal programming.

I have an eye for design, user interfaces, UX, DX. I would love to work with a team that takes interface quality seriously. Or to work with graphics!

I am pretty sure I am good at explaining stuff, including what we are building, why, why this way, why is it important, etc. For example.

The overarching theme is to understand computers deeply, and then use that to make better and simpler software. If you care about that too, we might be a great match!

Recent work

Instant DB is a US startup building a modern Firebase. I worked on the sync algorithm, performance, DX. A summary of my commit log.

Roam Research is an OG personal knowledge manager. I worked on database optimization and a plugin system.

At JetBrains, I developed a new Skia renderer for Fleet and Jetpack Compose Desktop.

I’ve built many open-source libraries, including a database, a GUI toolkit, a Clojure dev environment, a React wrapper, a well-known font... More recently, Clojure+ gives you a taste of my approach to DX, and Fast EDN—to performance.

I maintain several active projects — AlleKinos.de, Grumpy Website, this site.

If you want to dive deeper, here’s the usual stuff: Projects / Talks / LinkedIn / GitHub

I also made a two-page PDF CV.

Why this post?

It’s an attempt to reach beyond my immediate network. I’ve been doing Clojure for a long time, and now want to explore.

If you are working on a compiler, a database, an IDE, a programming language or another technically ambitious product, touching graphics, typography, algorithms, low-level programming, and you think my experience can help, let’s talk: niki@tonsky.me.

Permalink

Statistics made simple

I have a weird relationship with statistics: on one hand, I try not to look at it too often. Maybe once or twice a year. It’s because analytics is not actionable: what difference does it make if a thousand people saw my article or ten thousand?

I mean, sure, you might try to guess people’s tastes and only write about what’s popular, but that will destroy your soul pretty quickly.

On the other hand, I feel nervous when something is not accounted for, recorded, or saved for future reference. I might not need it now, but what if ten years later I change my mind?

Seeing your readers also helps to know you are not writing into the void. So I really don’t need much, something very basic: the number of readers per day/per article, maybe, would be enough.

Final piece of the puzzle: I self-host my web projects, and I use an old-fashioned web server instead of delegating that task to Nginx.

Static sites are popular and for a good reason: they are fast, lightweight, and fulfil their function. I, on the other hand, might have an unfinished gestalt or two: I want to feel the full power of the computer when serving my web pages, to be able to do fun stuff that is beyond static pages. I need that freedom that comes with a full programming language at your disposal. I want to program my own web server (in Clojure, sorry everybody else).

Existing options

All this led me on a quest for a statistics solution that would uniquely fit my needs. Google Analytics was out: bloated, not privacy-friendly, terrible UX, Google is evil, etc.

What is going on?

Some other JS solution might’ve been possible, but still questionable: SaaS? Paid? Will they be around in 10 years? Self-host? Are their cookies GDPR-compliant? How to count RSS feeds?

Nginx has access logs, so I tried server-side statistics that feed off those (namely, Goatcounter). Easy to set up, but then I needed to create domains for them, manage accounts, monitor the process, and it wasn’t even performant enough on my server/request volume!

My solution

So I ended up building my own. You are welcome to join, if your constraints are similar to mine. This is how it looks:

It’s pretty basic, but does a few things that were important to me.

Setup

Extremely easy to set up. And I mean it as a feature.

Just add our middleware to your Ring stack and get everything automatically: collecting and reporting.

(def app
  (-> routes
    ...
    (ring.middleware.params/wrap-params)
    (ring.middleware.cookies/wrap-cookies)
    ...
    (clj-simple-stats.core/wrap-stats))) ;; <-- just add this

It’s zero setup in the best sense: nothing to configure, nothing to monitor, minimal dependency. It starts to work immediately and doesn’t ask anything from you, ever.

See, you already have your web server, why not reuse all the setup you did for it anyway?

Request types

We distinguish between request types. In my case, I am only interested in live people, so I count them separately from RSS feed requests, favicon requests, redirects, wrong URLs, and bots. Bots are particularly active these days. Gotta get that AI training data from somewhere.

RSS feeds are live people in a sense, so extra work was done to count them properly. Same reader requesting feed.xml 100 times in a day will only count as one request.

Hosted RSS readers often report user count in User-Agent, like this:

Feedly/1.0 (+http://www.feedly.com/fetcher.html; 457 subscribers; like FeedFetcher-Google)

Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 6 subscribers)

Feedbin feed-id:1373711 - 142 subscribers

My personal respect and thank you to everybody on this list. I see you.

Graphs

Visualization is important, and so is choosing the correct graph type. This is wrong:

Continuous line suggests interpolation. It reads like between 1 visit at 5am and 11 visits at 6am there were points with 2, 3, 5, 9 visits in between. Maybe 5.5 visits even! That is not the case.

This is how a semantically correct version of that graph should look:

Some attention was also paid to having reasonable labels on axes. You won’t see something like 117, 234, 10875. We always choose round numbers appropriate to the scale: 100, 200, 500, 1K etc.

Goes without saying that all graphs have the same vertical scale and syncrhonized horizontal scroll.

Insights

We don’t offer much (as I don’t need much), but you can narrow reports down by page, query, referrer, user agent, and any date slice.

Not implemented (yet)

It would be nice to have some insights into “What was this spike caused by?”

Some basic breakdown by country would be nice. I do have IP addresses (for what they are worth), but I need a way to package GeoIP into some reasonable size (under 1 Mb, preferably; some loss of resolution is okay).

Finally, one thing I am really interested in is “Who wrote about me?” I do have referrers, only question is how to separate signal from noise.

Performance. DuckDB is a sport: it compresses data and runs column queries, so storing extra columns per row doesn’t affect query performance. Still, each dashboard hit is a query across the entire database, which at this moment (~3 years of data) sits around 600 MiB. I definitely need to look into building some pre-calculated aggregates.

One day.

How to get

Head to github.com/tonsky/clj-simple-stats and follow the instructions:

Let me know what you think! Is it usable to you? What could be improved?

Permalink

Clojure 1.13.0-alpha5

org.clojure/clojure {:mvn/version "1.13.0-alpha5"}

clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.13.0-alpha5"}}}

Permalink

Datalevin 1.0.0 Is Here: One Database for Application State and Agent Memory

Today, after six years of development, I am thrilled to announce the availability of Datalevin 1.0.0.

We started Datalevin in 2020 with a deceptively simple question: why should SQL databases remain the default center of application state?

Six years of research, engineering, benchmarking, production use, and community feedback later, Datalevin 1.0.0 is our answer. Datalevin is open source under the Eclipse Public License 2.0. It is a durable, high-performance, fact-first database that brings relational queries, graph traversal, logical reasoning, document access, full-text search, and vector search into one compact system.

The release completes the roadmap we set for 1.0: automatic path indexing for documents; write-ahead logging and transaction-log access; read-only replicas and high availability; a JSON API; broad libraries for Clojure, Java, Python, and JavaScript; and much more.

It also arrives with two new ways to learn Datalevin. The new Datalevin website contains the online guide, with examples in Clojure, Java, Python, and JavaScript. The complete book, Datalevin: The Definitive Guide to Logical and Intelligent Databases, is available in print and ebook formats. In addition to the full database guide, the book contains five chapters devoted to persistent memory for intelligent systems.

This is a release, a book, and a website. More importantly, it is the point at which the original Datalevin idea becomes a complete platform.

Replace SQL at the Center

Datalevin is not intended to be one more specialized database sitting beside a SQL system. Its goal is to replace SQL databases at the center of application state.

That does not mean recreating SQL with different syntax. It means replacing the table as the center of gravity with the fact. Datalevin stores small entity-attribute-value facts, or datoms. The same fact can participate in a row-like record, a graph edge, a nested document workflow, a search result, or a logical rule without being copied into a different data model.

Why make such a fundamental change? There are three main arguments.

1. SQL Is an Awkward Application Interface

SQL is a string-shaped language embedded inside programs. It has a large, English-like syntax, many dialects, and poor composition with host-language code. The enormous ecosystems of ORMs, query builders, migration tools, and object mapping layers are not signs that SQL is a natural application interface. They are evidence of how much machinery is needed to make it behave like one.

Datalevin queries are data. Datalog expresses the facts that must be true, while shared variables create joins implicitly. Rules package reusable logic, and recursive rules use the same form as ordinary queries. The programmer describes relationships instead of spelling out a sequence of join mechanics.

This smaller, more regular surface is easier for people to learn and easier for programs to construct. It is also a better target for AI-generated queries: fewer syntactic branches, fewer vendor-specific choices, and less planner-sensitive ceremony.

2. Table-Shaped Storage Makes Complex Query Planning Harder

Rows bundle many individual facts into containers. When data are sparse, skewed, or correlated, a SQL optimizer has a hard time estimating how many rows will survive each predicate and join. Those cardinality estimates often depend on histograms, independence assumptions, and other approximations. A bad estimate can turn a reasonable query into a huge intermediate result.

A fact-first store begins with explicit, independently indexed data items. Missing facts are absent rather than represented by positional NULL values. Datalevin can count and sample the same indexed facts that query execution will use, giving its cost-based optimizer better raw material for planning complex joins.

This is not only a theoretical advantage. In the Join Order Benchmark, Datalevin has demonstrated that a triplestore can outperform PostgreSQL and SQLite on complex relational queries. The same query engine also performs strongly on recursive logic and industry-standard graph workloads.

3. Stacking Extensions Creates an Integration Tax

Modern SQL databases can add JSON, full-text search, vector indexes, graph features, recursive queries, and procedural extensions. Each capability is useful. The trouble starts when one application question needs several of them at once.

Every extension tends to bring its own syntax, operators, index types, cost model, and operational rules. If the capabilities are split into separate services, the application must also synchronize copies of data and reconcile results across network boundaries. Either way, the glue moves into application code.

Datalevin makes these capabilities composable over one database state in the same elegant fact based model. A single query can ask for documents that contain a phrase, are close to a question in embedding space, belong to a particular entity in a graph, satisfy a nested document predicate, and pass exact permission and lifecycle rules. The database does the integration work where it belongs.

A Memory Substrate for AI Agents

That unified model matters even more for AI agents.

An agent needs more than a transcript and more than a vector database. It needs durable episodes, structured facts, goals, tasks, permissions, tool results, source documents, relationships, and a bounded view of what matters now. It needs to recall by similarity, but it must also know which facts are current, which source supports them, who is allowed to see them, and what they are connected to.

Datalevin is designed to be the memory substrate underneath that system:

• Full-text search recalls information by words, phrases, and boolean search expressions.
• Vector and embedding search recalls information by semantic similarity.
• Logical access uses Datalog queries and rules to enforce exact conditions, derive facts, and reason recursively.
• Graph access follows relationships among users, episodes, facts, goals, tasks, evidence, and documents.
• Document access keeps nested EDN, JSON, and Markdown values intact while automatically indexing their paths.
• Relational access joins structured application state without giving up the fact-first model.

These are not six disconnected products. They are six ways to see and retrieve the same durable fact based state.

That distinction is crucial. Similarity search can find plausible memories, but similarity alone cannot decide whether a fact is authorized, supported, superseded, or relevant to the active goal. Datalevin lets vector and full-text recall produce candidates, then lets logic, graph relationships, document predicates, and ordinary joins constrain and explain the result.

Datalevin does not try to be an agent runtime. The application still owns model calls, tool authorization, ingestion policy, consolidation, truth maintenance, and prompt assembly. Datalevin provides the durable, transactional environment in which those decisions can be stored, inspected, queried, and resumed. Its built-in MCP server can also expose this memory directly to MCP-compatible AI tools.

In other words, a context window is temporary attention. Datalevin is memory.

One Database, Almost the Same API Everywhere

Datalevin began as a Clojure library, but 1.0 is not limited to Clojure applications. The Clojure, Java, Python, and JavaScript APIs now cover almost the same public surface:

The remaining differences are small and explicit. JavaScript does not expose the Datalog transaction callback because callback re-entry through the Node/JVM bridge can deadlock. Staged mutation of an existing entity object remains a Clojure-only convenience; Java, Python, and JavaScript use transaction maps or builders instead. The full, current list lives in the language compatibility matrix.

Whatever language you choose, the important parts do not change: the same facts, schema, transactions, Datalog queries, and indexes.

Embedded, Server, or Script: Choose at Deployment Time

The data model should not have to change when the deployment topology changes. Datalevin therefore supports three primary ways to run:

There is also an MCP server mode for local AI-tool integration. You can begin with an embedded prototype, move to a shared server as the application grows, and automate it from scripts without rewriting the data model or query language.

Deployment changes. The facts do not.

Start Building

Datalevin 1.0.0 is available now:

• Read the new guide at datalevin.org.
• Get the complete print or ebook edition of Datalevin: The Definitive Guide.
• Explore, use, and contribute to the open-source code on GitHub.
• Use Datalevin from Clojure, Java, Python, or JavaScript.

Reaching 1.0 took six years because the goal was never merely to ship another query language or another storage wrapper. The goal was to build one coherent place for application state: simple enough to embed, serious enough to run as a server, expressive enough for relational, graph, document, and logical work, and intelligent enough to become durable memory for the next generation of AI systems.

Thank you to everyone who tested Datalevin, reported issues, contributed code, shared benchmarks, trusted it in production, or simply asked hard questions. You helped turn an ambitious idea into a 1.0 database.

Datalevin 1.0.0 is here. Let us build applications and agents that remember.

Permalink

Maybe not microservice: The Case for Pipes, Pipelines, and Functional Isolation

1. Subsystem Decomposition

1.1 The Decomposition Problem

A subsystem decomposes a codebase into smaller, cohesive units. Two primary axes of decomposition exist:

• Technical axis: grouping by component type (controller, service, model, view)
• Functional axis: grouping by business capability (cataloguing, circulation, etc.)

1.2 Tension Between Framework Prescriptions and Decomposition Strategy

Organizing top-level subsystems functionally may create friction with frameworks that prescribe a technical-first structure. Concrete examples:

• Rails enforces model, view, and controller directories at the root level, making functional decomposition awkward without additional mechanisms like Rails Engines
• Sinatra (a microframework) imposes minimal structure, leaving architectural decisions entirely to the team

Frameworks with rigid prescriptions constrain architectural choices. Frameworks with no structure shift the entire burden onto the team with no guidance. This second approach might be fine for teams that know what they are doing and how to shape the architecture properly. Not everyone needs guidance from the framework.

1.3 Contexts as a Middle Ground

Phoenix provides contexts as a compromise:

• Explicit, guideline-oriented subsystems that enable functional decomposition without rigid enforcement
• Contexts define functional boundaries while allowing technical organization to remain nested within them
• Functional blocks may later evolve into microservices, but this is optional
• The same decomposition serves equally well in a modular monolith or a distributed architecture
• The choice depends on team needs, scaling requirements, and operational maturity, not on the decomposition strategy itself

2. Pipeline Topology and Data Flow

2.1 The Unix Pipeline Model

Unix pipelines model data flow through a single stream connecting stdout to stdin. This forms a linear chain where each stage's output becomes the next stage's input. Key characteristics:

• Each stage has exactly one input and one output
• Cognitive overhead is minimized because the topology is trivial to trace
• The linear, single-stream characteristic is not mandatory for a pipeline, but it reduces complexity significantly

2.2 Arbitrary DAG Topologies

Orchestrators like Airflow allow arbitrary DAG topologies with fan-in and fan-out edges. Tradeoffs:

• Powerful for expressing complex dependencies
• DAGs with dense interconnections tend to become hard to read even with visual rendering
• Complex function-call topologies with many parameters outside Airflow and Unix pipelines also produce unreadable code

2.3 Byte Streams and Opaque Containers

Unix-like pipelines connect programs by passing data through unidirectional byte-streams:

• Programs at each end agree on a structure such as JSON, CSV, or tar archives
• The pipe mechanism itself transports only raw bytes
• This has a direct analogue in dynamically typed languages: in Lisp and Clojure, collections (lists, maps, vectors) serve as opaque containers that can hold almost arbitrary data
• The consumer interprets the contents rather than the container dictating them

3. Typing Heterogeneous Pipeline Data

3.1 The Problem

Strict type systems introduce complications when handling heterogeneous data flowing through a pipeline where each stage transforms the shape slightly.

3.2 Failed Approaches

Single large type with many optional fields:

• Creates dependencies between all pipeline steps
• Loses the ability to reject illegal data
• Makes reuse difficult
• Changes to one field propagate everywhere

Many separate types for each step:

• Exhaustive and adds noise to the program
• Structures may not be mutually exclusive yet are treated as such
• Maintenance burden grows with every new stage

Both approaches fail because each pipeline stage depends on more than it needs.

3.3 Partial Fixes From Functional Programming

Two techniques alleviate but do not fully resolve the problem:

• Functional record update: enables creating modified copies without mutation, reducing coupling related to state changes
• Sum types: restore the ability to discriminate valid from invalid data and support exhaustiveness checking

Remaining limitation: every step that pattern-matches on a sum type must know about all variants. Adding a new case still propagates changes through the pipeline.

3.4 Structural Type Compatibility

Structural type compatibility offers a complementary solution:

• Independently defined types become compatible based on shape alone without requiring any inheritance relationship
• A consumer can specify only the subset of fields it needs via a structural interface
• Each step depends on a minimal projection of the data rather than the full type
• This decouples pipeline stages more effectively than either naive approach or sum types alone

3.5 Python Implementation

Python implements several of these patterns:

• dataclasses.replace(): supports immutable record updates
• The | union operator: simplifies union type expressions
• Protocol classes: enable structural subtyping, allowing independent types to satisfy contracts based on method and attribute signatures
• Tagged unions: modeled using Literal discriminator fields on dataclasses or TypedDicts
• typing.assert_never with mypy: enforces exhaustiveness checking on pattern matching or if chains, providing compile-time guarantees similar to sum types in functional languages

Combined approach:

• Protocols decouple steps through structural conformance
• Tagged unions enable variant discrimination with exhaustiveness checking
• Functional record updates reduce mutation-related coupling

4. Microservices, Processes, and Isolation Patterns

4.1 The Shared Principle: Isolated State by Default

A microservice and a Unix process share architectural similarities:

• Microservices: in well-designed architectures, a service does not share variables or databases with other services. Communication happens through well-defined interfaces. This is a best practice, not a hard technical constraint.
• Unix processes: each process has its own virtual address space and does not share memory directly with other processes. Explicit sharing is possible through mechanisms such as shm_open (POSIX shared memory) or mmap.

4.2 Historical Lineage

Microservices on GNU/Linux are literally processes communicating via HTTP over TCP/IP. The historical chain:

• TCP/IP: first implemented in 4.2BSD Unix in 1983
• HTTP: developed at CERN in 1989 to 1990, building upon these networking foundations
• Tim Berners-Lee wrote the first HTTP server and web browser on a NeXT workstation running NeXTSTEP in fall 1990
• NeXTSTEP was heavily influenced by BSD Unix
• GNU/Linux copied many initial ideas from Unix while remaining free and open

The modern distributed system traces an unbroken lineage back to Unix.

4.3 Pipes as an Alternative to Microservices

Piping via stdin-stdout chains is another mode of interprocess communication:

• Not as powerful or generic as TCP/IP or HTTP
• Easy to use and reason about
• Naturally fits data pipelines
• A data pipeline can be built using command-line tools piped together, running as processes on GNU/Linux instead of using microservices and a full orchestration system
• Scalability can be achieved by SSH and distribution through GNU Parallel, which launches jobs across multiple machines accessed over the network

4.4 Erlang and Clojure as Additional Isolation Models

Erlang processes:

• Lightweight alternative to Unix processes
• Rich high-level interprocess communication via mailbox message passing
• The Erlang VM enforces process isolation as a runtime guarantee

Clojure and other functional runtimes:

• Do not have the same process isolation constraints as the Erlang VM
• Provide lightweight isolation via persistent data structures and Software Transactional Memory (STM)
• STM allows memory sharing while preventing conflicts even when multiple functions run in parallel or concurrently

Bottom Line: Think Twice Before Going Micro

Here is the real talk. You might want to pause before spinning up your first microservice. Ask yourself these questions:

• Do I actually need physical isolation, or will logical separation suffice?
• Can a simple pipe between processes do the job just as well?
• Am I solving a scaling problem that does not exist yet?
• Do I have the ops maturity to handle distributed tracing, service meshes, and deployment pipelines?
• Will my team understand this architecture six months from now?

The truth is, Unix pipes have been doing data transformation reliably since 1973. Erlang processes have handled millions of concurrent connections since the 1980s. Functional isolation with STM has been working since Clojure showed up in 2009. None of these require Kubernetes. None of them need a dedicated platform team. And none of them will haunt you with debugging nightmares at 3am.

Microservices are not evil. They are just heavy. They are the nuclear option for isolation. Use them when the problem demands the weight. Otherwise, reach for the lighter tool. A pipe, a context boundary, a protocol type. Try the easy solution first. If it breaks, then scale up. Most teams never get to that point. And their systems stay simpler, cheaper, and easier to maintain because of it.

So yeah, think twice. Maybe thrice. Then build the smallest thing that could possibly work.

Permalink