<!DOCTYPE html>
<html>
<head>
<title>Tarn Barford</title>
<meta charset="utf-8"/>
<link rel="icon" type="image/x-icon" href="/favicon.ico">
<link href="/style.css" media="screen" rel="stylesheet" type="text/css" />
<link rel="alternate" type="application/atom+xml" title="Journals of Tarn Barford" href="/atom" />
<link href="/highlight.css" media="screen" rel="stylesheet" type="text/css" />
<link href="/highlight-console.css" media="screen" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="container">
<div id="header">
<div id="header">
<p>From the <a href="/journal">Journals</a> of <a href="/">Tarn Barford</a></p>
<h1>
vim-slime
</h1>
<p>
Mar 26, 2012
</p>
</div>
</div>
<div id="post_content">
<html><body><p>Today I found the awesomeness that is <a href="https://github.com/jpalardy/vim-slime">vim-slime</a>, it's been an exciting day for me.
<a href="http://common-lisp.net/project/slime/">Slime</a> is the "The Superior Lisp Interaction Mode for Emacs", I can almost hear the emacs crowd laughing.</p>
<p>For those that use vim and haven't used Slime, vim-slime or <a href="https://github.com/vim-scripts/VimClojure">something similar</a>, this is why it's awesome:</p>
<p><strong>Text can be sent from any process to the stdin of a <a href="http://www.gnu.org/software/screen/">gnu screen</a> or <a href="http://tmux.sourceforge.net/">tmux</a> session.
The process in this case is vim and the screen/tmux session is a terminal</strong>.</p>
<p>Screen is a <a href="/journal/oh-screen-where-have-you-been">really neat</a> terminal multiplexer (you can run multiple terminals in a terminal window).
The multiplexed shell processes are children of the screen process, which itself is not a child of the terminal window process.
This means a screen process and its child processes keep running if you close the terminal window.
Later you can re-connect to it, this is what makes vim-slime possible.</p>
<p>Here is an screen shot, on the left is me in gVim writing some awful Clojure <a href="#footnote-1">[1]</a>.
On the right is a screen buffer in which I started a Clojure REPL.
When I want to try run some code I can send any vim text selection to the REPL in a keystroke (or two).</p>
<p><img alt="vim slime screenshot" src="screenshot.jpg"/></p>
<p>It doesn't have to be a Clojure REPL either, we can send anything to a screen shell.
We could run git commands, find, grep, sed, etc. Like with the Clojure REPL we can even interact with any terminal programs that use STDIN.</p>
<p>This concept can be taken even further,
You can even connect to a tmux session over SSH and share a terminal or a <a href="http://remotepairprogramming.com/remote-pair-programming-with-tmux-and-vim-the">terminal program like vim to do remote pairing</a>!</p>
<p>Hopefully remote pairing is the topic of my next post as there are a couple geographically distant people I know who are keen to do some pair hacking.
I stand to learn a lot!</p>
<p><a name="footnote-1">[1]</a>
I learnt almost everything I know about Lisp from <a href="http://www.ccs.neu.edu/home/matthias/BTLS/">The Little Schemer</a>. Great book.</p></body></html>
</div>
<div id="comments">
</div>
</div>
<div id="footer">
<p> </p>
<p>Questions, comments, suggestions? Email me, <a href="mailto:tarn@tarnbarford.net">tarn@tarnbarford.net</a> (<a href="/pgp.txt">public key</a>)</p>
<p> </p>
</div>
</body>
</html>
<!DOCTYPE html>
<html>
<head>
<title>Tarn Barford</title>
<meta charset="utf-8"/>
<link rel="icon" type="image/x-icon" href="/favicon.ico">
<link href="/style.css" media="screen" rel="stylesheet" type="text/css" />
<link rel="alternate" type="application/atom+xml" title="Journals of Tarn Barford" href="/atom" />
<link href="/highlight.css" media="screen" rel="stylesheet" type="text/css" />
<link href="/highlight-console.css" media="screen" rel="stylesheet" type="text/css" />
<style>
#swipe-canvas {
position: relative;
width: 900px;
height: 300px;
}
#swipe-results {
font-size: 30px;
padding-left: 50px;
padding-left: 50px;
}
#swipe-results ul {
margin: 0px;
padding: 0px;
}
#swipe-results li {
float: left;
background-color: #DDDDDD;
list-style-type: none;
padding: 10px;
margin: 5px;
border-radius: 5px;
}
#swipe {
position: relative;
}
#swipe-loading {
position: absolute;
height: 50px;
width: 300px;
top: 85px;
left: 300px;
background-color: darkgray;
border-radius: 10px;
text-align: center;
padding-top: 20px;
border: black;
border-width: 5px;
}
</style>
</head>
<body>
<div id="container">
<div id="header">
<div id="header">
<p>From the <a href="/journal">Journals</a> of <a href="/">Tarn Barford</a></p>
<h1>
Swipe Keyboard
</h1>
<p>
Apr 06, 2014
</p>
</div>
</div>
<div id="post_content">
<html><body><p>When I first tried a <a href="http://www.swype.com/">Swype</a> keyboard I was impressed how effective it
was. Even though I don't use the feature on my phone I was interested in how it
could be built, so I <a href="https://github.com/tarnacious/swipe-keyboard">implemented this otherwise useless swipe-able keyboard</a> below. It probably doesn't work on mobile devices, but works on modern
browsers with mouse pointers (although I've only really tried Chrome and
Firefox).</p>
<div id="swipe">
<canvas height="300px" id="swipe-canvas" width="900px"></canvas>
<div id="swipe-results"></div>
<div style="clear: both"></div>
<h2 id="swipe-loading">Loading<noscript>Javascript is Required</noscript></h2>
</div>
<p>I initially tried to solve this using the technique Peter Norvig famously uses
in his <a href="http://norvig.com/spell-correct.html]">spell checker</a>. He takes a sequence of characters and
generates a set of word candidates by adding, removing and swapping characters
in the original sequence, the generated candidates are removed if they are not
found a dictionary. This can work but to be effective too many combinations
need to be generated.</p>
<p>If the dictionary is indexed into a <a href="http://en.wikipedia.org/wiki/Trie">trie</a> the number of combinations
generated can be reduced significantly by traversing the trie and only
generating valid letter combinations. This is a pretty bare implementation of
that, it requires: </p>
<ul>
<li>The first and last characters of the initial sequence are used </li>
<li>Intermediate characters in the initial sequence can be repeated or discarded </li>
<li>No characters are added or swapped</li>
</ul>
<p>Basically, if you swipe through all the characters in a word in order, then the
word will be found if it is in the index regardless how many characters are
swiped in between. It is surprisingly quick and effective.</p>
<p>This implementation uses <a href="https://raw.github.com/first20hours/google-10000-english/master/google-10000-english.txt">these 10000 words</a>, I intended to use digital
books but never got around to it as these words demonstrate the concept well
enough.</p>
<p>This is the first thing I've written in <a href="https://github.com/clojure/clojurescript">ClojureScript</a> or
<a href="https://github.com/clojure/clojurescript">Clojure</a>, so my code my vary from non-idiomatic to shamblolic. I
initially used a <a href="http://clojuredocs.org/clojure_core/clojure.zip/zipper">zipper</a> to build the trie with immutable data
structures, but found the indexing took to long with my zipper implementation
so I <a href="https://github.com/tarnacious/swipe-keyboard/commit/6edd7b26e78121fbe8586b3f0ef54ca8277d9e32">switched to using native Javascript maps</a>.</p>
<p>I found that <a href="https://github.com/clojure/core.async">core.async</a> library is really awesome, the <a href="http://docs.closure-library.googlecode.com/git/index.html">Google
closure library</a> and <a href="https://developers.google.com/closure/compiler/">compiler</a> integration with <a href="http://leiningen.org/">Leiningen</a> the <a href="https://github.com/emezeske/lein-cljsbuild">cljsbuild plug-in</a> to be impressive. My main pains
were the slow JVM start-up time, the advanced closure compiler build of the web
worker script fails silently when run (but the main script works fine when
compiled with the advanced compiler), and at times I felt some compile time
type checking would be nice.</p>
<p>I would like to extend this experiment to index the word occurrence counts and
proceeding word counts in original text and rank the found words as most
likely. Support casing, umlauts, special characters, spelling correction and
compound words in the indexing and lookup. I think a live lookup while swiping
would also be possible.</p>
<p>Overall this was fun, turned out OK I think, and was a great learning
experience.</p></body></html>
</div>
<div id="comments">
</div>
</div>
<div id="footer">
<p> </p>
<p>Questions, comments, suggestions? Email me, <a href="mailto:tarn@tarnbarford.net">tarn@tarnbarford.net</a> (<a href="/pgp.txt">public key</a>)</p>
<p> </p>
</div>
<script src="swipe.js" type="text/javascript"></script>
</body>
</html>
I have a weird relationship with statistics: on one hand, I try not to look at it too often. Maybe once or twice a year. It’s because analytics is not actionable: what difference does it make if a thousand people saw my article or ten thousand?
I mean, sure, you might try to guess people’s tastes and only write about what’s popular, but that will destroy your soul pretty quickly.
On the other hand, I feel nervous when something is not accounted for, recorded, or saved for future reference. I might not need it now, but what if ten years later I change my mind?
Seeing your readers also helps to know you are not writing into the void. So I really don’t need much, something very basic: the number of readers per day/per article, maybe, would be enough.
Final piece of the puzzle: I self-host my web projects, and I use an old-fashioned web server instead of delegating that task to Nginx.
Static sites are popular and for a good reason: they are fast, lightweight, and fulfil their function. I, on the other hand, might have an unfinished gestalt or two: I want to feel the full power of the computer when serving my web pages, to be able to do fun stuff that is beyond static pages. I need that freedom that comes with a full programming language at your disposal. I want to program my own web server (in Clojure, sorry everybody else).
Existing options
All this led me on a quest for a statistics solution that would uniquely fit my needs. Google Analytics was out: bloated, not privacy-friendly, terrible UX, Google is evil, etc.
What is going on?
Some other JS solution might’ve been possible, but still questionable: SaaS? Paid? Will they be around in 10 years? Self-host? Are their cookies GDPR-compliant? How to count RSS feeds?
Nginx has access logs, so I tried server-side statistics that feed off those (namely, Goatcounter). Easy to set up, but then I needed to create domains for them, manage accounts, monitor the process, and it wasn’t even performant enough on my server/request volume!
My solution
So I ended up building my own. You are welcome to join, if your constraints are similar to mine. This is how it looks:
It’s pretty basic, but does a few things that were important to me.
Setup
Extremely easy to set up. And I mean it as a feature.
Just add our middleware to your Ring stack and get everything automatically: collecting and reporting.
(def app
(-> routes
...
(ring.middleware.params/wrap-params)
(ring.middleware.cookies/wrap-cookies)
...
(clj-simple-stats.core/wrap-stats))) ;; <-- just add this
It’s zero setup in the best sense: nothing to configure, nothing to monitor, minimal dependency. It starts to work immediately and doesn’t ask anything from you, ever.
See, you already have your web server, why not reuse all the setup you did for it anyway?
Request types
We distinguish between request types. In my case, I am only interested in live people, so I count them separately from RSS feed requests, favicon requests, redirects, wrong URLs, and bots. Bots are particularly active these days. Gotta get that AI training data from somewhere.
RSS feeds are live people in a sense, so extra work was done to count them properly. Same reader requesting feed.xml 100 times in a day will only count as one request.
Hosted RSS readers often report user count in User-Agent, like this:
My personal respect and thank you to everybody on this list. I see you.
Graphs
Visualization is important, and so is choosing the correct graph type. This is wrong:
Continuous line suggests interpolation. It reads like between 1 visit at 5am and 11 visits at 6am there were points with 2, 3, 5, 9 visits in between. Maybe 5.5 visits even! That is not the case.
This is how a semantically correct version of that graph should look:
Some attention was also paid to having reasonable labels on axes. You won’t see something like 117, 234, 10875. We always choose round numbers appropriate to the scale: 100, 200, 500, 1K etc.
Goes without saying that all graphs have the same vertical scale and syncrhonized horizontal scroll.
Insights
We don’t offer much (as I don’t need much), but you can narrow reports down by page, query, referrer, user agent, and any date slice.
Not implemented (yet)
It would be nice to have some insights into “What was this spike caused by?”
Some basic breakdown by country would be nice. I do have IP addresses (for what they are worth), but I need a way to package GeoIP into some reasonable size (under 1 Mb, preferably; some loss of resolution is okay).
Finally, one thing I am really interested in is “Who wrote about me?” I do have referrers, only question is how to separate signal from noise.
Performance. DuckDB is a sport: it compresses data and runs column queries, so storing extra columns per row doesn’t affect query performance. Still, each dashboard hit is a query across the entire database, which at this moment (~3 years of data) sits around 600 MiB. I definitely need to look into building some pre-calculated aggregates.
If you have used core.logic, you already know half of probabilistic programming. A logic program states constraints and lets the system search for the values that satisfy them. A probabilistic program does the same, except that a constraint can be soft: a value that fits the data poorly is not ruled out, only made less likely. Two dice show the idea below. A small forecasting model then runs under several inference methods in Foerster, an MIT-licensed probabilistic programming library for Clojure, and every result is checked against an exact answer.
Foerster continues the Anglican line of probabilistic programming in Clojure. Christian Weilbach, who wrote it, did his PhD in Frank Wood’s group at the University of British Columbia, where Anglican was built.
What is a probabilistic program?
Take two dice and the constraint that they sum to 7. Without any library, the plainest way to find the pairs that meet it is to generate pairs at random and keep the ones that pass:
defn roll []:
inc(rand-int(6))
end
defn generate-and-test [n]:
loop [kept []]:
if n = count(kept):
kept
else:
a := roll()
b := roll()
recur(if 7 = a + b:
conj(kept, [a b])
else:
kept
end)
end
end
end
frequencies(generate-and-test(6000))
;; each of [1 6] [2 5] [3 4] [4 3] [5 2] [6 1] about 1000 times
(defn roll [] (inc (rand-int 6)))
(defn generate-and-test [n]
(loop [kept []]
(if (= n (count kept))
kept
(let [a (roll) b (roll)]
(recur (if (= 7 (+ a b)) (conj kept [a b]) kept))))))
(frequencies (generate-and-test 6000))
;; each of [1 6] [2 5] [3 4] [4 3] [5 2] [6 1] about 1000 times
The random choices form a generative model, a small simulator of two dice. The test conditions the simulator on what we know. What survives is the posterior: the pairs that are still possible, and how often each occurs.
A probabilistic program states the same model without fixing how to search it:
sample introduces an unknown, as a fresh logic variable does in core.logic, but with a probability on each value: here a die whose six faces are equally likely. factor adds to the run’s log weight. Zero leaves the run unchanged, and minus infinity makes it impossible, as a failing goal does. sp/spin turns the body into a program that an inference method can pause, copy and resume at each sample and factor. With {:method :enumerate}, Foerster runs all 36 pairs and returns the six that hold, each with probability 1/6. That is the answer run* gives, with weights.
Now soften the constraint. We do not know the sum, but a sensor that measures it with noise of about one showed 7:
(observe (dist/normal (+ a b) 1.0) 7.0 :id :reading)
observe weights each run by how likely the reading is under it, here a normal density centred on the run’s sum. No pair is ruled out, and the arithmetic can be done by hand. Each of the six pairs that sum to 7 gets weight e⁰ = 1, each of the five pairs that sum to 6 or to 8 gets e^−½ ≈ 0.61, and the weights keep falling with the distance from 7. Normalized, a sum of 7 has probability 0.45, sums of 6 and 8 have 0.23 each, sums of 5 and 9 have 0.04 each, and the rest almost nothing. Foerster’s enumeration gives the same numbers.
core.logic
Foerster
a fresh logic variable
sample, an unknown with a probability on each value
a goal such as ==, which holds or fails
observe or factor, a weight; failing is weight zero
conde, a choice between branches
a discrete sample, with a probability on each branch
run*, the set of answers
the posterior, the answers with their weights
the search strategy
the inference method
Two more ideas carry over. The total weight before normalizing, called the evidence, says how well the model as a whole explains the data. A hard constraint that no run meets leaves nothing, and a soft one leaves a number that can be compared between models. And as core.logic leaves the search to the system, a probabilistic program leaves the inference method open. ProbLog joins the two traditions as probabilistic logic programming.
How does the system find the answer?
Enumeration works for 36 pairs. It fails for many choices, or for continuous ones, because no list of all runs exists. The other methods search the space of runs, and each resembles a familiar strategy:
Importance sampling runs the program many times and weights each run instead of discarding it. It is generate and test with soft scores, and it fails when good runs are rare.
Sequential Monte Carlo (SMC) runs many copies side by side. At each observation it copies the runs that fit and drops the ones that do not, like a search that prunes partial solutions as soon as a constraint can be checked.
Markov chain Monte Carlo (MCMC) changes one choice of a single run at a time and keeps the change by a rule that sometimes accepts a worse run. This local search visits runs in proportion to their weight.
Does the choice of method matter?
A shop sells around a hundred units of a product each week. The weekly figure is noisy, and the underlying demand drifts. How high is demand now, and how certain is that? The model says that demand starts near 100, moves by a random amount each week, and that each week’s figure is demand plus noise:
As with the dice, sample marks the unknowns, here twenty-one weekly demand levels, and observe scores each run against the twenty sales figures. The data are an ordinary argument, weeks. The :id names each choice, so a method can refer to the level in week 7 across runs. The pictures article explains priors and posteriors if those terms are new. Each method is one option of the same call:
The model is linear and Gaussian, so a Kalman filter computes the exact answer, which makes it a fair test. The data are twenty simulated weeks, with a true drift of 4 per week and observation noise of 8. Each particle method ran ten times with different seeds, so the table shows how much an answer varies from run to run. All numbers come from Foerster’s main branch on 15 October 2026.
Method
Final level (mean)
Log evidence
Check
Exact (Kalman filter)
102.89
−77.49
Sequential Monte Carlo, 1000 particles
102.92 ± 0.23
−77.44 ± 0.14
about 120 histories survive to week 0
Importance sampling, 1000 samples
102.9 ± 1.8
−77.7 ± 0.5
2 to 10 effective samples
Metropolis–Hastings, 4 chains
103.4
R-hat 1.03, 114 effective draws
Particle Gibbs, 4 chains
102.93
R-hat 1.002, about 1500 effective draws
SMC lands within 0.36 of the exact final level in every run, and its log evidence averages −77.44 against the exact −77.49. It pays for its pruning on the early weeks: of a thousand particles at the end, only 114 to 134 still descend from different starting points. Importance sampling never prunes, so after twenty weeks a handful of runs carry nearly all the weight, and the estimate moves by almost two units between runs.
Metropolis–Hastings (MH), the simplest MCMC, fails here, and its diagnostics say so before any exact answer is consulted. Its four chains disagree: their rank-normalized R-hat is 1.03, above the usual threshold of 1.01, and their 12,000 draws are worth about 114 independent ones. Foerster computes these numbers as Stan and ArviZ do. The cause lies in the model. Neighbouring weeks are strongly tied, so changing one week’s level at a time barely moves the run. Andrew Gelman’s folk theorem of statistical computing, restated in the Bayesian workflow paper, says as much: when the computation struggles, look at the model.
The fix is a method that changes all weeks together. Particle Gibbs runs a whole SMC through the twenty weeks at every step, holding one surviving run fixed, and so redraws all twenty-one levels at once. Four chains of 500 steps after warm-up, with 50 particles each, reached an R-hat of 1.002 and about 1500 effective draws of the final level, counting each step’s weighted average as one draw. Their mean was 102.93 against the exact 102.89. The first week’s level, which SMC sees through only a few surviving histories, reached an R-hat of 1.002 and about 660 effective draws. Particle Gibbs took four minutes against half a minute for MH, but produced about twice as many effective draws per second. Nothing in the model changed, only the search.
Each point is one week's observed demand. The shaded band is where sequential Monte Carlo puts the hidden demand level with 95% probability after that week, and the dashed line is the exact answer, which this simple model happens to allow. The two lines nearly coincide.
What if a parameter is unknown too?
So far the drift was given as 4. In practice nobody measures how fast demand moves, so the drift becomes one more unknown with a prior:
Particle marginal Metropolis–Hastings (PMMH) searches over the drift alone. For each proposed drift it runs SMC over the weekly levels and accepts or rejects the drift by the evidence that SMC reports. That evidence estimate is unbiased, so the chain targets the exact posterior of the drift even though every step uses an estimate.
Twenty weeks of data narrow the drift only partly: values from below 1 to about 6 remain plausible. The histogram of PMMH draws follows the exact density, which quadrature over the Kalman filter's evidence provides for this model.
The chain’s mean drift is 2.82 against an exact 2.85, and its 90% interval runs from 0.61 to 6.09 against an exact 0.61 to 6.28. The data were simulated with a drift of 4, inside the interval but not at its centre, because twenty noisy weeks cannot pin the drift down. A forecast with one fitted drift would hide that width. The 2000 iterations took five and a half minutes on a laptop. For this linear Gaussian model, Stan or a Kalman filter inside a one-parameter chain would take seconds. PMMH is for models without a gradient to follow, models that branch, count or call a simulator.
Build the model inside with-context, as here: outside a world, the program has nowhere to run. The scripts that produced every number above are in the Foerster repository.
What can Foerster not do yet?
Foerster is beta software. Beyond the methods shown here, it implements tempered and waste-free SMC, SMC², involutive MCMC with reversible jump, parallel tempering, the No-U-Turn sampler on numerical blocks, MAP estimation, the Laplace approximation, Pathfinder, and variational inference. The methods notebook describes each one with its papers, and the gallery works through classic models against exact answers.
For a static model with continuous parameters, Stan, PyMC and NumPyro remain the mature choice. Foerster’s No-U-Turn sampler works on numerical blocks whose gradient the model supplies or Raster compiles. On one hierarchical model with 89 parameters, it matched PyMC’s warm run with a gradient compiled by Raster, and ran 1.5 times faster with a hand-written one. Its general methods over program traces are slower: single-site MH takes about twelve times as long per move as Gen, because each move replays the rest of the program, and its particle filter is about four and a half times slower than Gen’s Unfold combinator. The comparison has the details, and coming from other systems maps Stan, PyMC, Turing, Gen, Anglican and WebPPL concepts onto Foerster, including what is missing.
What comes next?
Every model so far is a pure function. Useful models often are not: a forecast that reads current stock, a simulation that writes into a scenario database, an agent that calls tools. If a thousand particles write to the same database, inference corrupts the state it reasons about, which is why Gen requires that a model change no external state.
Foerster runs each particle in a Spindel world, whose state can be copied cheaply, as the copy-on-write article describes. When SMC copies a particle, it copies the particle’s world, and when it drops a particle, it discards the world. With :world-policy :fork, every particle starts from a copy of the caller’s world, so a model can read and change the systems registered there, and nothing a particle writes reaches the caller. This costs about three times as much as running in fresh worlds. Systems that cannot be copied safely, such as live network handles, are refused before the model runs, and Markov chains run in fresh worlds only. The same mechanism makes counterfactuals direct: intervene on one choice in the world where the posterior was computed (Pearl’s do operator, as in the causal graph article) and replay the program with everything else held to what was inferred. The worlds guide covers both, and a later article shows them on a model that books orders into a forked database.
The same machinery steers processes that are not statistical models. Foerster’s steer namespace runs SMC over the steps of an agent working on a task, following twisted SMC (Zhao et al., 2024) and the SMC steering of language models (Lew et al., 2023). On short spreadsheet tasks it matched picking the best of four attempts with 17% fewer tokens, but solved nothing more. It should pay on long tasks whose steps can be checked cheaply.
For Simmis, inference over forked worlds is research rather than a product feature. It is how a governed organizational record could later answer a question such as “what happens to our cash if this order slips” with a distribution instead of a single number.
I&aposve been having issues lately with my machine running out of RAM because of too many agents doing too many things at once. I know I know I need to get on the cloud-agent bandwagon, I&aposm working on it, but in the meantime I have become very productive with a dozen or so claude sessions running locally all the time. Anyway all that to say, I have started to pay a lot more attention to what is using up all my RAM because I ran out for the first time the other day. The desktop slack app is a pretty bad offender, regularly consuming 1-2GB of RAM, so I went looking for lighter-weight alternatives (since I need to have it running for work, and hate the experience of using it in a browser) and found this awesome slack TUI.
It&aposs really amazing and was immediately more ergonomic to my sensibilities, but there were many rough edges. I don&apost believe in putting up with paper cuts any more. In this day and age coding agents can fix all of this stuff more or less autonomously, so yesterday I decided to just do it. Slack is the software I use second-most at work, after Claude Code, so running into minor friction with it a hundred times a day was getting pretty frustrating, but I really like the lighter-weight experience of using slack from a terminal.
This raises a lot of meta questions for me about being a good open source citizen, but in this case I actually just needed to get some work done and the bugs in the app were getting in the way of that, so I decided to fork the repo and just fix them all. Over the course of the day I thought I was done three times (see the other two releases, v0.27.0 and v0.28.0), but every time I thought I had fixed the bugs, I realized I was running into more.
For better but mostly for worse, I do not suffer from bug blindness, so it took a few rounds until I stopped running into paper cuts in the course of my normal slack usage. Anyway.. I think there&aposs another post worth writing about the kinds of things I wanted to fix that made this worth forking rather than collaborating (there were some pretty fundamental things around the modelling of time and event architecture, among other things), and I don&apost love the dynamic of having just hijacked this project for my purposes, but also that is the point of open source, after all, right? I tried opening an issue for the first bug I ran into last week and haven&apost heard back, and now after just a few hours of vibing, I have a really awesome app that works really well. So anyway, it&aposs open source and available to everyone. If you find issues of your own feel free to contribute, or fork it and run with it on your own!
This week, the topic is: "ingesting, deciding, and merging". We curate data, merge it, and represent choices.
Our discussion includes:
Different patterns of pure models
Outside data vs inside data
Differences and modelling tension between external and internal data
Why pure models tend to have patterns of functions
Ingested model
Decision-making model
Merging model
Async join
How pure models are combined with necessary side effects
Using models in procedural flows and data flows
Real world integration examples from Kubernetes and e-sports
Selected quotes
A pure data model is data that models something in the real world and pure functions you pass that data through. That's it.
For an ingested model, you want to curate a subset of external data, or normalize it in some way, so it's useful to the rest of your application.
Two main pieces: a representation of the information that is pertinent to your application, and a set of functions that operate on that data. Those functions tend to fall into patterns.
The ingested model is a coherent window to that outside world. It's the lens through which your application sees that external data. It serves as a stabilizing force for the rest of your application.
The Kubernetes data is wide, and it's deep. It's got a lot of information. It's only rivaled by GitHub responses in its size and scope!
The decision, itself, is data. If you want to push side effects to the edges, you need to represent the result of a decision in some way.
You put the side effects on the edges. I/O on the way in. I/O on the way out. The pure models in the middle.
The code that connects the model to a side effect is as simple as taking things out of a map and passing them as arguments into a side effect function. It only does one thing. It's almost one line of code. It's so small that it's easy to understand.
Most of your mind is focused on the ingested model and the decision-making model. You're curating and representing information, and then you're reasoning about that information.
A merging model is a pure model that takes other pure models and creates a composite, or enriched, singular output. It's multiple pieces stitched together into a larger work.
Merging models together is a separate concern from the individual models. A merging model's job is to construct a cohesive view from separate, but related information.
If it's all pure, you can test it. You can make sure that it behaves the correct way in different situations.
A way to look at at different models is by responsibilities. The outside system takes responsibility for representing the data in a way that has utility for its concerns, design constraints, and realities. The ingested model's responsibility is figuring out what matters for the application. The decision-making model's responsibility is representing choices that matter and the consequences of those choices. The merging model's responsibility is about how to connect separate things into a larger whole
So each kind of model is a separate problem space, but they're all related, and it becomes clear how they are connected. By decomposing models into separate problem spaces, you end up with models that are relatively stable because they only need to change when that aspect of the problem changes. They are decoupled and orthogonal to each other because they're in different information domains, problem domains, responsibility domains.
One of the best parts of a decoupled system is that it's easy to understand one model at a time. The theme of this series is "re-entrant coding". I only want to understand the parts of the system that I have to understand to accomplish the current task. I don't want to have to understand everything altogether. By decoupling these things with clean responsibilities and boundaries, we get a system that's easy to start working with again.
Each model is visible because it takes data and produces data. You have something that can be seen, inspected, logged, recorded, and put in a test. You avoid the kind of functional composition where information goes through a bunch of internal transformations, and you can't quite get a hold of it in your head. You have a relatively shallow call stack because each model produces data. The orthogonality of the models helps keep the call stack shallow.
A nice, related set of pure models: ingesting, making choices, and merging things together.
You can make the functional jokes. I'll stick with the dysfunctional jokes.
I made a little web app called BudgetSWU. It recommends
decklists for a trading card game called Star Wars: Unlimited. I started playing
SWU almost a year and a half ago because I had a dream I was in a mall opening a
pack of Yugioh cards--another trading card game that I played as a kid--and it
gave me enough nostalgia to do some googlin' and discover SWU.
As an adult I now have lots more money to buy cards than I did as a kid (yay)
but much less time to play with them (boo). So I don't spend much time building
my own decks; I just find decklists on the internet and buy the cards for those.
BudgetSWU is now one of those places on the internet. It gets decklists from
melee.gg which is used in tournaments (large and small),
including each decklist's (player's) placement in the tournament. It also gets
pricing data for all the cards in those decklists from
tcgcsv.com, which in turn aggregates pricing data from
TCGPlayer, the ebay of trading cards (it is owned
by ebay, in fact).
As a final bit of data, I compute "popularity" by taking each deck's "archetype"
(the leader + base, for those in the know--except that I'm actually using just
base color) and seeing how many decklists in the entire corpus have that same
archetype. It's cooler to play decks that not everyone else is already playing,
know what I mean?
Then BudgetSWU filters all the decklists down to a handful of recommendations by
computing a Pareto Front. The
idea here is that we have multiple factors with which we can rank decks (cost,
performance, popularity/nicheness) and we don't necessarily know which factor is
most important (if deck A costs 50% more than deck B, how much better/niche does
deck A have to be for it to be a better recommendation than deck B?). But we
do know that if deck A beats deck B on all three of those factors, then deck A
is probably the better recommendation. So we filter out all the decks that have
a clearly better alternative, and then we tell the user "here's what's left; you
decide." As of writing, that ends up being less than 20 decks, which is a very
manageable number for a user to scroll through.
The next cool bit is that if you make an account, you can enter what cards you
already own, and then the recommendations will be computed based on the cost of
only the cards you don't have yet. Entering in all your cards would be a pain
since players old enough to have jobs usually have thousands (of cards, not
jobs). So BudgetSWU sorts all the cards by cost, most expensive first. (Right
now the most expensive cards go for about $30-ish a piece). You can start from
the top and enter quantities until you got to cards that cost, say, a dollar
each or whatever you feel like, which doesn't take long.
Finally: whenever a new decklist makes it into your personalized recommendations
(tournaments happen every week), you get a notification email. Booyah.
I generated the code in Clojure.
It uses my web framework Biff. I've recently been working
on some LLM skills that provide guidance for writing Biff code that doesn't
suck. And I have to say, I'm fairly pleased with the results. The code still
isn't anything I would print out and frame on my wall, but the overall structure
is generally inline with how you're supposed to write Biff apps. If this was an
app I planned to develop further (it's not), I would be happy with the state the
code is in as a starting point.
If you're into specifications, you can read the one for BudgetSWU over
here. i.e. that
page is the thing I actually typed, mostly.
AI can help organisations analyse legacy systems faster
One of the biggest barriers to modernising legacy technology is understanding how existing systems work.
To modernise systems, organisations first need a clear picture of how applications work, connect and depend on one another. That process can take weeks or even months. Documentation is often incomplete or out of date, knowledge may be concentrated in a small number of specialists, and some systems rely on technologies that few engineers still work with regularly.
Our pilots indicate that AI-assisted approaches could help technical teams analyse some legacy systems more quickly.
Across participating organisations, activities estimated to take weeks using traditional approaches were completed in hours.
Working with organisations including the Student Loans Company (SLC), the UK Hydrographic Office (UKHO), the Met Office, and our delivery partner Kainos, we found that AI can help teams analyse unfamiliar systems, identify dependencies and build technical knowledge more quickly than traditional approaches alone. The pilots covered different systems and technical environments, so the results should not be treated as directly applicable to every legacy estate.
However, for organisations managing large and complex technology estates, this could make some modernisation activities easier to prioritise, plan and deliver.
What happened at the Student Loans Company
At the Student Loans Company, teams explored how AI assisted approaches could be used to analyse an estate of 33 applications written in Clojure and support future modernisation planning.
Using traditional approaches, the work was estimated to take around five weeks. During the pilot, teams completed the analysis in less than a day.
The analysis gave the team a clearer view of the 33 applications and their dependencies, which it used to inform migration planning and work towards supported versions of Java.
The most significant finding was not the reduction in effort itself, but the ability to unlock work that had previously been difficult to prioritise.
What happened at the UK Hydrographic Office
At the UK Hydrographic Office, teams focused on a 20-year old desktop application that previously had no automated tests.
AI assisted techniques were used to generate automated tests for the application. Test coverage increased from 0% to 96%, while regression testing that had previously required around 15 days of manual effort was reduced to approximately 30 minutes.
The benefits extended beyond faster testing.
Teams reported greater confidence in making changes to the application safely and maintaining it over time. For many legacy systems, the perceived risk of change can be a barrier to improvement. The pilot indicated that AI-assisted testing could give engineers more evidence when assessing proposed changes.
Why this matters
The examples from SLC and UKHO illustrate a wider challenge facing organisations across the public sector.
Legacy systems often support critical services, but analysing them can be complex and expensive. Teams may spend weeks mapping dependencies, reviewing code and building technical knowledge before they can assess risks or plan change.
The strongest finding from the pilots was that AI can help accelerate this discovery and assessment phase. That may sound modest, but it addresses a critical point where many modernisation efforts slow down or stall.
By helping teams understand systems more quickly, AI assisted approaches can make it easier to:
assess risks
identify system dependencies
prioritise investment
improve technical documentation
develop modernisation plans
In short, AI can help organisations spend less time understanding their systems and more time improving them.
AI is not a substitute for good delivery
The pilots also reinforced an important lesson.
AI does not remove the need for good governance, engineering expertise or delivery capability.
Successful outcomes depended on organisations having the right foundations in place, including clear ownership, access to systems and information, experienced technical teams and established delivery processes.
In practice, the pilots showed that AI amplified existing strengths and weaknesses rather than replacing them.
Used well, AI can accelerate analysis, improve visibility of complex systems and help organisations make progress more quickly. Used without appropriate controls, it can amplify existing problems and accelerate poor decision making.
Where AI-assisted analysis may help
The pilots suggest that AI can significantly reduce the effort required to assess complex systems, helping organisations move more quickly from discovery to modernisation.
For organisations managing large and complex technology estates, this can help engineers, architects and delivery teams to identify risks earlier, making modernisation activities easier to plan, justify and deliver.
What happens next
We are continuing to work with organisations using AI assisted approaches to analyse, assess and modernise legacy systems, building on the lessons from these pilots.
Alongside this work, we are developing a playbook of good practice that captures practical approaches, lessons learned and implementation considerations for organisations looking to apply AI to legacy technology challenges. The playbook is currently undergoing user testing, review and approval ahead of publication.
Get involved
If your organisation is exploring AI assisted approaches to analysing, documenting or modernising legacy systems, or would like to share experiences and engage with our wider Firmer Foundations programme, we would like to hear from you. Get in touch atfirmer.foundations@dsit.gov.uk
Clean Code was the first text I handed to my students. Since then I have read the six other books fighting for the "clean code" shelf, and they do not all say the same thing. Sometimes they say the opposite.
I wrote a reading note for each one. Here is the order I would follow if I were starting over, and what you can leave on the shelf.
The craft before the rules
The Pragmatic Programmer by Thomas and Hunt (2019 edition, 8/10) is my way in. What strikes you first is the tone: few commandments, many stories, never a condescending look. And this edition publicly corrects the previous one. The Law of Demeter is demoted, DRY (do not duplicate a piece of knowledge) is clarified because people misread it, and on security the authors write "we were wrong". A book that owns its mistakes twenty years later earns trust for everything else.
The limit is mechanical: 53 topics in 352 pages means skimming. The tooling chapter is thin, and the code samples jump from Ruby to Elixir to Clojure without digging into any of them. You read it for the stories and mental models, not the code. Skip the samples, keep the rest.
The dogma and its counterweight
Clean Code by Robert C. Martin (2008, 6.8/10) taught a generation that code is read ten times more than it is written. The chapters on naming, functions and comments have not aged a day. The rest is verbose Java, and the last three case-study chapters are 90 pages of listings nobody reads anymore, me included. One thing the fans forget: Martin himself admits that no research backs the short-functions rule, and that his own functions start out long and messy before being refined under tests. The "three lines maximum" dogma comes more from his readers than from the book.
To hear the other bell, read A Philosophy of Software Design by John Ousterhout (2021, 8/10). A hundred and ninety-six pages, and in my eyes the best software design book of the last twenty years. His deep module idea changed how I judge a class: I stopped counting lines and started comparing what the interface costs to learn with what it gives back. The examples circle around two or three cases (a text editor, RAMCloud) and smell like a lecture hall. And "make your modules deep" is easier to understand than to apply. Where the two books contradict each other, on function size and on comments, Ousterhout is almost always right.
The first hundred pages of construction
Code Complete by Steve McConnell (2004, 960 pages, 7.4/10) is the book I wish I had read at the start of my career, but for its first hundred pages, not its 960. The thesis on complexity, abstract data types, cohesive routines, defensive barricades: you usually learn all of that by trial and error, after humbling code reviews. The book shortcuts that painful education.
What has aged is visible: samples in C++, Visual Basic 6 and Pascal, "code tuning" chapters about processor pipelines, a tools catalogue straight from a museum. What holds up is everything about humans: the numbers on code inspections, the psychology of debugging, complexity as the universal enemy. You do not need all of it. Parts I, II and VII plus chapters 20 and 21 make roughly 250 pages, and the rest is reference material you open now and then.
Refactoring and legacy, the two moves
Refactoring by Martin Fowler (2018, 8.2/10) holds the best code lesson I have ever read. Chapter 1 is seventy pages of watching someone work, every move explained, the code never broken. Fowler has a rare honesty: his central argument is labeled "unproven hypothesis". The limit is that half the book is a dictionary of transformations nobody reads in one sitting. Read chapters 1 to 4 like a novel and keep the rest within reach during code review. The Refactor menu in your editor exists because of this book.
Then comes Working Effectively with Legacy Code by Michael Feathers (2004, 7/10), which redefines the word: legacy is not old code, it is code without tests. His characterization tests (which snapshot current behavior) answer a question nobody phrases well: how do you start testing when you do not know what "correct" means for this code? The seam model (places where you can change behavior without editing the code) teaches you to read code by looking for joints. Feathers also admits that his techniques often make code uglier in the short term, and that the scar is temporary. That frankness makes the book reliable.
It is dated, and every example is Java, C++ or C. Part II, twenty chapters in FAQ format, often repeats the same four moves under a different class name. Open it at the chapter matching your problem of the day.
Test first, to lose the fear
Test-Driven Development: By Example by Kent Beck (2002, 240 pages, 7.6/10) does not sell a recipe, it shows a state of mind: the calm of knowing, every second, that the code still works. The fear of breaking things turned into quiet boredom. Since then I no longer write tests to "have coverage", I write them to move forward without dread.
Part I, the money example, drags: you repeat the same micro-cycle dozens of times and you got it long before. The code is 2002 Java and Python 2. TDD is still debated, and Beck is the first to say no study settles it. Read it for the mindset, keep the cycle, the TODO list and "fake it" when you are stuck, and adapt the micro-steps to your own pace.
The book I used to hand my students has the lowest rating on the list, 6.8/10. And it is Fowler and Feathers who hold the moves you actually use on a Monday morning, in code you did not write.
I stitched the marrow of dozens of tech books into a single thread, from bits and Big O up to the AI era: that is The Developer's Great Book. Four of these books (Clean Code, A Philosophy of Software Design, The Pragmatic Programmer, Refactoring) are melted into it at the "Writing for humans" level, and three others (TDD by Example, Working Effectively with Legacy Code, Code Complete) at the "Prove it, ship it, hold the load" level.
In his October 11, 2026 article, "'AI' is Not a Tool," Rich Hickey argues that
LLMs are automation rather than tools because they cannot be wielded. I agree
that outsourcing work can cost us opportunities to learn. But his conclusion
does not follow. Even by his definition, an LLM can be something a human wields.
And using it well takes skill, just as using other tools does.
Picking up a saw does not guarantee a straight cut. Holding a chisel does not
make you a sculptor. Tools can amplify poor judgment as readily as good
judgment. Their usefulness depends on the operator's ability to direct them,
interpret feedback, recognize mistakes, and make corrections. Those
requirements also apply to LLMs.
One of my colleagues used to say, "AI is a mirror." I find that insightful:
it reflects who you are, including the curiosity, judgment, and habits you
bring to it. The questions you ask and the standards you apply to its
answers matter.
An LLM's unpredictability is a meaningful limitation. But predictability,
reliability, simplicity, and bounded function are qualities by which we
evaluate tools, not prerequisites for membership in the category. A tool does
not stop being a tool because it is difficult to control or easy to misuse.
Hickey's distinction between tools and automation also creates a false
division. Automation can be a feature of a tool. A compiler performs
substantial work without the programmer continuously directing each
operation. A spreadsheet recalculates results without someone manually
carrying out every calculation. We still wield these systems by specifying
inputs, examining results, and deciding what to do next. Why should
generating text automatically disqualify an instrument from being a tool
when translating code or calculating numbers automatically does not?
The claim that LLM use lacks intention, feedback, direction, and expertise
is especially difficult to defend. A programmer can ask an LLM to explain
unfamiliar code, challenge that explanation, inspect the source, test a
proposed change, reject it, and refine the request. That is a feedback loop
directed by a human pursuing a purpose. Doing it well requires knowing what
to ask, what evidence matters, and when the answer is wrong. Expertise can
make the instrument more useful.
Hickey's concern about reading is valid for a particular use: accepting a
summary in place of reading a paper can sacrifice much of what reading
teaches. But someone can also read the paper and use an LLM to question
their interpretation, explore an objection, or identify something to
examine more closely. The possibility of avoiding intellectual work does
not establish that every use avoids it. The relevant question is what the
person actually does.
What disappoints me the most is the reliance on the AI industry's
anthropomorphic vocabulary: "go rogue," "escape containment," "conspire."
Repeating those expressions risks accepting the framing that makes these
systems sound like independent beings. Software can produce unexpected
outputs, fail to follow instructions, or cause harm when given access to
consequential actions. Those are real engineering problems. Calling them
"conspiracy" adds a suggestion of human motives without establishing any.
I would not claim that an LLM always does exactly what a human tells it to
do. It plainly can fail. But failure to obey perfectly does not make it
cease to be a tool, any more than successful use proves that its operator
has surrendered their judgment.
The useful distinction is between wielding an instrument with skill and
delegating work without adequate understanding or oversight. That
distinction applies to AI, and it gives us something more practical than
declaring that an entire class of software cannot be a tool.
It's been a while since I worked on my Clojure libraries, so I would like to give a little update on that. Two of them got a new release, and one got a debut release on Clojars.
clj-pnm [1]
Apart from the minor bug fixes and Clojure(Script) dependency update, the most exciting thing is that the Clojure dialect support matrix has grown. clj-pnm was built from the start as a cross-dialect library supporting both Clojure and ClojureScript. It is lightweight enough to expand that.
For the Clojure side, the library is compatible with Babashka, as it can be called from Babashka code and can be used as a Babashka dependency.
On the ClojureScript side, clj-pnm can target NodeJS and the browser as before. Switching from figwheel to shadow-cljs for the testing framework was integral in ensuring that. There is a special test runner for testing Squint compatibility, but with no code changes, this lightweight dialect of ClojureScript is also added to the compatibility matrix.
There are tests for each of these in the library repository[2].
cljs-map-pins [3]
cljs-map-pins got a maintenance release, so no new features yet. But this post can serve as a reminder of how fun it is to use this library (at least for me :-D). I love visualizing things on a map, and this library is made just for that. Two examples I like to refer to, albeit they are made with a much older version of the library, are The Warriors path through New York City[4] and Big Wayne tribute[5].
See the repository[6] for more details.
defns [7]
defns is a proof-of-concept library that provides a way for defining functions with type signatures for which :pre and :post conditions are generated. This is an experimental idea and should not be used for anything serious, but I've seen some interest in it and decided to release it to Clojars. The fun part was writing the library clj-kondo configuration and custom hooks for the macros. When using defns as a dependency, make sure to import the clj-kondo configuration for the linting to work.
For more than a decade the Clojure REPL most people saw in a terminal was
REPLy, the one behind lein repl. It
served us well, but it’s showing its age, and I kept wishing for a terminal
REPL that felt like CIDER, without the Emacs part. So I finally built one.
neorepl 0.1 is out today!
Why another REPL?
There’s no shortage of Clojure REPLs, so it’s fair to ask. Here’s what I was
after:
nREPL all the way down. neorepl only speaks nREPL, so it works with anything
that runs an nREPL server: Clojure, babashka, shadow-cljs, Basilisp,
ClojureCLR and so on. The smarts live on the server. When cider-nrepl is
there neorepl uses it, the same way CIDER does, and when it isn’t it falls
back to nREPL’s built-in ops, and then to evaluating a bit of code.
A REPL that’s pleasant to use: a proper line editor, highlighting,
completion, arglists as you type, pretty printing, history per project and
a Ctrl-C that actually interrupts the evaluation.
A CLI that’s just as good for scripts and coding agents. Every lookup the
REPL can do is also a command, with structured output and exit codes that
mean something.
Something small and modern. Clojure 1.10+, nREPL and JLine 4 are the only
dependencies, and the same code runs on the JVM and on babashka. The whole
thing is about 3,300 lines right now, and I’d like to keep it that way.
rebel-readline and bb repl --connect are both great, and I borrowed from
them freely, but neither tries to be an nREPL-first tool with CIDER’s tooling,
jack-in and a CLI for agents. That combination is the whole point of neorepl.
Update (2026-10-11): A few people asked how neorepl relates to the
Clojure CLI REPL, which the
Clojure team released a couple of weeks ago, and to bb repl --connect. (I’m a
bit embarrassed to admit I hadn’t heard of the former until after I announced
neorepl.) Funny enough, all three are nREPL clients under the hood, but they’re
after pretty different things. In practical terms:
The Clojure CLI REPL runs only on the JVM and needs Clojure 1.12+. It ships
its own nREPL server, and its docs and data inspector are ops of that server,
so against any other server you can only evaluate code. It’s deps.edn only.
Its editing is really nice, though - structural editing, evaluating the form
at the cursor and a datafy-based inspector - and I’ll be borrowing some
ideas from it. It doesn’t do completion yet.
bb repl --connect puts babashka’s console REPL in front of any nREPL server,
with completion, arglists and docs, in very little code. It’s a great way to
poke at a running server, but it won’t start one for you.
neorepl works with any nREPL server and gets the most out of the ones with
cider-nrepl (or babashka’s take on it), without any middleware of its own.
It starts a server for Leiningen, deps.edn, babashka and shadow-cljs projects,
follows you into ClojureScript, and everything the REPL does is also a
command with JSON output for scripts and agents. No structural editing yet,
however.
So, if you want the fanciest line editing and you live in deps.edn land, give
the Clojure CLI REPL a try. If you want CIDER in your terminal, for you and
for your agents, that’s what neorepl is for. The neat part is that babashka’s
server bundles CIDER’s inspector and test runner ops these days, so neorepl’s
test runner (already on main) and the inspector that’s coming next work with
babashka out of the box.
For humans
Here’s the REPL in action:
A few things worth pointing out:
Enter only submits the input once its forms are complete, so pasting a whole
function just works.
TAB completes, with the candidates’ types next to them, and the arglist of
the function you’re calling shows up after the cursor.
Values get pretty printed on the server and highlighted on the client.
Commands start with a comma, the CIDER way: ,doc, ,source, ,apropos,
,macroexpand, ,pst, ,reload, ,edit (which opens a definition in your
$EDITOR) and ,help for the rest. The Clojure reader sees a comma as
whitespace, so they can’t clash with real code.
Ctrl-C interrupts what’s running, and code that reads *in* gets its input,
something REPLy never quite got right.
Without a server to connect to, neorepl starts one for the project, the way
CIDER’s jack-in does, with cider-nrepl on the classpath, and stops it when you
exit. It knows Leiningen, the Clojure CLI, babashka and shadow-cljs projects.
ClojureScript works too. Start a ClojureScript REPL in the session (through
shadow-cljs or piggieback, which jack-in adds for you) and neorepl follows
along: evaluation, completion, docs and the rest switch to ClojureScript, and
:cljs/quit takes you back.
For agents
Coding agents are surprisingly good at Clojure, as long as they have a REPL
to talk to. Most of them drive CLIs much better than anything else, so neorepl
has a CLI that’s meant to be driven:
neorepl eval evaluates code on the project’s server. Calls share a
session, so *ns*, *1 and *e carry over from one call to the next, and
--session keeps separate lines of work apart.
--format json (or edn) gives you the values, the output and the
namespace as data.
--timeout interrupts the evaluation before giving up, so an infinite loop
doesn’t take the server down with it, and --max-output keeps a stray
(range) from eating the agent’s context.
neorepl check finds unbalanced parens, brackets and braces without a
server. That’s the most common way for an agent (or a human) to break a
Clojure file.
neorepl doc, source, apropos, complete, where, macroexpand and
reload do what the REPL’s commands do.
neorepl server starts a server that stays up, for editors, agents and
neorepl eval to share.
The exit codes are boring on purpose: 0 when all went well, 1 for an
evaluation error, 3 when there’s no server to talk to and 124 for a timeout
(like timeout(1)).
Playing nicely with nREPL
A new client is also a new way to break servers, or to be broken by them, and
the nREPL world has quite a few servers these days. Fortunately I’ve been
working on proof, a compatibility suite for
nREPL, which made neorepl a good test subject.
proof checks both sides of the conversation. proof proxy sits between a
client and a server and grades everything the client sends: every request has
an id, need-input gets answered with stdin in the right session, and so on.
proof serve is a small nREPL server that can behave like other servers in
very specific ways - sending only the last value of the code like Basilisp and
jank, dropping stderr, having no interrupt op, splitting output into one
message per character, and so on. I ran neorepl’s REPL and CLI through both,
against nREPL, babashka, Basilisp and jank, and through every quirk proof
serve knows about.
neorepl came out of it in decent shape, but not unscathed:
Code read from stdin (neorepl eval -) that then read *in* itself crashed
neorepl and left the evaluation waiting on the server for input that was
never coming.
neorepl’s errors could leave the shell’s prompt dangling at the end of a
line with servers that don’t end their error reports with a newline (hello,
jank!).
neorepl eval exited with 1 when it lost the server, which is the status
for code that failed.
All of those are fixed in 0.1. It also turned up a couple of things that are
nREPL’s fault rather than neorepl’s: need-input and the end of an
interrupted evaluation can go to the wrong connection when a session is used
from more than one. Those will be fixed in nREPL itself.
It works the other way around too. proof now checks servers with the exact
requests neorepl sends while starting its REPL and evaluating code, next to
the ones CIDER, Calva, Conjure, vim-fireplace and REPLy send, so a server that
breaks neorepl finds out from proof, not from a bug report. nREPL, babashka,
Basilisp and jank all pass those checks today.
Making the neighbours better
Building neorepl turned up a few problems elsewhere, which is one of my
favourite side effects of writing a new client. piggieback used to evaluate
only the first form you sent it and silently drop the rest, and it loaded the
ClojureScript compiler at startup even when nobody asked for a ClojureScript
REPL, which added almost half a second to the startup of every server with
ClojureScript on the classpath. The fixes for both
(piggieback#162 and
piggieback#161) will be in the
next piggieback release. The Node.js
REPL in ClojureScript itself doesn’t cope well with interrupted evaluations,
and that one is next on my list.
What’s next
0.1 is the foundation. Here’s what I’d like to tackle next:
a test runner with proper reports
a full-screen inspector and stacktrace browser
an MCP server and a ready-made skill for coding agents
It’s early days, so I’m sure you’ll find rough edges. Please file issues,
and tell me what you’d like to see next. And if you’re one of the people who
kept lein repl and REPLy going over the years - thank you! neorepl wouldn’t
exist without you.
What is going on?
