vim-slime

<!DOCTYPE html> <html> <head> <title>Tarn Barford</title> <meta charset="utf-8"/> <link rel="icon" type="image/x-icon" href="/favicon.ico"> <link href="/style.css" media="screen" rel="stylesheet" type="text/css" /> <link rel="alternate" type="application/atom+xml" title="Journals of Tarn Barford" href="/atom" /> <link href="/highlight.css" media="screen" rel="stylesheet" type="text/css" /> <link href="/highlight-console.css" media="screen" rel="stylesheet" type="text/css" /> </head> <body> <div id="container"> <div id="header"> <div id="header"> <p>From the <a href="/journal">Journals</a> of <a href="/">Tarn Barford</a></p> <h1> vim-slime </h1> <p> Mar 26, 2012 </p> </div> </div> <div id="post_content"> <html><body><p>Today I found the awesomeness that is <a href="https://github.com/jpalardy/vim-slime">vim-slime</a>, it's been an exciting day for me. <a href="http://common-lisp.net/project/slime/">Slime</a> is the "The Superior Lisp Interaction Mode for Emacs", I can almost hear the emacs crowd laughing.</p> <p>For those that use vim and haven't used Slime, vim-slime or <a href="https://github.com/vim-scripts/VimClojure">something similar</a>, this is why it's awesome:</p> <p><strong>Text can be sent from any process to the stdin of a <a href="http://www.gnu.org/software/screen/">gnu screen</a> or <a href="http://tmux.sourceforge.net/">tmux</a> session. The process in this case is vim and the screen/tmux session is a terminal</strong>.</p> <p>Screen is a <a href="/journal/oh-screen-where-have-you-been">really neat</a> terminal multiplexer (you can run multiple terminals in a terminal window). The multiplexed shell processes are children of the screen process, which itself is not a child of the terminal window process. This means a screen process and its child processes keep running if you close the terminal window. Later you can re-connect to it, this is what makes vim-slime possible.</p> <p>Here is an screen shot, on the left is me in gVim writing some awful Clojure <a href="#footnote-1">[1]</a>. On the right is a screen buffer in which I started a Clojure REPL. When I want to try run some code I can send any vim text selection to the REPL in a keystroke (or two).</p> <p><img alt="vim slime screenshot" src="screenshot.jpg"/></p> <p>It doesn't have to be a Clojure REPL either, we can send anything to a screen shell. We could run git commands, find, grep, sed, etc. Like with the Clojure REPL we can even interact with any terminal programs that use STDIN.</p> <p>This concept can be taken even further, You can even connect to a tmux session over SSH and share a terminal or a <a href="http://remotepairprogramming.com/remote-pair-programming-with-tmux-and-vim-the">terminal program like vim to do remote pairing</a>!</p> <p>Hopefully remote pairing is the topic of my next post as there are a couple geographically distant people I know who are keen to do some pair hacking. I stand to learn a lot!</p> <p><a name="footnote-1">[1]</a> I learnt almost everything I know about Lisp from <a href="http://www.ccs.neu.edu/home/matthias/BTLS/">The Little Schemer</a>. Great book.</p></body></html> </div> <div id="comments"> </div> </div> <div id="footer"> <p>&nbsp;</p> <p>Questions, comments, suggestions? Email me, <a href="mailto:tarn@tarnbarford.net">tarn@tarnbarford.net</a> (<a href="/pgp.txt">public key</a>)</p> <p>&nbsp;</p> </div> </body> </html>

Permalink

Swipe Keyboard

<!DOCTYPE html> <html> <head> <title>Tarn Barford</title> <meta charset="utf-8"/> <link rel="icon" type="image/x-icon" href="/favicon.ico"> <link href="/style.css" media="screen" rel="stylesheet" type="text/css" /> <link rel="alternate" type="application/atom+xml" title="Journals of Tarn Barford" href="/atom" /> <link href="/highlight.css" media="screen" rel="stylesheet" type="text/css" /> <link href="/highlight-console.css" media="screen" rel="stylesheet" type="text/css" /> <style> #swipe-canvas { position: relative; width: 900px; height: 300px; } #swipe-results { font-size: 30px; padding-left: 50px; padding-left: 50px; } #swipe-results ul { margin: 0px; padding: 0px; } #swipe-results li { float: left; background-color: #DDDDDD; list-style-type: none; padding: 10px; margin: 5px; border-radius: 5px; } #swipe { position: relative; } #swipe-loading { position: absolute; height: 50px; width: 300px; top: 85px; left: 300px; background-color: darkgray; border-radius: 10px; text-align: center; padding-top: 20px; border: black; border-width: 5px; } </style> </head> <body> <div id="container"> <div id="header"> <div id="header"> <p>From the <a href="/journal">Journals</a> of <a href="/">Tarn Barford</a></p> <h1> Swipe Keyboard </h1> <p> Apr 06, 2014 </p> </div> </div> <div id="post_content"> <html><body><p>When I first tried a <a href="http://www.swype.com/">Swype</a> keyboard I was impressed how effective it was. Even though I don't use the feature on my phone I was interested in how it could be built, so I <a href="https://github.com/tarnacious/swipe-keyboard">implemented this otherwise useless swipe-able keyboard</a> below. It probably doesn't work on mobile devices, but works on modern browsers with mouse pointers (although I've only really tried Chrome and Firefox).</p> <div id="swipe"> <canvas height="300px" id="swipe-canvas" width="900px"></canvas> <div id="swipe-results"></div> <div style="clear: both"></div> <h2 id="swipe-loading">Loading<noscript>Javascript is Required</noscript></h2> </div> <p>I initially tried to solve this using the technique Peter Norvig famously uses in his <a href="http://norvig.com/spell-correct.html]">spell checker</a>. He takes a sequence of characters and generates a set of word candidates by adding, removing and swapping characters in the original sequence, the generated candidates are removed if they are not found a dictionary. This can work but to be effective too many combinations need to be generated.</p> <p>If the dictionary is indexed into a <a href="http://en.wikipedia.org/wiki/Trie">trie</a> the number of combinations generated can be reduced significantly by traversing the trie and only generating valid letter combinations. This is a pretty bare implementation of that, it requires: </p> <ul> <li>The first and last characters of the initial sequence are used </li> <li>Intermediate characters in the initial sequence can be repeated or discarded </li> <li>No characters are added or swapped</li> </ul> <p>Basically, if you swipe through all the characters in a word in order, then the word will be found if it is in the index regardless how many characters are swiped in between. It is surprisingly quick and effective.</p> <p>This implementation uses <a href="https://raw.github.com/first20hours/google-10000-english/master/google-10000-english.txt">these 10000 words</a>, I intended to use digital books but never got around to it as these words demonstrate the concept well enough.</p> <p>This is the first thing I've written in <a href="https://github.com/clojure/clojurescript">ClojureScript</a> or <a href="https://github.com/clojure/clojurescript">Clojure</a>, so my code my vary from non-idiomatic to shamblolic. I initially used a <a href="http://clojuredocs.org/clojure_core/clojure.zip/zipper">zipper</a> to build the trie with immutable data structures, but found the indexing took to long with my zipper implementation so I <a href="https://github.com/tarnacious/swipe-keyboard/commit/6edd7b26e78121fbe8586b3f0ef54ca8277d9e32">switched to using native Javascript maps</a>.</p> <p>I found that <a href="https://github.com/clojure/core.async">core.async</a> library is really awesome, the <a href="http://docs.closure-library.googlecode.com/git/index.html">Google closure library</a> and <a href="https://developers.google.com/closure/compiler/">compiler</a> integration with <a href="http://leiningen.org/">Leiningen</a> the <a href="https://github.com/emezeske/lein-cljsbuild">cljsbuild plug-in</a> to be impressive. My main pains were the slow JVM start-up time, the advanced closure compiler build of the web worker script fails silently when run (but the main script works fine when compiled with the advanced compiler), and at times I felt some compile time type checking would be nice.</p> <p>I would like to extend this experiment to index the word occurrence counts and proceeding word counts in original text and rank the found words as most likely. Support casing, umlauts, special characters, spelling correction and compound words in the indexing and lookup. I think a live lookup while swiping would also be possible.</p> <p>Overall this was fun, turned out OK I think, and was a great learning experience.</p></body></html> </div> <div id="comments"> </div> </div> <div id="footer"> <p>&nbsp;</p> <p>Questions, comments, suggestions? Email me, <a href="mailto:tarn@tarnbarford.net">tarn@tarnbarford.net</a> (<a href="/pgp.txt">public key</a>)</p> <p>&nbsp;</p> </div> <script src="swipe.js" type="text/javascript"></script> </body> </html>

Permalink

BudgetSWU: my latest pile of code

I made a little web app called BudgetSWU. It recommends decklists for a trading card game called Star Wars: Unlimited. I started playing SWU almost a year and a half ago because I had a dream I was in a mall opening a pack of Yugioh cards--another trading card game that I played as a kid--and it gave me enough nostalgia to do some googlin' and discover SWU.

As an adult I now have lots more money to buy cards than I did as a kid (yay) but much less time to play with them (boo). So I don't spend much time building my own decks; I just find decklists on the internet and buy the cards for those.

BudgetSWU is now one of those places on the internet. It gets decklists from melee.gg which is used in tournaments (large and small), including each decklist's (player's) placement in the tournament. It also gets pricing data for all the cards in those decklists from tcgcsv.com, which in turn aggregates pricing data from TCGPlayer, the ebay of trading cards (it is owned by ebay, in fact).

As a final bit of data, I compute "popularity" by taking each deck's "archetype" (the leader + base, for those in the know--except that I'm actually using just base color) and seeing how many decklists in the entire corpus have that same archetype. It's cooler to play decks that not everyone else is already playing, know what I mean?

Then BudgetSWU filters all the decklists down to a handful of recommendations by computing a Pareto Front. The idea here is that we have multiple factors with which we can rank decks (cost, performance, popularity/nicheness) and we don't necessarily know which factor is most important (if deck A costs 50% more than deck B, how much better/niche does deck A have to be for it to be a better recommendation than deck B?). But we do know that if deck A beats deck B on all three of those factors, then deck A is probably the better recommendation. So we filter out all the decks that have a clearly better alternative, and then we tell the user "here's what's left; you decide." As of writing, that ends up being less than 20 decks, which is a very manageable number for a user to scroll through.

The next cool bit is that if you make an account, you can enter what cards you already own, and then the recommendations will be computed based on the cost of only the cards you don't have yet. Entering in all your cards would be a pain since players old enough to have jobs usually have thousands (of cards, not jobs). So BudgetSWU sorts all the cards by cost, most expensive first. (Right now the most expensive cards go for about $30-ish a piece). You can start from the top and enter quantities until you got to cards that cost, say, a dollar each or whatever you feel like, which doesn't take long.

Finally: whenever a new decklist makes it into your personalized recommendations (tournaments happen every week), you get a notification email. Booyah.

I generated the code in Clojure. It uses my web framework Biff. I've recently been working on some LLM skills that provide guidance for writing Biff code that doesn't suck. And I have to say, I'm fairly pleased with the results. The code still isn't anything I would print out and frame on my wall, but the overall structure is generally inline with how you're supposed to write Biff apps. If this was an app I planned to develop further (it's not), I would be happy with the state the code is in as a starting point.

If you're into specifications, you can read the one for BudgetSWU over here. i.e. that page is the thing I actually typed, mostly.

Permalink

Using AI to analyse legacy systems faster: what our pilots revealed

Futuristic in Industry 4.0 and business virtual diagram with Ai, robot assistant, Cloud, big data and automation. two colleague web designer discussing data and digital tablet docking keyboard.

AI can help organisations analyse legacy systems faster 

One of the biggest barriers to modernising legacy technology is understanding how existing systems work.   

To modernise systems, organisations first need a clear picture of how applications work, connect and depend on one another. That process can take weeks or even months. Documentation is often incomplete or out of date, knowledge may be concentrated in a small number of specialists, and some systems rely on technologies that few engineers still work with regularly. 

Our pilots indicate that AI-assisted approaches could help technical teams analyse some legacy systems more quickly. 

Across participating organisations, activities estimated to take weeks using traditional approaches were completed in hours. 

Working with organisations including the Student Loans Company (SLC), the UK Hydrographic Office (UKHO), the Met Office, and our delivery partner Kainos, we found that AI can help teams analyse unfamiliar systems, identify dependencies and build technical knowledge more quickly than traditional approaches alone. The pilots covered different systems and technical environments, so the results should not be treated as directly applicable to every legacy estate. 

However, for organisations managing large and complex technology estates, this could make some modernisation activities easier to prioritise, plan and deliver.  

What happened at the Student Loans Company 

At the Student Loans Company, teams explored how AI assisted approaches could be used to analyse an estate of 33 applications written in Clojure and support future modernisation planning. 

Using traditional approaches, the work was estimated to take around five weeks. During the pilot, teams completed the analysis in less than a day. 

The analysis gave the team a clearer view of the 33 applications and their dependencies, which it used to inform migration planning and work towards supported versions of Java. 

The most significant finding was not the reduction in effort itself, but the ability to unlock work that had previously been difficult to prioritise.  

What happened at the UK Hydrographic Office 

At the UK Hydrographic Office, teams focused on a 20-year old desktop application that previously had no automated tests. 

AI assisted techniques were used to generate automated tests for the application. Test coverage increased from 0% to 96%, while regression testing that had previously required around 15 days of manual effort was reduced to approximately 30 minutes. 

The benefits extended beyond faster testing. 

Teams reported greater confidence in making changes to the application safely and maintaining it over time. For many legacy systems, the perceived risk of change can be a barrier to improvement. The pilot indicated that AI-assisted testing could give engineers more evidence when assessing proposed changes. 

Why this matters 

The examples from SLC and UKHO illustrate a wider challenge facing organisations across the public sector. 

Legacy systems often support critical services, but analysing them can be complex and expensive. Teams may spend weeks mapping dependencies, reviewing code and building technical knowledge before they can assess risks or plan change. 

The strongest finding from the pilots was that AI can help accelerate this discovery and assessment phase. That may sound modest, but it addresses a critical point where many modernisation efforts slow down or stall. 

By helping teams understand systems more quickly, AI assisted approaches can make it easier to: 

  • assess risks 
  • identify system dependencies 
  • prioritise investment 
  • improve technical documentation 
  • develop modernisation plans 

In short, AI can help organisations spend less time understanding their systems and more time improving them. 

AI is not a substitute for good delivery 

The pilots also reinforced an important lesson. 

AI does not remove the need for good governance, engineering expertise or delivery capability. 

Successful outcomes depended on organisations having the right foundations in place, including clear ownership, access to systems and information, experienced technical teams and established delivery processes. 

In practice, the pilots showed that AI amplified existing strengths and weaknesses rather than replacing them.  

Used well, AI can accelerate analysis, improve visibility of complex systems and help organisations make progress more quickly. Used without appropriate controls, it can amplify existing problems and accelerate poor decision making. 

Where AI-assisted analysis may help 

The pilots suggest that AI can significantly reduce the effort required to assess complex systems, helping organisations move more quickly from discovery to modernisation. 

For organisations managing large and complex technology estates, this can help engineers, architects and delivery teams to identify risks earlier, making modernisation activities easier to plan, justify and deliver.  

What happens next 

We are continuing to work with organisations using AI assisted approaches to analyse, assess and modernise legacy systems, building on the lessons from these pilots. 

Alongside this work, we are developing a playbook of good practice that captures practical approaches, lessons learned and implementation considerations for organisations looking to apply AI to legacy technology challenges. The playbook is currently undergoing user testing, review and approval ahead of publication. 

Get involved 

If your organisation is exploring AI assisted approaches to analysing, documenting or modernising legacy systems, or would like to share experiences and engage with our wider Firmer Foundations programme, we would like to hear from you. Get in touch at firmer.foundations@dsit.gov.uk 

Permalink

Best Clean Code and Refactoring Books, and the Order to Read Them

Clean Code was the first text I handed to my students. Since then I have read the six other books fighting for the "clean code" shelf, and they do not all say the same thing. Sometimes they say the opposite.

I wrote a reading note for each one. Here is the order I would follow if I were starting over, and what you can leave on the shelf.

The craft before the rules

The Pragmatic Programmer by Thomas and Hunt (2019 edition, 8/10) is my way in. What strikes you first is the tone: few commandments, many stories, never a condescending look. And this edition publicly corrects the previous one. The Law of Demeter is demoted, DRY (do not duplicate a piece of knowledge) is clarified because people misread it, and on security the authors write "we were wrong". A book that owns its mistakes twenty years later earns trust for everything else.

The limit is mechanical: 53 topics in 352 pages means skimming. The tooling chapter is thin, and the code samples jump from Ruby to Elixir to Clojure without digging into any of them. You read it for the stories and mental models, not the code. Skip the samples, keep the rest.

The dogma and its counterweight

Clean Code by Robert C. Martin (2008, 6.8/10) taught a generation that code is read ten times more than it is written. The chapters on naming, functions and comments have not aged a day. The rest is verbose Java, and the last three case-study chapters are 90 pages of listings nobody reads anymore, me included. One thing the fans forget: Martin himself admits that no research backs the short-functions rule, and that his own functions start out long and messy before being refined under tests. The "three lines maximum" dogma comes more from his readers than from the book.

To hear the other bell, read A Philosophy of Software Design by John Ousterhout (2021, 8/10). A hundred and ninety-six pages, and in my eyes the best software design book of the last twenty years. His deep module idea changed how I judge a class: I stopped counting lines and started comparing what the interface costs to learn with what it gives back. The examples circle around two or three cases (a text editor, RAMCloud) and smell like a lecture hall. And "make your modules deep" is easier to understand than to apply. Where the two books contradict each other, on function size and on comments, Ousterhout is almost always right.

The first hundred pages of construction

Code Complete by Steve McConnell (2004, 960 pages, 7.4/10) is the book I wish I had read at the start of my career, but for its first hundred pages, not its 960. The thesis on complexity, abstract data types, cohesive routines, defensive barricades: you usually learn all of that by trial and error, after humbling code reviews. The book shortcuts that painful education.

What has aged is visible: samples in C++, Visual Basic 6 and Pascal, "code tuning" chapters about processor pipelines, a tools catalogue straight from a museum. What holds up is everything about humans: the numbers on code inspections, the psychology of debugging, complexity as the universal enemy. You do not need all of it. Parts I, II and VII plus chapters 20 and 21 make roughly 250 pages, and the rest is reference material you open now and then.

Refactoring and legacy, the two moves

Refactoring by Martin Fowler (2018, 8.2/10) holds the best code lesson I have ever read. Chapter 1 is seventy pages of watching someone work, every move explained, the code never broken. Fowler has a rare honesty: his central argument is labeled "unproven hypothesis". The limit is that half the book is a dictionary of transformations nobody reads in one sitting. Read chapters 1 to 4 like a novel and keep the rest within reach during code review. The Refactor menu in your editor exists because of this book.

Then comes Working Effectively with Legacy Code by Michael Feathers (2004, 7/10), which redefines the word: legacy is not old code, it is code without tests. His characterization tests (which snapshot current behavior) answer a question nobody phrases well: how do you start testing when you do not know what "correct" means for this code? The seam model (places where you can change behavior without editing the code) teaches you to read code by looking for joints. Feathers also admits that his techniques often make code uglier in the short term, and that the scar is temporary. That frankness makes the book reliable.

It is dated, and every example is Java, C++ or C. Part II, twenty chapters in FAQ format, often repeats the same four moves under a different class name. Open it at the chapter matching your problem of the day.

Test first, to lose the fear

Test-Driven Development: By Example by Kent Beck (2002, 240 pages, 7.6/10) does not sell a recipe, it shows a state of mind: the calm of knowing, every second, that the code still works. The fear of breaking things turned into quiet boredom. Since then I no longer write tests to "have coverage", I write them to move forward without dread.

Part I, the money example, drags: you repeat the same micro-cycle dozens of times and you got it long before. The code is 2002 Java and Python 2. TDD is still debated, and Beck is the first to say no study settles it. Read it for the mindset, keep the cycle, the TODO list and "fake it" when you are stuck, and adapt the micro-steps to your own pace.

The order I would follow

Book

Pages

Rating

What it is for

1

The Pragmatic Programmer

352

8

The craft's mental models

2

Clean Code

464

6.8

Naming, functions, comments

3

A Philosophy of Software Design

196

8

The counterweight: deep modules

4

Refactoring

448

8.2

Chapters 1 to 4, then the catalog as needed

5

TDD by Example

240

7.6

Moving forward without fear of breaking

6

Working Effectively with Legacy Code

440

7

Testing code that never had tests

as needed

Code Complete

960

7.4

The useful 250 pages, then reference

Conclusion

The book I used to hand my students has the lowest rating on the list, 6.8/10. And it is Fowler and Feathers who hold the moves you actually use on a Monday morning, in code you did not write.

If you want to see these moves applied to a real project, I wrote about the order of a legacy Symfony audit.

I stitched the marrow of dozens of tech books into a single thread, from bits and Big O up to the AI era: that is The Developer's Great Book. Four of these books (Clean Code, A Philosophy of Software Design, The Pragmatic Programmer, Refactoring) are melted into it at the "Writing for humans" level, and three others (TDD by Example, Working Effectively with Legacy Code, Code Complete) at the "Prove it, ship it, hold the load" level.

📚 Other comparisons. Best DevOps and Engineering Team Books, and the Order to Read Them and Best Event Sourcing and CQRS Books, and the Order to Read Them.

Permalink

AI Is a Tool. Using It Well Takes Skill.

In his October 11, 2026 article, "'AI' is Not a Tool," Rich Hickey argues that LLMs are automation rather than tools because they cannot be wielded. I agree that outsourcing work can cost us opportunities to learn. But his conclusion does not follow. Even by his definition, an LLM can be something a human wields. And using it well takes skill, just as using other tools does.

Picking up a saw does not guarantee a straight cut. Holding a chisel does not make you a sculptor. Tools can amplify poor judgment as readily as good judgment. Their usefulness depends on the operator's ability to direct them, interpret feedback, recognize mistakes, and make corrections. Those requirements also apply to LLMs.

One of my colleagues used to say, "AI is a mirror." I find that insightful: it reflects who you are, including the curiosity, judgment, and habits you bring to it. The questions you ask and the standards you apply to its answers matter.

An LLM's unpredictability is a meaningful limitation. But predictability, reliability, simplicity, and bounded function are qualities by which we evaluate tools, not prerequisites for membership in the category. A tool does not stop being a tool because it is difficult to control or easy to misuse.

Hickey's distinction between tools and automation also creates a false division. Automation can be a feature of a tool. A compiler performs substantial work without the programmer continuously directing each operation. A spreadsheet recalculates results without someone manually carrying out every calculation. We still wield these systems by specifying inputs, examining results, and deciding what to do next. Why should generating text automatically disqualify an instrument from being a tool when translating code or calculating numbers automatically does not?

The claim that LLM use lacks intention, feedback, direction, and expertise is especially difficult to defend. A programmer can ask an LLM to explain unfamiliar code, challenge that explanation, inspect the source, test a proposed change, reject it, and refine the request. That is a feedback loop directed by a human pursuing a purpose. Doing it well requires knowing what to ask, what evidence matters, and when the answer is wrong. Expertise can make the instrument more useful.

Hickey's concern about reading is valid for a particular use: accepting a summary in place of reading a paper can sacrifice much of what reading teaches. But someone can also read the paper and use an LLM to question their interpretation, explore an objection, or identify something to examine more closely. The possibility of avoiding intellectual work does not establish that every use avoids it. The relevant question is what the person actually does.

What disappoints me the most is the reliance on the AI industry's anthropomorphic vocabulary: "go rogue," "escape containment," "conspire." Repeating those expressions risks accepting the framing that makes these systems sound like independent beings. Software can produce unexpected outputs, fail to follow instructions, or cause harm when given access to consequential actions. Those are real engineering problems. Calling them "conspiracy" adds a suggestion of human motives without establishing any.

I would not claim that an LLM always does exactly what a human tells it to do. It plainly can fail. But failure to obey perfectly does not make it cease to be a tool, any more than successful use proves that its operator has surrendered their judgment.

The useful distinction is between wielding an instrument with skill and delegating work without adequate understanding or oversight. That distinction applies to AI, and it gives us something more practical than declaring that an entire class of software cannot be a tool.

Permalink

Clojure OSS update

It's been a while since I worked on my Clojure libraries, so I would like to give a little update on that. Two of them got a new release, and one got a debut release on Clojars.

clj-pnm [1]

Apart from the minor bug fixes and Clojure(Script) dependency update, the most exciting thing is that the Clojure dialect support matrix has grown. clj-pnm was built from the start as a cross-dialect library supporting both Clojure and ClojureScript. It is lightweight enough to expand that.

For the Clojure side, the library is compatible with Babashka, as it can be called from Babashka code and can be used as a Babashka dependency.

On the ClojureScript side, clj-pnm can target NodeJS and the browser as before. Switching from figwheel to shadow-cljs for the testing framework was integral in ensuring that. There is a special test runner for testing Squint compatibility, but with no code changes, this lightweight dialect of ClojureScript is also added to the compatibility matrix.

There are tests for each of these in the library repository[2].

cljs-map-pins [3]

cljs-map-pins got a maintenance release, so no new features yet. But this post can serve as a reminder of how fun it is to use this library (at least for me :-D). I love visualizing things on a map, and this library is made just for that. Two examples I like to refer to, albeit they are made with a much older version of the library, are The Warriors path through New York City[4] and Big Wayne tribute[5].

See the repository[6] for more details.

defns [7]

defns is a proof-of-concept library that provides a way for defining functions with type signatures for which :pre and :post conditions are generated. This is an experimental idea and should not be used for anything serious, but I've seen some interest in it and decided to release it to Clojars. The fun part was writing the library clj-kondo configuration and custom hooks for the macros. When using defns as a dependency, make sure to import the clj-kondo configuration for the linting to work.

See the repository[8] for more details.

Enjoy!

[1] clj-pnm

[2] clj-pnm repository

[3] clj-map-pins

[4] The Warriors path through New York City

[5] Big Wayne tribute

[6] clj-map-pins repository

[7] defns

[8] defns repository

Permalink

Week Notes 2026.41

Babashka tasks for deploying Datomic Cloud ions; making AWS API Gateway requests with SigV4 authentication.

Permalink

neorepl: A Terminal REPL for Clojure (and Friends)

For more than a decade the Clojure REPL most people saw in a terminal was REPLy, the one behind lein repl. It served us well, but it’s showing its age, and I kept wishing for a terminal REPL that felt like CIDER, without the Emacs part. So I finally built one. neorepl 0.1 is out today!

Why another REPL?

There’s no shortage of Clojure REPLs, so it’s fair to ask. Here’s what I was after:

  • nREPL all the way down. neorepl only speaks nREPL, so it works with anything that runs an nREPL server: Clojure, babashka, shadow-cljs, Basilisp, ClojureCLR and so on. The smarts live on the server. When cider-nrepl is there neorepl uses it, the same way CIDER does, and when it isn’t it falls back to nREPL’s built-in ops, and then to evaluating a bit of code.
  • A REPL that’s pleasant to use: a proper line editor, highlighting, completion, arglists as you type, pretty printing, history per project and a Ctrl-C that actually interrupts the evaluation.
  • A CLI that’s just as good for scripts and coding agents. Every lookup the REPL can do is also a command, with structured output and exit codes that mean something.
  • Something small and modern. Clojure 1.10+, nREPL and JLine 4 are the only dependencies, and the same code runs on the JVM and on babashka. The whole thing is about 3,300 lines right now, and I’d like to keep it that way.

rebel-readline and bb repl --connect are both great, and I borrowed from them freely, but neither tries to be an nREPL-first tool with CIDER’s tooling, jack-in and a CLI for agents. That combination is the whole point of neorepl.

Update (2026-10-11): A few people asked how neorepl relates to the Clojure CLI REPL, which the Clojure team released a couple of weeks ago, and to bb repl --connect. (I’m a bit embarrassed to admit I hadn’t heard of the former until after I announced neorepl.) Funny enough, all three are nREPL clients under the hood, but they’re after pretty different things. In practical terms:

  • The Clojure CLI REPL runs only on the JVM and needs Clojure 1.12+. It ships its own nREPL server, and its docs and data inspector are ops of that server, so against any other server you can only evaluate code. It’s deps.edn only. Its editing is really nice, though - structural editing, evaluating the form at the cursor and a datafy-based inspector - and I’ll be borrowing some ideas from it. It doesn’t do completion yet.
  • bb repl --connect puts babashka’s console REPL in front of any nREPL server, with completion, arglists and docs, in very little code. It’s a great way to poke at a running server, but it won’t start one for you.
  • neorepl works with any nREPL server and gets the most out of the ones with cider-nrepl (or babashka’s take on it), without any middleware of its own. It starts a server for Leiningen, deps.edn, babashka and shadow-cljs projects, follows you into ClojureScript, and everything the REPL does is also a command with JSON output for scripts and agents. No structural editing yet, however.

So, if you want the fanciest line editing and you live in deps.edn land, give the Clojure CLI REPL a try. If you want CIDER in your terminal, for you and for your agents, that’s what neorepl is for. The neat part is that babashka’s server bundles CIDER’s inspector and test runner ops these days, so neorepl’s test runner (already on main) and the inspector that’s coming next work with babashka out of the box.

For humans

Here’s the REPL in action:

The neorepl REPL

A few things worth pointing out:

  • Enter only submits the input once its forms are complete, so pasting a whole function just works.
  • TAB completes, with the candidates’ types next to them, and the arglist of the function you’re calling shows up after the cursor.
  • Values get pretty printed on the server and highlighted on the client.
  • Commands start with a comma, the CIDER way: ,doc, ,source, ,apropos, ,macroexpand, ,pst, ,reload, ,edit (which opens a definition in your $EDITOR) and ,help for the rest. The Clojure reader sees a comma as whitespace, so they can’t clash with real code.
  • Ctrl-C interrupts what’s running, and code that reads *in* gets its input, something REPLy never quite got right.

Without a server to connect to, neorepl starts one for the project, the way CIDER’s jack-in does, with cider-nrepl on the classpath, and stops it when you exit. It knows Leiningen, the Clojure CLI, babashka and shadow-cljs projects.

ClojureScript works too. Start a ClojureScript REPL in the session (through shadow-cljs or piggieback, which jack-in adds for you) and neorepl follows along: evaluation, completion, docs and the rest switch to ClojureScript, and :cljs/quit takes you back.

For agents

Coding agents are surprisingly good at Clojure, as long as they have a REPL to talk to. Most of them drive CLIs much better than anything else, so neorepl has a CLI that’s meant to be driven:

neorepl's CLI

  • neorepl eval evaluates code on the project’s server. Calls share a session, so *ns*, *1 and *e carry over from one call to the next, and --session keeps separate lines of work apart.
  • --format json (or edn) gives you the values, the output and the namespace as data.
  • --timeout interrupts the evaluation before giving up, so an infinite loop doesn’t take the server down with it, and --max-output keeps a stray (range) from eating the agent’s context.
  • neorepl check finds unbalanced parens, brackets and braces without a server. That’s the most common way for an agent (or a human) to break a Clojure file.
  • neorepl doc, source, apropos, complete, where, macroexpand and reload do what the REPL’s commands do.
  • neorepl server starts a server that stays up, for editors, agents and neorepl eval to share.

The exit codes are boring on purpose: 0 when all went well, 1 for an evaluation error, 3 when there’s no server to talk to and 124 for a timeout (like timeout(1)).

Playing nicely with nREPL

A new client is also a new way to break servers, or to be broken by them, and the nREPL world has quite a few servers these days. Fortunately I’ve been working on proof, a compatibility suite for nREPL, which made neorepl a good test subject.

proof checks both sides of the conversation. proof proxy sits between a client and a server and grades everything the client sends: every request has an id, need-input gets answered with stdin in the right session, and so on. proof serve is a small nREPL server that can behave like other servers in very specific ways - sending only the last value of the code like Basilisp and jank, dropping stderr, having no interrupt op, splitting output into one message per character, and so on. I ran neorepl’s REPL and CLI through both, against nREPL, babashka, Basilisp and jank, and through every quirk proof serve knows about.

neorepl came out of it in decent shape, but not unscathed:

  • Code read from stdin (neorepl eval -) that then read *in* itself crashed neorepl and left the evaluation waiting on the server for input that was never coming.
  • neorepl’s errors could leave the shell’s prompt dangling at the end of a line with servers that don’t end their error reports with a newline (hello, jank!).
  • neorepl eval exited with 1 when it lost the server, which is the status for code that failed.

All of those are fixed in 0.1. It also turned up a couple of things that are nREPL’s fault rather than neorepl’s: need-input and the end of an interrupted evaluation can go to the wrong connection when a session is used from more than one. Those will be fixed in nREPL itself.

It works the other way around too. proof now checks servers with the exact requests neorepl sends while starting its REPL and evaluating code, next to the ones CIDER, Calva, Conjure, vim-fireplace and REPLy send, so a server that breaks neorepl finds out from proof, not from a bug report. nREPL, babashka, Basilisp and jank all pass those checks today.

Making the neighbours better

Building neorepl turned up a few problems elsewhere, which is one of my favourite side effects of writing a new client. piggieback used to evaluate only the first form you sent it and silently drop the rest, and it loaded the ClojureScript compiler at startup even when nobody asked for a ClojureScript REPL, which added almost half a second to the startup of every server with ClojureScript on the classpath. The fixes for both (piggieback#162 and piggieback#161) will be in the next piggieback release. The Node.js REPL in ClojureScript itself doesn’t cope well with interrupted evaluations, and that one is next on my list.

What’s next

0.1 is the foundation. Here’s what I’d like to tackle next:

  • a test runner with proper reports
  • a full-screen inspector and stacktrace browser
  • an MCP server and a ready-made skill for coding agents
  • better Windows support

Try it

The easiest way to install neorepl is bbin:

$ bbin install io.github.nrepl/neorepl
$ cd my-project
$ neorepl

The README has the details.

It’s early days, so I’m sure you’ll find rough edges. Please file issues, and tell me what you’d like to see next. And if you’re one of the people who kept lein repl and REPLy going over the years - thank you! neorepl wouldn’t exist without you.

Keep hacking!

Permalink

Clojure 1.13.0-alpha9

Clojure 1.13.0-alpha9 is now available! Find download and usage information on the Downloads page.

  • Dropped :defaults directive - use cases better handled by others now

  • CLJ-2983 Removed :defaults tests

  • Faster transient propagation of metadata in into and update-vals

  • Revise impl and inline of meta via RT

  • CLJ-2978 Catch protocol extension of non-protocol methods

  • CLJ-2982 TransientArrayMap should expand to same threshold as PersistentArrayMap

  • Favor class files when reproducible build sets class and source timestamps the same

  • CLJ-2984 Set file timestamps to last git commit time

Try it out

Update your deps.edn :deps with:

org.clojure/clojure {:mvn/version "1.13.0-alpha9"}

Start a REPL with the Clojure CLI (any version) with:

clj -Sdeps '{:deps {org.clojure/clojure {:mvn/version "1.13.0-alpha9"}}}'

Permalink

Workflow Ergonomics with AI

How AI streamlined my digital life

Build your own web server

My road to a more productive digital life started when I asked Claude Code (CC) to build a web server for me. It&aposs called mdserve (short for markdown server), a full-stack web application in a single file, that has grown into the control panel for my whole desktop. Its generic core is on GitHub, for you to build your own on.

It started with wanting a more pleasant reading experience than what CC produces. Instead of reading a long answer inside CC itself, I tell it to write the answer to a markdown file and open it in the browser for me. CC will do that without any help. The problem mdserve solved was where the page turns up: I wanted it to always be in a known browser&aposs window, on the workspace I&aposm working in.

It has grown since then. A "personal agent", one that goes off and does things for you while you&aposre not there, isn&apost something I want. But I do like the convenience such agents promise, and this is how I get it. Anything I find myself doing a lot gets streamlined. Sometimes that&aposs a small ~/bin command, sometimes a button or page in mdserve, and sometimes just a rule CC follows for me, such as always backing up a file before changing it.

When I start mdserve from the command line, it opens several browser windows, each with many tabs. I have three areas of focus at the moment, so I work across three workspaces. The one step I haven&apost been able to automate is moving each window to its workspace, but that only takes a few seconds.

About the videos: they&aposre screen recordings, and the text in them is small. Open a panel, then go full screen before you press play. YouTube picks the picture quality from the size of the player, so a small player gets a blurry picture.

Watch: mdserve starting up

Why a browser and localhost

Why a web server and not a desktop application? Because everything talks HTTP. CC shows me a page with a single curl to mdserve, with no plugins or plumbing needed. mdserve in turn drives the browser with web requests too, through the Chrome DevTools Protocol. The server holds the state, so closing a tab loses nothing. And it isn&apost tied to one machine: CC sends documents to my tablet over the LAN with the same kind of request.

Everything lives on a tab

As you saw in the video, I start mdserve.sh from workspace 2. That&aposs the workspace I use for control, and for anything that isn&apost a programming project. The idea is that mdserve bootstraps the machine. After that, all the work I need to do is on an mdserve browser tab somewhere.

When I want to work on mdserve itself, I start a new CC session and ask it to read MDSERVE.md, where its documentation lives. If you haven&apost worked with CC, or a similar coding agent such as Codex CLI or Gemini CLI: maintenance documents like this are written by the AI itself, from the conversations you have with it at its prompt.

Watch: Improving mdserve with Claude Code

Browser tabs are just URLs. On startup mdserve reads them from an EDN file, and that file gets changed during my conversations with CC.

It&aposs interesting to see how tabs come to be there. There are three ways:

  1. Always wanted. These are often complete programs, such as the dashboards for my projects, or ordinary sites such as Gmail and Slack, and they come back every time mdserve starts.
  2. Asked for. I&aposve asked CC to "put it up". This is usually markdown documentation.
  3. Opened by another tab. A button on another tab was pressed, usually to take a closer look at something, such as a log.

Each tab also gets a coloured dot as its icon, so I can tell at a glance what kind of tab it is. Ordinary websites keep their own icons, but they&aposre just as much part of mdserve: remembered and reopened with the rest, and CC can put one up in the correct window as easily as a document. The idea is to keep it down to one browser window per workspace.

More features, and your own Windows Explorer

The last video showed "Refresh" for how much memory is in use, and "Remember Tabs", which saves the open tabs.

One rule Claude follows for me is to always make a backup before changing a markdown file. That makes a mess of backup files, which of course can be tidied up.

The next video shows the tidying up.

Watch: mdserve's Explorer tidying up backups

Imagine having your own programmable version of Windows Explorer per workspace. You just saw that in the last video! There&aposs always at least one of these tabs in each workspace. The video doesn&apost show it, but clicking a file opens a new tab that displays that type of file. Markdown files matter most, so they&aposre listed with the most recently changed first.

Many of mdserve&aposs buttons don&apost do the work themselves. They run a small command from my ~/bin directory, the same command I could type at a shell prompt, or that CC can run for me. The "Flotsam" button in the video runs archive-flotsam, a script that moves backup files and screenshots into a timestamped archive folder. Most of the commands behind buttons are just a few lines of bash. A few buttons, such as the memory "Refresh" and "Remember Tabs", are built into mdserve itself.

Fewer terminals

Working as a developer or a support person, you end up with terminals everywhere, some temporary and some running permanently, and they clutter your desktop. There are plenty of ways to tame them: tmux, tiling window managers such as i3, and on macOS, iTerm2.

The terminals that cluttered my desktop were mostly long-running processes: servers, build watchers and the like. Those now run in the background, each terminal having been replaced by a row of buttons ("Launch", "Stop" and "Log") on an mdserve dashboard, one at a time or a whole group at once. "Log" opens the output in a tab, which I close when I&aposm done. All I keep is one tabbed terminal window per workspace, for my CC sessions and the occasional one-off terminal session.

Use runbooks

That occasional one-off terminal session I mentioned is often a runbook, often on another machine.

Some jobs need me, rather than CC, to type the commands, because they need my password or a console. For those, CC writes a runbook: a plain file of shell commands, with comments saying why each step is there. I press Alt-n and the next step appears at my prompt, already typed, and nothing runs until I press Enter. No more copying commands out of the chat. The tool is called cmdq, and it&aposs a gist you can install in a minute.

Each step&aposs output is saved for CC to read back, and my place is saved too, so a job can span reboots. When the job is on another machine, CC copies the runbook there, and I log in and keep pressing Alt-n. Each runbook is a dated file kept with its output, so there&aposs a history of what ran on each machine.

At this stage I&aposve written the main points I wanted to make. Please read on if you are still curious...

Complex deployment

Buttons aren&apost only for small jobs. Our point-of-sale system (POSES) is four webapps (chat, till, prods and pos) that run on one production server, and deploying them is a job with several steps. It now lives in mdserve in a Deploy panel on the POSES dashboard tab, as five buttons in the order they&aposre used:

  1. Plan compares what&aposs live with the latest code (whether committed or not), app by app, and shows what a deploy would ship.
  2. Ship builds and uploads an uberjar for each changed app, stopping at the first failure.
  3. Release doesn&apost actually do anything! Instead, it prints the step for the human deployer to run on the server.
  4. Record checks that every app that was shipped is really live, and fails loudly if one isn&apost, which catches a step missed on the server. It then notes what went live, so the next Plan can work from that note instead of having to query the live server. It&aposs also the only way Plan knows which version of the shared library is in production, because the server doesn&apost say.
  5. Confirm runs the plan again, read-only, and should find nothing left to deploy. Its log ends: "Confirm done. If every app above shows &aposup-to-date&apos, the deploy is complete." It then restarts the development build watchers that Ship had to stop.

Each step has a coloured dot showing whether it has run and how it went, and a "Log" button for its output.

Watch: A complex deployment from mdserve

Life hacks

The same habit works away from the desk. Four examples, and how each one works.

Reading on the couch. When I say "put it up on the tab", CC adds a document to a reading list, which a second, deliberately tiny web server shows on my tablet. That server can only show the list and the documents on it. mdserve itself only listens on the machine&aposs own internal address, so nothing on the LAN can reach it. The terminal, editor and money pages simply don&apost exist on the tablet&aposs server, so nothing I add to mdserve later can leak onto the network by accident. A web page can go on the list too; it&aposs just a link to the real site.

The big screen. "Put it on the big screen" opens a page on the TV. Behind that phrase is bigscreen, a small command on my desktop: give it a web address and the page appears on the TV. CC runs it when I ask, and I can type it myself too. The TV is really just an HDMI screen for a Linux laptop. That beats a smart TV: the laptop is a normal computer, so it can run a web server and a fully controllable Chrome. A page on the TV is no use if you can&apost scroll it from the couch, so the TV runs its own little web server, tvremote, and my phone is the remote control. In reading mode it scrolls the page up and down, makes the text bigger or smaller, reloads it, and plays the page&aposs YouTube videos; the same remote also does Netflix. tvremote shows pages in a separate Chrome with its own profile, because Chrome won&apost let a program take control of your everyday one. It&aposs on GitHub if you have a spare laptop and a TV.

What makes tvremote work is the trick mdserve uses, stretched across two devices. In mdserve, the pages I click on and the browser being driven are one and the same: a button on a page asks mdserve, and mdserve drives that same browser through the Chrome DevTools Protocol. With tvremote there are two browsers. My phone&aposs browser shows the remote, but it has no power over the TV at all, because Chrome only accepts that kind of remote control from programs on its own machine. So each press goes to the web server on the laptop, which sits right next to the TV&aposs Chrome and drives it. A web page can&apost control another browser, but it can ask a web server that can. Any device on the LAN can be the remote, and bigscreen on my desktop is just one more of them.

Email and messages. When I need an email written, CC drafts it straight into Gmail and leaves it open on my screen: a new message, or a reply at the end of the correct thread. Slack messages work the same way. There&aposs no email integration behind this. Gmail and Slack are just tabs in mdserve&aposs browser, already logged in, and CC works inside them through the Chrome DevTools Protocol, the same remote control mdserve uses to open its windows.

Errands. My second favourite (behind the TV remote): I needed an official photo from one of several local photo shops. CC made me a one-page guide with the questions to ask and the spec, and the shops in order, each with a button that opened Google Maps to it. As I rode from shop to shop, the next one was always a tap away. This one couldn&apost be served from home: my web servers there reach only my own machine and my LAN, and out on the bike my phone was on mobile data. So CC published it on the web server I rent on the internet, the one that hosts this blog, as a temporary page under my own domain. That&aposs a single command for CC, and the page needs no login to read.

Financial flows

Money gets the same treatment. My mdserve pages only ever talk to the mdserve backend, which in turn makes the calls to Binance, holds the secret API keys, and signs each request, so a key never appears in a web page, and the browser never talks to the money.

The main flow:

  1. Selling crypto for local currency. Binance&aposs peer-to-peer (P2P) market shows who is offering what for my USDT, paid into my GCash account. My page picks the best offer I can actually fill, not just the cheapest one listed, skips people I&aposve blocked, and keeps track of GCash&aposs monthly limit. The counterparty in a P2P trade often asks for a payment QR code, so mine is on the page, ready to send.
  2. Knowing where the money is. One page shows every wallet, with a button to move funds between my Binance wallets.
  3. Withdrawals. Funds go out to Kinesis, an asset-backed digital currency platform. mdserve guards the send: the destination address is fixed on the server, so the page can only choose an amount; an address is checksum-verified before it&aposs ever used; the button needs two clicks; mdserve never retries; and after a send there&aposs a 30-minute cooling-off period, so nothing can go out twice by accident.

Every trade is saved locally, as an EDN file, so the history is mine and doesn&apost depend on Binance.

Try this tomorrow

You don&apost need mdserve to start. Each of these is something to say to your own coding agent, and each one stands alone.

  • "Don&apost give me long answers here. Write them to a markdown file and open it in my browser." This is where mdserve started, and it needs no server at all.
  • "Always back up a file before you change it." It&aposs one line in your agent&aposs rules file, and you never lose a version again.
  • "Before you build this, run falsifier on it." falsifier is a skill (a file of instructions your agent loads) that checks a spec, a request or code for contradictions, hidden rules and wrong assumptions, and reports a short ranked list of what to fix.
  • "When I have to run commands myself, write them as a runbook file, with the reasons in comments, not as a list in the chat." To step through it, cmdq is the Alt-n tool from the runbooks section: one script and three key bindings.
  • Want more than markdown in the browser? mdserve-starter is the generic core of mdserve, ready to build your own on.
  • Got a spare laptop and a TV? tvremote is the phone remote from The big screen, above. Install it on the laptop, open its page on your phone, and you have Netflix, YouTube and reading on the TV. Reading is for pages your agent sends: its bigscreen command puts a page on the TV, and you scroll it from the couch with your phone.

Conclusion

AI, or rather the software it can build quickly, has finally let me get rid of most of the friction involved in controlling computers. I&aposve never looked at mdserve&aposs code, nor at the EDN file its tabs come from. I know what it does from using it every day.

Now imagine arriving for work at a new job, as a developer or a production support person, and being told to run one thing from the command line. Up comes everything that role needs: the dashboards, the logs, the runbooks, the documents, each in its place. Each role could have its own mdserve. A turnkey introduction to a job, long desired, is finally practical.

The next stage is, of course, doing some of these things in a loop, without a person driving each step. That&aposs already happening in software development: agents make several attempts, and the best are kept. Two products now arriving work like this. Both take an evolutionary approach.

samizdat is an agentic harness for Clojure that splits larger tasks among several agents, and each sub-agent gets a contract plus tests as its spec. Several attempts can run side by side, and the winner is chosen by ranking each attempt on checked evidence. The loop itself is a state machine written as data, and the agent can rewrite it while it runs, behind a validate-and-rollback gate.

simmis is a self-hosted app where people and agents share one memory, including the code repository, and its central idea is cheap forking. Every change an agent makes to shared knowledge, documents or the code lands on its own copy-on-write branch, so agents can make several attempts at the same thing side by side. Whoever holds merge authority, a person or another agent, reviews the differences and merges the ones worth keeping. That&aposs what makes it safer to let agents work on software in a loop. Two things stay off the branches: chat, where people and agents talk and the work gets asked for, and actions with outside effects, such as sending an email, which a branch can&apost undo. simmis is an early public release; the overview and roadmap are at simm.is.

Waterfall plans everything up front, then builds; agile builds in small steps and changes course as it learns. Waterfall&aposs weakness is going back up: once building starts, changing the spec is expensive. The evolutionary approach allows both, and removes that weakness. Each attempt can start from a spec, as in a waterfall, and what the attempts reveal goes back into the spec before the next round, as in agile. samizdat gives each sub-agent a contract and tests as its spec and ranks the attempts on evidence; in simmis, attempts sit on forks until a person or agent with merge authority keeps the good ones. And when the code takes hours rather than months, going back up is cheap.

Glossary

  • Agentic harness: software that runs AI agents in a loop, handing them tasks, checking their work and feeding back results, so they can keep working without a person driving each step.
  • Agile: a way of running software projects in short steps, each one delivering something that works, so the plan can change as you learn. It largely replaced the waterfall.
  • API key: a long secret string that lets a program log in to a service on your behalf, like a password for software rather than people.
  • Babashka: a fast-starting way to run Clojure as everyday scripts, the kind of small programs you&aposd otherwise write in a shell language. babashka.org
  • Checksum: a few extra characters built into a code or address and calculated from the rest, so a single typo makes it fail to check out.
  • Chrome DevTools Protocol: the remote control built into Chrome. Developer tools, and programs like mdserve, use it to open tabs, read pages and click on them.
  • Clojure / ClojureScript: a programming language from the Lisp family, popular for working with data. ClojureScript is the same language compiled to run in the browser.
  • Copy-on-write branch: a cheap copy of some data that only stores what changes. It can be kept, merged back or thrown away without touching the original.
  • Counterparty: the person on the other side of a trade.
  • curl: a small command-line program that sends a single web request, the way a browser would but without the window.
  • E-wallet (GCash): money kept on a phone app rather than in a bank account, spent and transferred by phone. GCash is the big one in the Philippines.
  • EDN: Clojure&aposs format for data files, a cousin of JSON. edn-format
  • HTTP: the language browsers and web servers use to talk to each other.
  • LAN: the network inside your home or office, usually just your Wi-Fi, which links your own devices to each other. Devices out on the internet can&apost reach in.
  • Localhost: a machine talking to itself. A web server listening only on localhost can&apost be reached by anything else, even on the same home network.
  • Markdown: plain text with a few simple marks for formatting, like # for a heading and * for emphasis. Easy to write, and easy to turn into a web page.
  • P2P (peer-to-peer): buying and selling directly between people rather than through a shop or a bank, with the exchange only holding the money in escrow until both sides are done.
  • REPL: a live session inside a running program, where you can type code and see the result at once, reading and changing the program while it runs. Epupp puts a Clojure REPL into a live browser tab, so an agent can see what the page actually drew, not just the HTML it was sent, and even change the page until a reload puts it back. clojure.org
  • Spec: the written specification, a precise description of what software must do, written before the code.
  • Stablecoin (USDT): a cryptocurrency designed to stay worth one US dollar. USDT (Tether) is the most widely used.
  • Terminal: the text window where you type commands to the computer, instead of clicking.
  • tmux: a program that keeps terminal sessions running in the background and lets you split one terminal window into several.
  • Uberjar: a single file that packages a Java or Clojure program with everything it needs, ready to run on a server.
  • Waterfall: the old way of running software projects, in strict stages: write the whole specification first, then design, then build, then test, with little going back. It fell out of favour because mistakes in the spec were found late and were expensive to fix.
  • Webapp: an application you use in a web browser. The part you see and click runs in the browser itself, and it talks to a server that stores the data and does the heavy work.
  • Workspace: a whole separate desktop with its own windows, switched with a keystroke. Windows calls them virtual desktops, and macOS calls them Spaces.

Comments or questions? Email me at (turn on JavaScript to see my address).

Permalink

A useful personal agent

I started building a useful personal agent. What makes it useful is that it runs on my computer, but this immediately raises concerns about data exfiltration. I use Apple reminders (among other things) to organise my life so I had it write a little script to be able to interact with those, which works great. One of the main things I want help with is getting a handle on my chaotic piles of todos. They are scattered all over the place and all mixed up. Part of the problem with my current system is that aspirational things are mixed up with hard deadlines, so on busy or unpredictable weeks things that aren’t essential just pile up and stay there and then the “today” list just keeps rolling everything over until there are dozens of things on it, which I couldn’t possibly get done in a single day if I tried. And then I’m just mentally keeping tabs on what is actually important or on a real deadline, which defeats the purpose of the system.

The problem is that since I had a baby every day is unpredictable. I used to have a pretty good handle on my life, but none of my old systems really work anymore. There’s no way to predict how my nights will go anymore so my energy levels are very inconsistent, and I can’t even know what the days will be like. Anyway sounds like I need a follow-up post on the woes of working parenthood. Point being, I need a new system and so far am having fun building an LLM-based one.

Like I said what makes the agent useful is that it runs on my computer. It has access to my data and the internet, and when it was just helping me with reminders it only had access to those, which I write, so I trust them.

The problem is that in the process of having it help me organize the reminders, I realized that the answers to most of the questions it was asking me could be found in my emails. It was still helpful and less overwhelming having a bot help me organize things, but it still needed a lot of information and context from me that it could have found in my emails if it’d had access. But the problem with giving a bot access to email is that emails come from other people. A way to inject a prompt to my bot is the missing piece of the lethal trifecta, so I need to find a way to do it safely. Turns out it’s just a hard problem, but there is some interesting research out there and I’m working on coming up with something that will work to safely allow the bot get answers from my email without a way to pass malicious or sensitive information it finds there forward to an agent that has ways to access the internet.

Permalink

Stringulation

I’ve been stringing you along for years. Time for a (breaking?) change?

Shall I compare thee to a … string?

When porting Clojure for the JVM to the CLR, the question often arises of when a particular aspect of computation is intrinsic to Clojure or just an exposure of a feature of the JVM. For the former, I try to duplicate behavior; for the latter, I try to expose the corresponding CLR behavior. I have run into this not infrequently, particularly in the early days of the porting effort.

One area where it came up very early was with string comparison. And, frankly, I did not give it a lot of thought. Where ClojureJVM used java.lang.String.compareTo, I used System.String.CompareTo. In other words, I made the decision to expose the underlying platform mechanism. In retrospect, this was probably not the right decision. These methods are significantly different.

The JVM String.compareTo is an ordinal comparison of UTF-16 strings. This is a straightforward lexicographic comparison, performed character-by-character. The CLR String.CompareTo is culture-sensitive, i.e., the result of comparing two strings varies depending on the culture in effect at that time, which is thread-dependent.

There are several consequences of using CompareTo on the CLR, of varying import.

ClojureJVM and ClojureCLR differ on things such as comparisons.

(compare "a\u00ADb" "ab") ;; => non-zero meaning not equal (JVM)
(compare "a\u00ADb" "ab") ;; => zero, meaning equal        (CLR) 

(\u00AD is the soft-hyphen character.)

And, thus, sort order:

(sort ["b" "B" "a" "A"]) ;; => ("A" "B" "a" "b")  (JVM)
(sort ["b" "B" "a" "A"]) ;; => ("a" "A" "b" "B")  (CLR)

compare and = don’t agree.

(compare "a\u00ADb" "ab") ;; => 0  (equal)     (CLR)
(= "a\u00ADb" "ab")       ;; false (not equal) (CLR)

The difference here is that compare uses String.CompareTo (culture-sensitive) and = uses String.Equals (ordinal comparison).

sorted-set and hash-set yield different sets on the same inputs.

 (count (sorted-set "a\u00ADb" "ab")) ;; => 1, one string silently dropped because it compares as equal (CLR)
 (count (hash-set "a\u00ADb" "ab"))   ;; => 2, because hash sets use ordinal = (CLR)

Culture-sensitivity bites in some related places where string comparisons occur.

For example,

(compare :b :B) ;; => 32 (positive, indicating :b > :B) (JVM)
(compare :b :B) ;; => -1 (negative, indicating :b < :B) (CLR)

Culture-sensitive string compares are thread-dependent.

For example, ASP.NET Core sets the culture per request from Accept-Language, so (sort names) silently follows each user’s collation. Picking up information from the thread is fine; I feel it is better for it to be a deliberate choice rather than delivered behind your back.

Culture-sensitive comparisons are more expensive than ordinal comparisons.

Sorting a bunch of strings or creating a sorted map under ordinal comparison executes roughly 54-61% fewer instructions (instruction counts measured on one machine, not timings) than a culture-sensitive comparison using en-US. Capturing one culture comparer at startup to avoid thread lookup of the culture decreases the instruction count by roughly 5% compared to String.CompareTo today.

Culture-sensitive comparisons have not been consistent over time.

“Before .NET 5, the .NET globalization APIs used different underlying libraries on different platforms. … If you upgrade your app to target .NET 5 or later, you might see changes in your app even if you don’t realize you’re using globalization facilities.” (See Globalization and ICU - .NET). And you can switch globalization providers. ClojureCLR on .NET Framework uses NLS, not ICU, so Framework and .NET builds of ClojureCLR on the same machine can yield different results. That’s just on Windows. Let us not discuss Linux and Mac. Read it and weep.

Going deeper

The original focus in the benchmarking investigation surfaced the compare issue outlined above. After the initial results, I decided to expand the search to all string manipulation in the ClojureCLR, with comparisons against ClojureJVM where appropriate. Most of this internal string manipulation is related to reading data (which in Lisp-land includes programs) which arguably should not be culture/locale sensitive. It is not in ClojureJVM – all is ordinal. There are a few places where culture-sensitivity snuck into the ClojureCLR code, by carelessness or ignorance. Some are so marginal that I’m guessing they have never been encountered. For example, if we let <SHY> represent the soft-hyphen character, when reading source code:

  • JVM reads foo:<SHY> => creates symbol foo:<SHY>
  • CLR reads foo:<SHY> => throws an exception under en-US (at least)

The most significant culture-dependent bugs are

  • tr-TR: the flag to turn on direct linking in the compiler is ignored – the dotless ‘i’ (U+0131) makes an appearance.
  • sv-SE: (+ 1 1E-10M) doesn’t compile. Swedish uses a different minus sign character.

I consider these examples to be bugs. Fixing them is not a breaking change.

There are five functions (starts-with?, ends-with?, index-of, last-index-of, replace-first) in the clojure.string library that are in conflict with the JVM version and also demonstrably incorrect. These bugs are visible to users today, but they are bugs and should be fixed. Example: (replace-first "a<SHY>b" "ab" "X") gives "Xb" and corrupts the string. Results would change only for strings containing ignorable characters (soft hyphen, NUL, combining marks) and the changes would match the JVM’s answers.

A proposal

I plan to make two sets of changes. The first set is to fix the bugs above: the reader, the compiler, number literals, and clojure.string. I am not <SHY> about making these changes.

The second set of changes will be more user-visible and thus have the potential to break user code. This involves changing the definition of compare to be ordinal-based. To minimize impact, we can make this change selectable at startup. Set the switch to ‘culture’ and you get the current behavior; set it to ‘ordinal’ and you get ordinal-based comparisons. The ‘ordinal’ mode is pretty much identical to ClojureJVM behavior.

There is a third possibility: A captured culture mode which at startup creates a string comparator based on the CurrentCulture in effect at system startup. That is used by the compare function. If you change CurrentCulture, it will not be seen by this; there is no thread-dependency. You don’t get the large speedup, but do get the 5% savings from the no-thread-lookup. I’m not sure it is really worth it. The audience would seem to be a culture-mode user who sorts heavily and doesn’t care about threads. I’m not sure that’s much of a market. (This would require a little more testing before full validation as an option. )

For ordinal mode users, varying culture in things such as sorting is still possible. Most of the Clojure-defined functions, such as sort and sorted-set, have variations that take a comparator function. In fact, I think the general advice should be to be explicit in your sorting and comparator options always. To make this easier, we will add a culture-comparator function that will return a comparator function based either on the value of CurrentCulture when called or a supplied culture. You could write (sorted-set-by (culture-comparator "sv-SE")).

The choices

The bug fixes will happen regardless. For the compare changes, we have some choices.

  1. Do nothing. Not really an option, from my viewpoint.
  2. Put in the switch. Default is ‘culture’. If you want performance, you have to ask for it.
  3. Put in the switch. Default is ‘ordinal’. Performance and consistency are the default.

To be clear, my own preference is #3. The current situation has internal inconsistencies, is inconsistent with the JVM, and is slower overall.

An additional choice is

  1. Do we need the ‘captured culture’ mode?

I plan to put a link to this document up on the #clr channel in the Clojurians Slack for feedback. I’ll amend this post to reflect that discussion.

The result

TBD.

AI disclaimer/acknowledgement

I’ve been using AI coding tools extensively in my benchmarking work. There is a lot of tedious coding involved and the tools do it for me a lot faster and more correctly than I would on my own. This document reports on the results of that work. I drafted the document, then used the AI coding tool to vet it for accuracy, resulting in some typo corrections and a few suggested rewordings of inaccurate phrases. A stray comment in its review of my first draft sent me down a new line of inquiry that ended up in the two-phase approach here. The instructions to the agent include not drafting any code that might go into the ClojureCLR code base. It can identify locations and make prose suggestions only. (And it can write all the testing code its non-existent heart desires. It makes measurement runs on my commits as we go along.)

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.