Migrating from a Monolithic Orchestrator to Apache Airflow

Photo by Corinne Kutz on Unsplash

Before we knew better

Our orchestration system started as a simple internal solution to manage event pipelines and trigger downstream jobs. Over time, as more workflows and dependencies were added, it gradually evolved into a tightly coupled monolithic scheduler that became increasingly difficult to understand and maintain.

Understanding how a workflow executed often meant looking through multiple files, configurations and database tables.

For newer team members, onboarding into the system took time because much of the workflow context was distributed across different parts of the codebase. Even relatively small changes required careful coordination to ensure existing pipelines continued to work as expected. Similarly, debugging typically involved manually tracing logs and rerunning jobs to better understand execution behavior.

Limitations of our legacy design

We had a monolithic architecture written in Clojure that bundled all our event pipelines together, added dependencies between them and triggered a Lambda function.

Legacy Workflow

This Lambda function added a single monolithic step to the EMR cluster. If there was an issue in any one of the pipelines, the entire flow would fail due to the single step on the cluster.

We did not have step-wise monitoring in the old design, so during on-call situations it became very difficult to identify which part of the pipeline was causing the issue.

Photo by Tim Gouw on Unsplash

There was no single place to answer basic questions like:

• What runs first?
• What happens if this step fails?
• How do I re-run just one part safely?

The scheduler worked, but it was hard to understand, hard to maintain and even harder to explain. That’s when we realized we needed a better way.

What we actually needed

Our aim was less about fancy scheduling features and more about making our daily work easier and more reliable.

1. Simpler onboarding, less mental overhead

Our existing step scheduler was built in Clojure and came with a steep learning curve. New engineers had to spend a lot of time just understanding how jobs were defined, chained and monitored before they could confidently make changes. In contrast, moving to Apache Airflow and its Python-based DAGs made workflows far more readable and intuitive. The structure of dependencies, retries and scheduling became obvious at a glance and because Python was already familiar to a much broader set of engineers, onboarding became significantly faster. The shift not only improved maintainability but also made the scheduler more accessible to the wider engineering team.

2. One failure shouldn’t kill everything

In our old setup, if one step in a multi-step job failed, the entire job would terminate. This was painful, especially for long-running pipelines. A single transient issue like a network glitch or a temporary dependency failure meant rerunning everything from scratch. We needed better fault isolation, where a failed step doesn’t automatically bring down the whole workflow.

3. Clear visibility and easy recovery from failures

When something failed, it wasn’t always obvious what failed or why. We wanted a clear view of each step or task, with the ability to:

• Quickly see which step failed
• Get alerted when failures happen
• Re-trigger only the failed step instead of restarting the entire job

In short, we were looking for a workflow system that was easier to understand, more forgiving when things go wrong and faster to recover from failures without adding more operational complexity.

Why we chose Apache Airflow

Once we were clear with our needs, Apache Airflow was a good fit. Airflow is designed to act as a robust orchestration platform.

1. Airflow Was Already Available and Production-Ready

One of the biggest factors was that Apache Airflow was already set up and running in our environment. This allowed us to avoid the upfront cost of evaluating, deploying and writing a new scheduler from scratch. Instead, we could focus on solving workflow problems rather than infrastructure problems.

2. Better Observability and Debugging

Airflow provides rich visibility into workflows:

• Task-level logs
• Retry history
• Execution timelines
• Clear success/failure states
• Better dependency management between tasks

This significantly improved our ability to debug failures compared to the limited visibility offered by the step scheduler.

3. Support for Retries, Backfills and Scheduling

Apache Airflow provides core orchestration capabilities such as configurable retries, historical backfills, and flexible time- and event-based scheduling out of the box. These features are built into the platform and require minimal configuration. As a result, we were able to eliminate a large amount of custom logic from our legacy scheduler. This simplified our workflows and reduced long-term maintenance overhead.

4. Separation of Orchestration and Business Logic

With Airflow, orchestration logic lives in DAGs while business logic lives in independent services or jobs. This separation made workflows easier to test, reason about and evolve over time — something that was harder to enforce with a step scheduler.

Migration approach

The first step in the migration was mapping the existing scheduler concepts into Airflow primitives.

• Each pipeline was converted into an Airflow DAG.
• Existing steps became Airflow tasks/operators.
• Dependencies were explicitly defined using DAG relationships instead of implicit execution logic.

This made the execution flow much easier to understand and visualize.

In the new setup, Airflow is responsible for workflow coordination, scheduling DAGs, managing task dependencies, handling retries, monitoring execution and sending notifications. The compute-heavy processing workloads such as Spark, Hive and Presto jobs run independently on transient EMR clusters.

The execution flow starts when a scheduled trigger initiates a DAG run in Airflow. Airflow then creates an EMR cluster and submits the required processing steps using EMR operators. After submitting the processing steps to EMR, Airflow monitors their execution using EMR Sensors such as EmrStepSensor.

During this process the sensor continues polling until the step reaches a terminal state such as COMPLETED, FAILED or CANCELLED. If the step completes successfully, Airflow marks the sensor task as successful and proceeds with downstream tasks. If the step fails, Airflow immediately updates the task status and triggers the configured retry policies, failure callbacks or alerting mechanisms.

This allows Airflow to track the progress of jobs while EMR handles the actual processing and compute resources. The Airflow UI reflects the current state of each task, making it easy to track long-running jobs and troubleshoot failures.

In addition to status monitoring, Airflow captures execution metadata and integrates with EMR logs stored in locations such as S3 and CloudWatch. This allows us to quickly investigate failed jobs by navigating from the Airflow task instance to the corresponding EMR step logs.

Once all submitted steps have completed successfully, Airflow initiates cluster termination using EmrTerminateJobFlowOperator. Since we use transient clusters, this ensures that compute resources exist only for the duration of the workflow, helping optimize infrastructure costs and avoid idle cluster usage.

Finally, Airflow updates the DAG status, records execution metrics and sends notifications through configured channels such as Slack and email. This end-to-end orchestration model provides a clear separation between workflow management and data processing while improving observability, reliability and operational efficiency.

Separating business logic

Previously, orchestration and execution logic were tightly coupled in the same system. During migration:

• Spark jobs, Hive queries and scripts were extracted into reusable components.
• Airflow was used purely for orchestration.
• DAGs became lightweight and focused only on workflow coordination.

This improved maintainability and simplified testing.

Incremental rollout strategy

Instead of migrating everything at once:

• Low-risk workflows were migrated first
• Legacy scheduler and Airflow ran in parallel initially
• Compared workflow executions across both systems over a defined observation window, validating not only data correctness but also runtime behavior and infrastructure efficiency. Key metrics such as total execution time, EMR instance-hours consumed, job completion rates and failure patterns were analyzed across multiple pipeline runs.
• Critical pipelines were migrated gradually

This minimized operational risk during the transition.

Logging and observability improvements

The previous system required manually tracing logs across multiple components.

With Airflow:

• Each task had centralized logs
• DAG execution history became visible through the UI
• Failures could be traced directly from task graphs

This reduced troubleshooting effort for developers.

Success metrics (post migration)

Conclusion

Migrating from a monolithic orchestration system to Apache Airflow was more than just a technology change, it was an architectural shift toward building a more scalable, maintainable and observable workflow platform.

By decoupling orchestration from business logic, standardizing workflow management and leveraging Airflow’s built-in capabilities such as scheduling, retries and monitoring, we significantly reduced operational complexity and improved developer experience.

While there are still areas we plan to improve, moving to Airflow gave us a solid foundation to scale our data and event-driven workflows more reliably in the future.

Migrating from a Monolithic Orchestrator to Apache Airflow was originally published in helpshift-engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Permalink

Putting Code Under a Microscope: Wavelet-Based Context for LLMs

Every developer who has tried an AI coding tool is familiar with the problem of watching the model fumble with the codebase to find relevant sections to edit. Since it's not possible to load an entire codebase into the context for large projects, it greps through a few files to give it some context, and guesses what to do next. But code has a hierarchical structure with layers and boundaries. Functions sit inside classes. Classes live in files. Files make up modules. One 400-line file can contain six different conceptual areas, each with its own distinct purpose.

When a human developer reads the code, we leverage the structure when we try to understand it. We examine the way files and classes are organized, and try to find relevant logic based on that. Wouldn't it be nice if the agent could, like you, zoom in and out of the code so that it could look at the big picture, then jump directly to the exact function it needs without having to open the entire file.

That’s what WaveScope does. It’s an MCP server that uses wavelet transforms to give LLMs a multi-resolution view of the codebase. Imagine something like progressive image loading, but for source files. Even though newer models can handle large contexts on paper, what happens in practice is that they start losing focus. The more context the agent has the harder it is to figure out what it actually needs to work on, and what to prioritize leading to context rot.

Currently, there are two main ways of dealing with the problem. Grep-based search finds exact matches but ignores structure since pattern matching on individual lines can’t tell you where a class boundary is or where an error handling region starts. Embedding-based RAG is another approach which understands semantic meaning but loses position and structure. Neither gives the model a real sense of the architecture of the code.

Code Has a Rhythm

Open a file in any language like Clojure, TypeScript, Rust, or Go and you’ll see repeating common structures throughout the code. Imports sit at the top. Class and function definitions pop up at regular intervals. Indentation also has its own distinct pattern in every language. Comment blocks and blank lines are pauses in between. What if there was some way to extract these patterns, and create a structure similar to an AST without actually having to know the syntax of the language.

Luckily for us, wavelets were made for processing exactly this sort of signal by decomposing it into multiple scales at once. The transform gives you all the fine details along with the large-scale patterns at the same time. This family of algorithms is very versatile and has been used in many different fields. Seismologists use them to spot earthquakes, doctors sharpen MRI scans with them, and audio engineers separate basslines from vocals. Code structure, it turns out, is just another kind of signal that can be decomposed in a similar fashion.

So how does that actually work? Once each line has a score, the file can be treated as a 1D signal representing a sequence of numbers that rises and falls with the density of the structure. The Ricker wavelet is a little template shaped like a bump with a dip on each side. You slide it across the signal one position at a time and, at every position, measure how well the signal underneath matches that shape. A strong match means there's an elevated region sitting between two quieter regions which suggests a structural boundary. The output is a coefficient at every line scored on how much it looks like a boundary.

The trick for encoding different resolutions lies in the width of the template. You can slide the same shape stretched to many widths, each representing a different scale. A narrow wavelet only matches when the elevated region is a line or two wide, so it fires on small, sharp features. A wide one ignores line-level noise and responds to larger regions elevated relative to their surroundings, firing for big structures. So, the next trick is to run multiple widths at once to see boundaries which light up across a band of widths rather than just one. Features consistent across different scales tend to be genuine structural edges that we care about.

Since code structure nests at different sizes, each size shows up at the width that fits it. We can see a concrete example of how this works by running WaveScope on its own src/context.ts. A single line such as a lone import statement, or export class FileContext { declaration stands out over its quieter neighbors making it a sharp one-line spike that the fine, narrow wavelets lock onto. Its coefficient peaks at scale 1 or 2 and fades away at wider scales. At the other extreme is the long keyword-dense cascade inside the inferLabel method, which has a wide run of consecutive if (tokens.includes("class")), interface, enum, and struct branches. That whole region reads as one broad elevated plateau, and its coefficient climbs steadily as the wavelet widens to roughly 0.5 at scale 16, 1.0 at scale 32, 1.3 at scale 64, and about 2.3 at scale 128, where it peaks as the strongest structural response in the entire file. The biggest structure produces the strongest coarse-scale signal, which is exactly the spot you want surfaced first when you zoom out.

The line scores that feed all of this come from the same pass. In our example, export class FileContext { scores 1.6 because the class keyword weight of 1.0 and export at 0.6, the one-line get lineCount() accessor scores about 0.58, the readonly field declarations beneath it score roughly 0.08 each, while comments and blank lines around the class score a flat 0.0. The class declaration towers over its own body, the body towers over the whitespace around it, and the wavelet reads those relative heights at every width.

The intuition above is the front half of the pipeline where you score every line into a 1D signal, then slide the Ricker wavelet across it at eight scales which are 1, 2, 4, 8, 16, 32, 64, and 128 lines giving us a coefficient at every line and width. Next, we need a couple more steps to turn raw coefficient arrays into something the model can use.

First is to do multi-scale peak detection where the arrays are scanned for local maxima and ranked by magnitude. Strongest boundaries represent features such as the beginning of a class or the transition between imports and code. Because a real boundary shows up at several adjacent scales, as we saw above, these repeats can be safely collapsed into a single peak to avoid flooding the ranking with duplicates of the same spot.

The second step is the band assembly, where the peaks are separated into three broad zoom bands. The fine band at scales 1–2 shows raw source lines in a close window around the query center. The medium band at scales 4–16 tracks function and class signatures with some context around them. Finally, the coarse band at scales 32–128 compresses the whole radius into a section-level summary.

All of that processing is handled by the MCP server, and the model simply sees structured JSON with bands and peak positions without having to worry about any of the wavelet math.

What the Model Actually Gets

Let's say an agent calls query_wavelet_context which is centered on line 150 inside a 500 line TypeScript file that has some authentication logic. In this case, the fine band will have the actual lines of code being inspected. The medium band will provide a summary of lines 0–400, guided by peaks such as the imports at top and test helpers at bottom.

The model derives its knowledge of what updateUser does by paying attention to the fine band, but it also knows about authentication context from the coarse band. It's able to jump to related code by recognising class and function boundaries in the wavelet peaks without needing to see all 500 lines of text from the file.

There is also a utility called get_important_positions, which operates on the whole project. It goes through every source file, smooths out the wavelet peaks, and gives you a ranked list of the most important places in the code.

Beyond locating structural boundaries, the server can also measure how complex or irregular those structures are, using a pair of entropy analysis tools driven by a Haar discrete wavelet transform and bit-cost estimation I discussed in my last post. Ricker coefficients can be quantized at each scale using get_entropy_bands which computes their bit-plane counts with higher cost indicating more structural irregularity at that resolution. The original per-line structural signal can also be decomposed through multiple Haar levels using get_complexity_heatmap to project the entropy cost back onto a per-line irregularity score. The model can use these scores as a sort of texture channel to understand where gnarly parts of the code live. Any boilerplate regions will end up with a low score, and so they can be safely summarized or skipped, while high-entropy regions will likely contain dense logic or unusual patterns that warrant extra attention. These tools give the model a data-driven way to triage code at high level and works great for refactoring tasks where the model can easily find sections with tangled logic and break it up.

The key advantage of wavelets is that they focus on the overall structure of the code which is something that can only be inferred using tools like regex. It's worth mentioning that it's possible to get a lot of the same type of analysis using AST parsing which is even more exact. However, AST tools necessitate having a parser for each language while wavelets are completely agnostic of the semantics of the text they're applied to. They simply flag statistical regularities, and that's what makes them such a broadly applicable tool in the first place.

Wavescope’s approach strikes a happy medium between grepping and full blown AST analysis since the transform works on any 1D signal. Language awareness comes from a simple keyword-weighting layer rather than a full parser with each language just needing a 10-line config to describe its core semantics. And the whole thing is cheap to run, able to process an entire file in milliseconds to create hierarchical and multi-scale outputs. The scales happen to be a natural representation of code structure which provides the LLM with a map to navigate it.

Since the wavelet locates edge positions at all levels simultaneously, it trivially locates structural changes that even sophisticated parsers might struggle to detect. For example, a long sequence of if/else blocks looks structurally different from a class with many short methods while regions of documentation appear as valleys between peaks. The wavelet doesn't need to know what these things actually are to perceive that they are structurally different. And this often happens to be just what the model needs to figure out what to do.

By the Numbers

To illustrate the concept, I ran three realistic development tasks against WaveScope's own 14 file codebase containing just under five thousand lines of TypeScript to compare the token cost with the traditional way. "Traditional" here means what LLM coding agents actually do: grep for landmarks, read targeted chunks of code, and skim file headers to get a sample of what the structure looks like.

One common task is to understand the structure of a large file. So, I had the agent analyze index.ts weighing in at 854 lines to see where imports end, to find where the tool registrations cluster, and identify the startup code. These are your typical tasks where the approach would be to grep for export landmarks and section comments, then read the import block, find a registration example, and recognize the startup tail. That costs about 2,000 tokens producing a patchy picture which is just a heuristic. WaveScope's coarse band plus the top 15 important positions give the same structural overview using just 750 tokens with section-level boundaries that grep can't surface because it's only matching lines without the structural context around them.

Now, let's take a look at a different kind of task, which I've alluded to earlier, where we want to find tangled code that needs refactoring. A naive way would be to run wc -l to find the biggest files, then skim large chunks looking for deep nesting and high cyclomatic complexity. Such a task runs at about 5,200 tokens to read 600 lines across two files, and you would still miss tangled code in any files you didn't scan. On the other hand, WaveScope's get_complexity_heatmap flags exact per-line irregularity scores across every file you point it at. Running it on the three largest files, which are index.ts, wce.ts, and context.ts costs a mere 436 tokens. That's a whopping 92% reduction, but also surfaces precise hotspots at line 287 in handleDiffWaveletContext scoring 0.92, line 59 in FileContext with a score of 0.92, and so on. The analysis finds every section of interest across the whole codebase producing a ranked list of spots to look at closely.

Another example would be to identify which files are architecturally core versus peripheral. The traditional approach runs head -25 on every file to read imports, which ends up costing around 2,900 tokens for all 14 files. As a result, you learn what each file imports without actually knowing which ones define the project's architecture, and the model will have to spend more tokens digging deeper based on its best guesses. WaveScope's project-wide get_important_positions returns a structural-density ranking of every file in just 1,700 tokens along with meaningful rankings. Now it's clear that signal.ts tops the list with the highest keyword density per line. The heavyweight algorithmic files are wce.ts, context.ts, and index.ts which have lower average density because their structural features are spread across hundreds of lines of implementation. Other files, such as the haar.ts and file-cache.ts utilities, end up with a low ranking. Now it's clear which files need to be read first in order to understand the conceptual skeleton of the project.

Across all three tasks, WaveScope used significantly fewer tokens while producing meaningful answers that are structural rather than textual which is precisely what we were interested in. The structure provides the model with understanding of relationships within the code base that is simply not possible to do by doing regex based heuristics. On top of that, a 128K token window would burn 8% of its capacity using the traditional approach for these three tasks while WaveScope needs only 2%. And the gap scales proportionally with the size of the codebase, making the tool particularly effective for analyzing large projects.

It's also worth noting that processing all 14 files using signal extraction, Ricker CWT at 8 scales, peak detection, and band assembly all averages at just 3 milliseconds per file. And the entropy heatmap adds about 1ms per file on top. In the context of reasoning time for the model, this is effectively free to do.

Hopefully, the examples above make the value of using structural peak analysis clear already, but we can leverage the information that's already been produced even further by passing it through an entropy encoder to see just how complex those regions actually are. Running get_entropy_bands against index.ts gives a breakdown of the bit-cost estimates per wavelet scale, which is an indirect measure of structural irregularity. Here, it can be seen that the finest detail at scale costs 613 bits while the coarsest costs 322, suggesting that the file is structurally busy at the line level. A high density of handler functions and schema definitions found in a tool registration class is what's responsible for the pattern.

Next, we can call get_complexity_heatmap to take things further by back-projecting entropy onto individual lines via a Haar DWT. Doing that identifies the top irregularity hotspots in index.ts such as handleDiffWaveletContext with a score of 0.92 and handleGetCursorImportantPositions with a score of 0.90. These are non-trivial async handler functions with the most branching logic in them, and the heatmap is able to flag them without knowing anything about TypeScript! The entire heatmap costs 474 tokens as well, giving the model a data-driven triage list to focus on and to safely skip the boilerplate.

WaveScope is open source and can be found at https://github.com/yogthos/wavescope-mcp. Add it as a drop-in MCP server to give your agent a zoom lens for code.

Permalink

Clojure Deref (Jun 2, 2026)

Permalink

I Deleted the Hand-Written Version of BigConfig

After reading that Bun was rewritten from Zig to Rust, I tried a similar experiment on BigConfig: port the broader stack across TypeScript, Python, and Clojure with an agent, then redeploy the public BigConfig website with the agent-written version.

Permalink

Tracing rays with jank

I&aposve continued my optimization work through May and today I&aposll be reporting the status of the next benchmark: a little Clojure ray tracer. Before jumping into the details, I want to say thank you to my Github sponsors and to Clojurists Together for sponsoring me this whole year. Thank you!

Permalink

cljs-ajax 0.9.0-beta1 is out

I’ve been trying to make a point of not talking about stuff that I’m doing and concentrating on actually delivering. I find that this avoids a sense of “false progress”, which is why I finally get to talk about something I’ve been working on for the last three months. Or more accurately, Codex on the free plan has been working on it.

The cljs-ajax library was started by the incredibly productive Dmitri Sotnikov, and transferred to me because he had many larger and more important projects. It was the most popular cljs http library by a small margin. The maintainer of the main competitor, cljs-http, (which is an excellent library that explores a different design space), r0man, sadly hit burn out on his project. And approximately six months later, I hit the same thing. Let’s be clear: this is not really a sign that I’m no longer burnt out on this project. Without Codex there’s no way this would have happened.

Burnout

One of the things you learn when maintaining an open source library of even the modest size of cljs-ajax is that engineering concerns predominate. I think everyone starts because they want to write some code, but you look at any project’s mailing list and it’s all about the build server.* cljs-ajax really suffered in this regard: even with all of the unit testing coverage, HTTP fundamentally requires integration testing. Not only that, it needs to work across a wide-variety of platforms, and against a shocking number of badly-behaved endpoints. And I had… no automated integration testing. Instead, I had a test server that I ran repl commands against. Which I needed to do for every last PR.

Equally, the deployment process was a pain in the neck. And relied on a bunch of secrets that I kept losing. I’m sure the Leiningen deploy process was well thought out, it was just more work than I particularly wished to do.

Needless to say, both of these things contributed to my burnout.

So what happened during burnout? Well, nothing very much. A bunch of people submitted PRs and I couldn’t work up the energy to test them. I had one approach to take over the project that I believe was a supply-chain attack. (A pretty low effort one compared to the xz attack.) Nothing really came to take the place. The Closure Library and Compiler was abandoned by Google and adopted by the ClojureScript team. And finally, a lot of the people who had contributed… stopped using Clojure and ClojureScript entirely. There’s been no actively-maintained general http libraries for ClojureScript for five years. On the other hand… fetch happened. Which has rendered a lot of the “cross-API” code rather redundant.

What didn’t happen is slightly remarkable: the code didn’t break. People are still using it and it still works. The innate conservatism of the Clojure project has finally rubbed off on the ClojureScript project and backwards compatiblity is still strong. This isn’t to say that old library references can’t be a problem, but it’s much better than you might expect from many platforms after a five year hiatus.

* Look at a really big project and it’s all about governance.

What now?

So, as the title says, I’ve published a beta version of Clojure 0.9.0. The differences between 0.8.4 and this are… zero. Instead I have:

• Replaced Leiningen with cljs-build. I have no opinion about which is better, I’m just using the default option, as I did when Leiningen was employed in the first place. This also replaced doo and shadow with karma.
• Developed a proper set of integration tests replacing my old manual process.
• Developed a new CI process using github workflows and a release process to match, which also means the keys are now on my github account.
• Fixed dependabot issues.
• Lightly updated the documentation (including the first merged PR from rgkirch).

In other words, I’ve prioritised just getting the project into a fit shape and addressing some of the issues that caused so much burnout in the first place.

What’s next?

So, honestly, I have no idea what’s next. I plan to merge down the rest of the existing PRs. Then, I plan to triage the open issue list. There’s an obvious argument to be made for making fetch the default implementation, and maybe changing the API to be more fetch-like. Adding in clojure.spec would obviously be desirable, although I have no idea how workable that would be given some of the heroic things cljs-ajax does to support its very wide set of deployment scenarios. Building a proper objective framework for measuring the deploy-size implications of various options would be desirable since one of the unspoken principles of cljs-ajax has always been to avoid paying for stuff you don’t use. (Within reason, this isn’t C++.) Again, this is mostly engineering stuff. There’s no particular ambition for supporting anything other than the existing RPC model. Equally, I have no particular interest in achieving feature parity with clj-http or hato, although to the best of my knowledge cljs-ajax is still a pretty good, if unpopular, choice if you want an asynchronous HTTP API in Clojure.

But more importantly: I don’t know if there’s actually any interest in this project. There’s other alternatives. The research I’ve done suggests that, even after five years, cljs-ajax is still pretty popular in the community but I’m very outside the community these days, have no active Clojure/ClojureScript projects and I worry cljs-ajax may be a relic of a bygone age.

What if I don’t want to use LLM code?

There are many good arguments for not wanting to rely on LLM code, and especially code that isn’t fully understood by a human. These arguments are valid. I admire and understand those taking a stand against AI slop. For the record, this blog post is entirely human-produced. However, the situation with cljs-ajax is, quite simply, absolutely nothing is likely to happen at all without it. There’s no prospect of a new maintainer stepping in the way that I did. The community is too small and, given my previous experiences, I’d need to put a fair bit of effort into vetting anyone unknown. I still have a day job that occupies most of my time, and I use AI heavily there. The old code is still there. The latest release still has no AI-written code in production pathways, but that’s likely to change.

Permalink

Fast HTML-to-Markdown extraction from any URL, for LLMs (r11y)

r11y is a fast, native CLI and Clojure library that extracts the main content of any web page as clean Markdown with metadata. Built for feeding LLMs, faster than trafilatura and Defuddle.

Permalink

Teaching LLMs to one-shot complex backends at scale, report #1

Our workflow

Current status

Skill files

• Rather than refer to the var bound by (defmodule FooModule ...) directly as FooModule , it would try to construct it as (FooModule.)
• Unbounded locations in a PState would not be subindexed
• Object would be used for schemas instead of something precise
• Partitioners would be missing, especially before writes, causing the module to write to the wrong locations

1. Implicit spec. Derive every edge case and invariant the protocol implies but doesn’t state. Produces an IMPLICIT_SPEC.md document.
2. Plan. Design the depots, PState schemas, and topologies before writing any code. Produces PLAN.md .
3. Plan validation. Review the plan against a checklist of scenarios (e.g. race conditions, failures/retries). Produces PLAN_VALIDATION.md with a pass or fail verdict.
4. Implement. Write the module source, adhering strictly to the plan.
5. Implementation validation. Review the implementation against a checklist of common mistakes. Produces IMPLEMENTATION_VALIDATION.md .
6. Tests. Write thorough tests covering every protocol method and every edge case from the implicit spec.
7. Test validation. Review the test code against a checklist of common mistakes. Produces TEST_VALIDATION.md .
8. Run tests. Run the test suite.

Orchestration

Current challenge

• Minimizing data in the cache and using memory and GC-efficient data structures. The correct solution uses a ring buffer of long values for just the user ID and post ID for each entry in the timeline. The agent so far is always using a TreeMap for the timeline and also storing the post content in the cache, which massively increases memory usage unnecessarily since that can just be fetched in the query topology that fetches a timeline page.
• Fairness is not handled at all. It eagerly delivers posts to all followers immediately instead of spreading out delivery for large users over a longer period of time, violating the spec.

Model choice

Open questions

LLM usage during development

Conclusion

Permalink

Started Learning Rust

Reading The Book

I have started learning Rust from The Book, currently I am reading The Slice Type. For some reason ownership did not look confusing to me, but references & borrowing did.

I learn a programming language by doing a lot of exercises. I will soon put a repo in Codeberg, where I will upload my Rust learnings. The Rust book seems to be a book written for people who already know other programming languages. I don’t think you should write a programming language book like that. It should be a book where anybody can pick it up — in plain natural language — and be able to understand it. If the Rust community doesn’t do it, I think a lot of people are going to avoid Rust. It’s the same with Clojure. Clojure has been dominated by people who are really experienced in the industry, and they don’t even bother about people who are starting with Clojure as their first programming language.

In fact, it’s the people who are already familiar with other programming languages that are the ones who are going to start with Clojure. I have never seen a total noob to programming pick up Clojure. Somehow the Clojure community seems to have turned a blind eye to this huge problem. They are happy the way they are.

Even though you have AI, you do need muscle memory to write good code. Or at least in today’s case, to check for improvements that could be made by A.I., and to find corrections in it without much effort. Mastery is very important.

Should I vlog or blog about Rust? I don’t know. I don’t have the bandwidth. I already have a channel called Clojure Diary where I vlog about Clojure. That was because I couldn’t find sufficient material that would help a beginner learn Clojure. I also write a book on Clojure, for almost the same reason. There aren’t many libre books on Clojure. Maybe the community is very small, so there aren’t many who will talk and write about it. This could be the result of the community never trying to promote Clojure as a beginner-friendly first programming language that one can learn.

Don’t Have Expectations

I want to tell the reader of this blog: please don’t have expectations. This is a highly ignorant guy, who is passionate, maybe foolhardy, and for some reason thinks he can host Clojure on top of Rust. It might never happen. At most, I think I can write a Lisp interpreter in Rust. Right now I have no idea how to create a compiler for Lisp using Rust. Compiling stuff to machine code, I feel is far difficult than interpreting stuff.

I do know that Build Your Own Lisp might be helpful. Right now I want to learn Rust, then use this book to build Lisp with Rust.

I’m determined to put just another foot forward, and I am dedicated enough to tell the world what step I have taken — that’s about it for now.

Permalink

Structural diffing in Emacs; deterministic agent harnesses

Difftron

Last newsletter I wished for structural diffs in a Magit-like UI and, a few weekends later, it turned out as awesome as I’d imagined.

Behold, Difftron:

entity type hierarchy with one function change expanded to show a side-by-side diff with highlighting"/>

See the 15 minute demo video for more details.

Conceptually, the tool works as follows:

• A Rust binary parses code into semantic entities (functions, types, classes, etc., depending on the language)
• Entities are matched between the left and right sides of the diff based on entity type and name
• Matched entities are shown as a side-by-side diff
• Unmatched entities are marked as added/removed

This barely scratches the surface of the complex computer sciencey questions about tree diffing and heuristics for matching entities, but so far I don’t mind at all — as it turns out, just this basic scheme combined with a rich user interface already feels miles better than the standard file-and-line-based diffs I’ve been using. All of the UI credit goes to Magit and Emacs, for a few reasons:

First, Magit espouses the idea that everything can be interacted with:

Commands are invoked, not by typing them out, but by pressing short mnemonic key sequences. Many commands act on the thing the cursor is on, either by showing more detailed or alternative information about it in a separate buffer or by performing an action with side-effects on it.

In Difftron:

• the hierarchy levels can be expanded/collapsed (individually or globally) to quickly switch between “overview” and “detail” views of the diff
• the diff text itself can be used to jump to the code — to the file in the current worktree if applicable, in a buffer, or a new worktree at that ref (thus allowing one to use LSP, etc. while exploring)
• the left and right-side labels can be used to select different comparisons (press “N” or “P” to move to the next/previous commit; “enter” to pull up an autocomplete across all refs)

Second, Emacs is pretty much just text with different colors, so the default data-density is quite high. I’m sure Emacs is capable of supporting excessive amounts of whitespace to match even the most “minimal, clean, modern” (i.e., useless) web UI, but you’d have to work against the grain to mess things up.

Third, the entire system is “live”, meaning that I can imagine some new feature, prompt an LLM to implement it, and then test it without restarting Emacs.

I just defined:

(defun difftron-reload ()
  (interactive)
  (when (featurep 'difftron)
    (unload-feature 'difftron t))
  (load "/Users/dev/work/difftron/emacs/difftron.el"))

and ran it whenever I wanted to test some changes.

This made the iteration loop quite fast, which made it low-friction and fun to polish away rough edges.

All in all, it took about 8 hours over two afternoons to knock out the initial implementation in Magit and record the first demo video, then another 16 hours polishing it as I used it myself and ran it by a few friends for critique.

The entire implementation was done by Codex with GPT-5.5 High (via my $20/month ChatGPT Plus subscription), and I’ve barely touched the code myself. I mainly provided guidance in terms of:

• telling it what Rust crates to use for the language analysis (Rust Analyzer’s ra_ap_syntax for Rust; arborium for Clojure and TypeScript)
• telling it to set up dedicated scripts to lint, format, and test
• telling it to set up a minimal Emacs environment that it could use to drive the package itself and reproduce firsthand any bugs I ran into

My agents.md instructions were minimal:

• when editing Emacs code, review Magit and other package source code in /emacs/test-config/straight/repos/
• review /scripts/ for available project commands like formatting and testing
• use red/green TDD workflow
  • never add tests for config or script changes

Same advice I give myself, honestly: Read the source code of what you’re using, put frequent/complex tasks into scripts, and specify the success condition before trying to implement it.

On open-sourcing a vibe-coded project

Last newsletter I said:

Mayyybe if I’m happy with it I’ll end up releasing something. But I’m not trying to collect Github stars or HN karma, so I might just happily use it in the privacy of my own home without trying to “commercialize it”.

While I am indeed happy with Difftron so far, I’ve hesitated about sharing it because of a few lingering questions in the back of my mind:

• for the code, how should I distinguish between:
  • code I thought hard about and wrote myself
  • code I prompted and reviewed
  • code I prompted and only “black box” tested
• how open am I to accepting potentially YOLO’d code from others (e.g., adding support for languages I don’t use / care about)?
• how much do I want to popularize this as a useful tool versus keep it as a playground for my own experiments?

However, all the same questions apply to the Whispertron dictation app I vibe-coded last fall, and I’m glad I released that because I’ve heard from several folks who have been enjoying it daily, which makes me happy.

Furthermore, all these concerns revolve around setting expectations, which I can simply do:

• in a project README, I can say upfront whether I intend for something to be a useful, reliable tool or a personal playground (and thus be evaluated on those merits)
• in the commit history, I can specify myself or the LLM as the author or co-author (all responsibility, of course, remains with me — it’ll just be helpful to keep track of whether some code “made sense” in a human brain or was just the result of a series of next-token predictions)

So, between that and the general principle of increasing my luck surface area, I figure I should release Difftron and any other generally useful tool I might cook up.

LLM determinism

When delegating to humans, there’s a delicate balance between:

• establishing systems and procedures to counteract our fallibility
• giving people the freedom to explore and grow (including through mistakes)

LLMs, though, can’t learn from mistakes, nor do they have creative flames that can be snuffed out by overbearing procedures, so I’m all about using them as cogs in a deterministic machine.

Contrary to the majority of GitHub repos I’ve seen lately, this cannot be done via the context window.

LLMs on their own cannot follow, for example, Graydon’s Not Rocket Science Rule of Software Engineering: “automatically maintain a repository of code that always passes all the tests”.

No matter how much you plead in markdown:

You MUST run test.sh before committing

there’s a chance they’ll just go ahead and commit anyway (or “fix” the failing test by deleting it, etc.).

If you want LLMs to follow a deterministic process, you must use them via a deterministic harness.

Understanding this idea is easy; the tricky part is actually building that harness.

The Not Rocket Science Rule, for example, is phrased in terms of git commits and usually implemented via a continuous integration server: Once a commit is pushed, a script attempts the merge, runs the tests, and updates the branch only if the merged code passed the tests.

This does maintain the desired invariant, but the feedback loop is long: The continuous integration failure occurs many minutes after the code change.

This wastes time, tokens, and (much worse!) potentially a full turn of human feedback (if the agent stopped before getting the CI feedback).

This article on Stripe’s coding agents puts it nicely:

We seek to “shift feedback left” when thinking about developer productivity. That means that it’s best for humans and agents if any lint step that would fail in CI is enforced in the IDE or on a git push, and presented to the engineer immediately.

So how can we change the harness to shorten the feedback loop and give agents the chance to correct their mistake?

• Running the test script after every tool call would shorten the loop, but this doesn’t seem like a great strategy — even if the testing time is negligible, the context would fill with spurious failures from testing unfinished work.
• Running the test script after the agent finishes is better, but we’d need to make sure the test script correctly handles cases like the agent organizing its work across multiple commits (I direct ‘em this way, as focused commits make for an easier-to-review history).
• Alternatively, we could run the agent without direct access to git, instead giving it a specific “git commit tool” that always runs the tests. (Similar in spirit to a git pre-commit hook, except without the --no-verify flag.)

I have similar implementation questions about other useful invariants beyond the Not Rocket Science Rule. E.g., how do we ensure an agent doesn’t modify the tests or write slop to the README?

• Have the harness run git status and discard illegal edits after every tool call?
• Limit tool calls, so illegal edits cannot be made (no arbitrary bash/shell, provide an “edit” tool that refuses to work on specific files/directories)?
• Allow arbitrary bash, but only within a fine-grained Linux sandbox (UNIX user permissions?)

I’m reminded of the dichotomy between static and dynamic language strategies:

• “correct by construction”, where you only give the LLM tools that allow it to do valid operations (akin to a structural editor or typed programming language where “illegal states are unrepresentable”), versus
• letting the LLM do whatever, with the harness “raising exceptions at runtime” to prevent invariant violations

For example, if we want an LLM to rename a bunch of methods/types/variables under the invariant “don’t change the program behavior”, we could:

• follow the former strategy: only give the agent access to an LSP server’s deterministic “rename” tools
• follow the latter strategy: give the agent a shell to execute any commands, then test the invariant afterwards (by, e.g., comparing the abstract syntax trees sans identifier names)

When I’m programming, I’ll switch between these strategies depending on the specific problem and my mood. For agents, it’s not obvious to me whether one of these strategies in general dominates the other.

In terms of my work as an implementer/specifier, it’s probably easier for me to define invariants via runtime tests than by designing a set of “safe” tools:

• writing a runtime assert is easier than designing a type system / algebra
• agent harnesses already come with a bunch of generic tools
• the underlying models have been trained on their tools, so even if you did define great custom ones, the models might be less effective at using them compared to their favorite bash commands.

Ultimately which strategy to pursue is an empirical question about balancing the setup hassle (for me) with the combined performance of the model and harness.

My interest in invariants is, I hope, not some midwit folly (dimwit and genius: “Just ask the model for what you want”).

Nor does it stem from the religious purity / “safety” mindset that afflicts certain programmers. (You know, the ones whose aesthetic appreciation of category theory-inspired, borrow-checked phantom types has them constructing complex defenses against technically-possible-but-rare-in-practice “problems”.)

Rather, I’m stubbornly attached to the idea of understanding what I’m building. Having more constraints — immutable data, pure functions, limited scope, etc. — makes it easier to hold a system in my head. Trying to understand code written by other people (or one’s past self) is hard enough, so if I’m going to have any chance of understanding, shaping, and directing code written by a machine, I’ll take all of the invariants I can get =D

If you have any favorite approaches for building this kind of stuff — Linux sandbox APIs, version control hooks, build-your-own-harness libraries, etc. — or if you’re interested in exploring/collaborating in this space, please let me know!

Misc. stuff

• Deterministic workflows for LLMs aren’t just about the code itself — my friend Colin built the specific workflow he wanted using the Pi coding agent:

  The core idea is: instead of asking an agent to “go build this”, the extension drives it through a structured lifecycle: refine, plan, review the plan, implement, review, fix findings, manual test, commit, and merge.

  Agents tend to get distracted by spurious context, influenced by context earlier in the conversation, etc. The extension uses Pi’s conversation tree to give each workflow step an isolated context.

• Can Financial Engineering Cure Cancer? | Andrew Lo

• Buttery Smooth Emacs: This flickering is a predictable consequence of the “do what the fuck I want when I want it” redisplay strategy Emacs uses. There’s no coordination between your video card, your GUI system, Emacs, and your sinful soul.

• Akin’s Laws of Spacecraft Design

• Improve your code by separating mechanism from policy

• The Mystery in the Medicine Cabinet: Acetaminophen, ibuprofen, and what doctors probably want you to know.

• Legibility is Ruining You

• “Value capture occurs when an agent’s values are rich and subtle; they enter a social environment that presents simplified - typically quantified — versions of those values; and those simplified articulations come to dominate their practical reasoning.”

• AI & Software Quality with Shawn Wang (aka swyx). “To succeed in technology and AI you need to have a sincere yearning for science fiction. If you don’t fundamentally believe or want to achieve science fiction and make the thing that you saw on TV, the thing that you talk about at late night or on podcasts, you don’t fundamentally want to make that real, then you’re just criticizing things as like "oh well, yeah, everything is hard and we’re not there yet and we’ll never be”, then you’re not going to be part of that conversation because you’re not trying to be. And it’s really difficult to embrace because I haven’t been that, right?“

• no hello

• Heterofriendly: The Intuition for Why You Always Need Robust Standard Errors – Data Colada

• All 11.333.878 buildings in the Netherlands

• RE#: how we built the world’s fastest regex engine in F#. "intersection and complement are genuinely useful operators that have been missing from regex engines for far too long. being able to describe what you want as a combination of properties, rather than cramming everything into one monolithic pattern, is a much more natural way to think about matching. and now you can do it with linear-time guarantees.”

• A Two Minute Video, Two Years in the Making

• Heuristics for lab robotics, and where its future may go

• How to Make a Living as an Artist.

• On The Need For Understanding

Permalink

Data Governance in Versioned Systems

require('[datahike.api :as d])

;; Before purge
d/q('[:find ?n :where [?e :name ?n]] @conn)
;; => #{["Alice"] ["Bob"]}

;; Purge Alice (requires :keep-history? true)
d/transact(conn [[:db.purge/entity [:name "Alice"]]])

;; After purge — gone from current state AND from the new commit's history
d/q('[:find ?n :where [?e :name ?n]] @conn)
;; => #{["Bob"]}

d/q('[:find ?n :where [?e :name ?n]] d/history(@conn))
;; => #{["Bob"]}

(require '[datahike.api :as d])

;; Before purge
(d/q '[:find ?n :where [?e :name ?n]] @conn)
;; => #{["Alice"] ["Bob"]}

;; Purge Alice (requires :keep-history? true)
(d/transact conn [[:db.purge/entity [:name "Alice"]]])

;; After purge — gone from current state AND from the new commit's history
(d/q '[:find ?n :where [?e :name ?n]] @conn)
;; => #{["Bob"]}

(d/q '[:find ?n :where [?e :name ?n]] (d/history @conn))
;; => #{["Bob"]}

require('[superv.async :refer [<?? S]])

;; 1. Purge — produces a new commit whose roots no longer reach Alice.
d/transact(conn [[:db.purge/entity [:name "Alice"]]])

;; 2. gc-storage WITHOUT a cutoff only reclaims storage on deleted branches.
;;    The pre-purge commit on :db is a live intermediate commit, so its
;;    tree nodes (containing Alice) survive this sweep.
<??(S d/gc-storage(conn))

;; 3. gc-storage WITH a cutoff is what physically evicts those nodes.
;;    Pick a cutoff that exceeds your longest-running reader's lifetime.
let [seven-days-ago new java.util.Date(System/currentTimeMillis() - 7 * 24 * 60 * 60 * 1000)]:
  <??(S d/gc-storage(conn seven-days-ago))
end

(require '[superv.async :refer [<?? S]])

;; 1. Purge — produces a new commit whose roots no longer reach Alice.
(d/transact conn [[:db.purge/entity [:name "Alice"]]])

;; 2. gc-storage WITHOUT a cutoff only reclaims storage on deleted branches.
;;    The pre-purge commit on :db is a live intermediate commit, so its
;;    tree nodes (containing Alice) survive this sweep.
(<?? S (d/gc-storage conn))

;; 3. gc-storage WITH a cutoff is what physically evicts those nodes.
;;    Pick a cutoff that exceeds your longest-running reader's lifetime.
(let [seven-days-ago (java.util.Date. (- (System/currentTimeMillis)
                                         (* 7 24 60 60 1000)))]
  (<?? S (d/gc-storage conn seven-days-ago)))

Permalink

Advent of Code 2015 days 3-8 in Clojure

Yeah, I'm starting to think my initial comment about taking until the heat death of the universe to complete this wasn't far off the mark. I'm not really slow at finishing the tasks, it's just between working , the drudgery of everything else in life and possible undiagnosed ADHD, progress becomes a sporadic thing. Gone forever I think are the late-nighters of chugging caffeinated drinks...

But with that said, I have been slowly chugging away and gotten myself up to the heady heights of puzzle number 8.

I don't think I shall labour over each and every puzzle, instead I'll just gloss over the interesting parts.

Day 03

You're trying to determine how many points are visited if you're given a list of directions (N/S/E/W effectively). One pattern with this sort of thing I like to do is to store the position transformation as a map with the keys being the direction and the values being values to add to x and y coordinates.

(def moves {\^ [0  1]
            \v [0 -1]
            \> [1  0]
            \< [-1 0]})

The rest is just bookkeeping ;D

Day 04

Trying to mine a proprietary cryptocoin. I just used Java interop here - (java.security.MessageDigest/getInstance "MD5"). It is then a matter of looping through numbers to get the required number of leading zeros in the hexadecimal representation of the digest.

This was pretty brute-force and I don't expect to retire from crypto riches any time soon. It's quite slow too - clearly the hashing is going to be but I suspect we could possibly run multiple threads in parallel and/or optimise the way we're creating the byte-array to feed into Java each loop.

As it is, I was content just to wait for it to finish...

Day 05

String parsing really. I'm not a massive fan!

Was nice going back to see part of my original solution looks like this;

(defn nice?
  "is a word nice?"
  [word]
  (and (has-three-vowels? word)
       (has-repeated-letter? word)
       (no-forbidden-strings? word)))

Pretty legible. I guess you'd write something similar in most languages, but the convention of adding a question mark to tests is a nice touch.

Day 06

Mapping text instructions into turning a series of lights on/off/toggling in a given rectangle. Nothing to write about but I was interested to see if anyone had plotted the result in case there was a picture.

There isn't.

Day 07

Ah, we're getting interesting now, we had to simulate a basic logic circuit. We're given a input like;

123 -> x
456 -> y
x AND y -> d
x OR y -> e
x LSHIFT 2 -> f
y RSHIFT 2 -> g
NOT x -> h
NOT y -> i

I suspect there's a proper way of doing this in which we create a graph of connections and evaluate everything in the proper order. But what I did was just to brute force the whole thing; just keep running through the instructions until every instruction could be executed e.g. if you imagine in the example above that the numbers aren't applied to x and y until the last line, then we couldn't perform most of it until the second loop round the instructions.

Yes, as I edit this, I'm seeing a theme forming here where my solutions are just brute forced.

Day 08

String parsing again :D Trying to count the number of characters in a string and in the final representation. Not much to say again. Slogged through it.

Parser?

So far each problem's input is basic text, but I'd be interested to see if there's a tidy way of specifying the puzzle input. For example Day 2's input looks like;

1x2x3
1x3x6
...
...

So I'm thinking we almost want a function like;

(parse-input "day02.input" "<int>x<int>x<int>")

;; -> ((1 2 3) (1 3 6) ... ...)

Usually I'd either split or use regex for input, but it seems like what I'd like is to also be able to specify the basic type. Puzzle 8's example;

London to Dublin = 464
London to Belfast = 518
Dublin to Belfast = 141

Would be;

(parse-input "day08.input" "<string> to <string> = <int>")

;; -> (("London" "Dublin" 464) ...)

Or even better!

(parse-input "day08.input" "<string:from> to <string:to> = <int:distance>")
;; -->
({:from "London" :to "Dublin" :distance 464} ...)

I'd be interested if you know of anything like this feature.

End

I'll carry on with some more problems. I don't think I'll be entering any leaderboards soon ... unless there are some for slowest?

Permalink

Machine learning using Clojure, libpython-clj2, and Pytorch

Machine learning explained using the parabola example

Permalink

Clojure Deref (May 26, 2026)

Permalink

Designing Kolam in Clojure Through Visit Order and Tangency

A geometry notebook about drawing curved paths inspired by South Indian floor art

Permalink