Finding transitive var usages with clj-kondo

Today fellow Clojurian Søren Knudsen asked the following question on Clojurians Slack:

Say I&aposd like an overview of which fns in my Clojurescript app don&apost have :x metadata and aren&apost children of functions that have :x. I&aposd love this overview as data.

Anyone know a relevant analysis tool for this purpose?

Let&aposs represent this problem in code form. Read it from bottom to top.

(defn grandchild [] ; no :x, but reachable via child: ignore
  :leaf)

(defn child [] ; no :x, called by ^:x grandparent: ignore
  (grandchild))

(defn ^:x grandparent [] ;; has :x metadata, ignore
  (child))

(defn standalone [] ;; has no :x metadata and not reachable from anything with :x metadata, include
  :other)

It turns out that clj-kondo analysis data is well suited to solve this problem. In this blog post, let&aposs write a babashka script, that uses the clj-kondo pod. This bit of setup lets you do that. Of course you could also use clj-kondo as a regular JVM dependency, but we&aposre going for ease here, since it&aposs just a tiny script at this point.

#!/usr/bin/env bb
(require &apos[babashka.pods :as pods]
         &apos[clojure.set :as set])

(pods/load-pod &aposclj-kondo/clj-kondo "2025.06.05")
(require &apos[pod.borkdude.clj-kondo :as clj-kondo])

Clj-kondo lets you find var-definitions and var-usages. Clj-kondo can also include var metadata. The arguments to clj-kondo&aposs run! API function then should look like this:

(def analysis
  (-> (clj-kondo/run! {:lint ["src"]
                       :config {:analysis {:var-definitions {:meta [:x]}
                                           :var-usages true}}})
      :analysis))

To illustrate how it works, we&aposll introduce a multi-namespace project:

;; src/app/core.cljs
(ns app.core
  (:require [app.util :as util]))

(defn ^:x grandparent []
  (util/child))

(defn standalone []
  :other)

;; a top level var usage, not inside any var definition:
(util/child)

;; src/app/util.cljs
(ns app.util)

(defn grandchild []
  :leaf)

(defn child []
  (grandchild))

To illustrate what a var usage looks like in clj-kondo&aposs analysis data, let&aposs look at the usage in app.core of util/child:

{:from app.core
 :from-var grandparent
 :to app.util
 :name child
 ...}

The :from key describes from which namespace the reference was used. The :from-var key describes in which var definition the var was used, and this is the key ingredient of tracking transitive var usages. The :to + :name keys describe which var was used.

Vars

In clj-kondo&aposs analysis you can request metadata from vars with :meta [:x] (or all metadata with true). To distinguish all project vars from those that have :x metadata we can do the following:

(defn fq [ns name] (symbol (str ns) (str name)))

(def defs (:var-definitions analysis))
(def project-vars (set (map #(fq (:ns %) (:name %)) defs)))
(def with-x (set (keep #(when (-> % :meta :x) (fq (:ns %) (:name %))) defs)))

Here project-vars is a set of symbols of all the project vars and with-x are only those that have :x metadata.

Building the call graph

Now we&aposre ready to build the call graph that lets us solve our problem. In the following we&aposre making a map that looks like: caller -> callees, but we limit callees only to project vars since we&aposre not interested in vars like cljs.core/assoc, reagent.core/atom etc.

(def graph
  (reduce (fn [g {:keys [from from-var to name]}]
            (let [callee (fq to name)]
              (if (and from-var (contains? project-vars callee))
                (update g (fq from from-var) (fnil conj #{}) callee)
                g)))
          {}
          (:var-usages analysis)))

The from-var condition leaves out any top level var usages. The (contains? project-vars callee) takes care of filtering only on project vars. After running this, we&aposll end up with a graph (map) that looks like:

{app.core/grandparent #{app.util/child}
 app.util/child       #{app.util/grandchild}}

So app.core/grandparent calls app.util/child and app.util/child calls app.util/grandchild.

Transitive reachability

Next we write a function to find out what vars are reachable from a set of vars starts.

(defn reachable [starts]
  (loop [seen #{}
         todo (set starts)]
    (if (empty? todo)
      seen
      (let [seen (into seen todo)
            used-vars (set (mapcat graph todo))
            unvisited (set/difference used-vars seen)]
        (recur seen unvisited)))))

(def children (set/difference (reachable with-x) with-x))

(prn {:graph graph
      :with-x with-x
      :children-of-x children
      :without-x (set/difference project-vars with-x children)})

The reachable function just calculates the transitive closure of the graph, given a set of starting nodes (vars). The children var is the set of reachable vars without the starting points (the vars with :x metadata).

{:graph {app.core/grandparent #{app.util/child}
         app.util/child #{app.util/grandchild}}
 :with-x #{app.core/grandparent}
 :children-of-x #{app.util/child app.util/grandchild}
 :without-x #{app.core/standalone}}

So the answer we were looking for is #{app.core/standalone}. This function is neither a transitive child of any function with :x metadata, nor does it have any :x metadata itself.

The code

Here&aposs the full script once again.

#!/usr/bin/env bb
(require &apos[babashka.pods :as pods]
         &apos[clojure.set :as set])

(pods/load-pod &aposclj-kondo/clj-kondo "2025.06.05")
(require &apos[pod.borkdude.clj-kondo :as clj-kondo])

(def analysis
  (-> (clj-kondo/run! {:lint ["src"]
                       :config {:analysis {:var-definitions {:meta [:x]}
                                           :var-usages true}}})
      :analysis))

(defn fq [ns name] (symbol (str ns) (str name)))

(def defs (:var-definitions analysis))
(def project-vars (set (map #(fq (:ns %) (:name %)) defs)))
(def with-x (set (keep #(when (-> % :meta :x) (fq (:ns %) (:name %))) defs)))

;; caller -> callees, project vars only
(def graph
  (reduce (fn [g {:keys [from from-var to name]}]
            (let [callee (fq to name)]
              (if (and from-var (contains? project-vars callee))
                (update g (fq from from-var) (fnil conj #{}) callee)
                g)))
          {}
          (:var-usages analysis)))

(defn reachable [starts]
  (loop [seen #{} todo (set starts)]
    (if (empty? todo)
      seen
      (let [seen (into seen todo)
            used-vars (set (mapcat graph todo))
            unvisited (set/difference used-vars seen)]
        (recur seen unvisited)))))

(def children (set/difference (reachable with-x) with-x))

(prn {:graph graph
      :with-x with-x
      :children-of-x children
      :without-x (set/difference project-vars with-x children)})

Conclusion

I hope you learned how useful clj-kondo analysis data can be for tracking relations between vars and that you can use this data in casual babashka scripts as well!

Permalink

New library: biff.core

As I wrote about previously, I've been working on splitting Biff up into a bunch of separate libraries and changing various things along the way. I've completed a rough draft of all twelve libraries and am now going through them one-by-one to polish and release them. The first library is now ready.

biff.core: system composition and other interfaces for Biff projects. This is the glue that holds all the other libraries together, and that's why I'm releasing it first.

For a long time Biff has had this "modules and components" structure where each application namespace in your project exposes a "module" map, then you have a bunch of boilerplate to combine stuff from those modules into a single "system" map, and then we thread the system map through your "component" functions on startup. Biff 2 retains that structure, and it has some additional stuff to deal with that boilerplate.

For an example of what I'm talking about, see this code which takes the :routes (and :api-routes) keys from your modules and turns them into a :biff/handler value for the system map. I wanted a first-class way to be able to extract that kind of logic cleanly into a library so that the library's instructions can just be "add this module to your project" without an accompanying "and then paste all this stuff into your main namespace."

So this new biff.core library includes a concept of "init functions." These are functions that take a collection of modules and return a single map that can be merged into your system map. Ta da. Here's an example. Init functions are stored in the :biff.core/init key in your module maps, so we get that nice "all you need are modules (well, and components)" effect.

The main complication here is that the boilerplate of defining a (def handler ...) var in your application code actually has a nice side benefit: late binding. If you change any of your modules, the handler var will get updated, and if you set :biff/handler in your system map to the var instead of the value (#’handler), incoming Ring requests get the latest handler without you having to restart the web server. If we extract that boilerplate into library code, we don't get the var.

I ended up on this solution:

  • Init functions take a var of your modules vector, not the vector value itself.
  • Anything in the system map that you want to get updated without a restart needs to be a function. In some cases this means instead of setting a :com.example/my-thing key on the system map, you need to set a :com.example/get-my-thing function which returns my-thing.
  • That function on the system map should dereference the modules var and pass it to a memoized function that builds whatever thing it is you need (like the Ring handler).

Again, see this example. The result is kind of aesthetically pleasing: you get a nice clean main namespace that shouldn't need to change much, and all you do is add modules and components.

There's always the temptation to consolidate things further. Why even have a separate components vector? Why not have modules support :biff.core/on-start and :biff.core/on-stop keys and then have some way to express dependencies between these lifecycle functions so we can call them in the right order?

And the answer is so that we don't have to have some way to express dependencies between these lifecycle functions so we can call them in the right order. It's not that hard to put the components in the right order yourself (especially since the Biff starter project does that for you), and then it's easier to understand how components work. It's just a sequence of functions that you pass a map through. If you work on a project with so many stateful resources that it's hard to keep track of them all, you can always layer something on top that figures out what your components vector should be before you pass it to biff.core.

Plug: my team is hiring for a senior software engineer, writing ClojureScript and Python mostly. We make modeling software for renewable energy projects.

Permalink

Clojure Deref (Jun 9, 2026)

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem (feed: RSS).

Clojure/Conj CFP Closing

Less than one week to submit a talk for the 2026 Clojure/Conj. Don’t miss your chance!

Apply to the CFP

Clojure/Conj 2026 is Sep 30 - Oct 2 in Charlotte, NC, USA.

Selected Highlights

Squint added a browser REPL. Go beyond hot-reloading your Squint code. Now you can eval on the live page over nREPL. Try it out!

Jolt takes SCI to new places: the Janet runtime. If you like Janet’s lightweight runtime, but you want a full Clojure dialect, you might be interested in trying the early version.

Do you want to run Clojure in outer space? Why not fly it there yourself? Check out sfsim. It aims to be a realistic 3D space flight similator. See if you can take off, orbit the planet, make it home safe, and live to code another day.

Who needs Twitch when you can just stream your terminal to the web? Try Atomstream. To use it, swap out charm.clj, launch your application in the terminal, and watch all your TUI goodness dance before your eyes in the browser.

EDN and Transit in C? Was that on your bingo card for 2026? Perhaps not, but if you’re in the market for a native, high-performance, dependency-free implementation, you may want to take a look.

Upcoming Events

Podcasts, videos, and media

Blogs, articles, and news

Libraries and Tools

Debut release

  • jolt - A Clojure interpreter running on Janet

  • sfsim - Space flight simulator (under development)

  • magit-difftastic - Integrate difftastic into magit

  • atomstream - Cross-render TUIs to the Web

  • transit.c - A data interchange format and set of libraries for conveying values between applications written in different programming languages.

  • kontor - A trans-national accounting kernel with modules for all business aspects.

  • parens-to-production - How to build your next Clojure application with SSR with Datomic

  • jon-nested-menu - Nested MUI menus for Reagent/ClojureScript and React: a dropdown, a right-click context menu, per-item icons, custom labels and keyboard navigation.

  • biff.core - Defines the interfaces and code that connect all the other Biff libs

Updates

  • core.async 1.10.874-alpha3 - Facilities for async programming and communication in Clojure

  • tools.deps.edn 0.9.38 - Reader for deps.edn files

  • spel 0.9.8 - Idiomatic Clojure wrapper for Playwright. Browser automation, API testing, Allure reporting, and native CLI - for Chromium, Firefox, and WebKit

  • ansatz 0.1.22 - Dependently typed Clojure DSL with a Lean4 compatible kernel.

  • guardrails 1.3.3 - Efficient, hassle-free function call validation with a concise inline syntax for clojure.spec and Malli

  • fulcro-tui 1.0.2 - A JLine-base TUI wrapper, allowing you to write TUI projects in CLJ or Babashka via Fulcro

  • fulcro-rad-tui 1.0.0-RC2 - UI Plugin for Terminal UI support in Fulcro RAD (fulcro-rad-statecharts)

  • calva 2.0.590 - Clojure & ClojureScript Interactive Programming for VS Code

  • clj-tg-bot-api 1.2.266 - 🤖 The latest Telegram Bot API spec and client lib for Clojure-based apps

  • aleph 0.9.9 - Asynchronous streaming communication for Clojure - web server, web client, and raw TCP/UDP

  • rephrase 1.0.3 - Rephrase exceptions to be more beginner-friendly

  • c4k-keycloak 2.1.0 - a kubernetes deployment for keycloak

  • phel-lang 0.42.0 - A functional, Lisp-inspired language that compiles to PHP. Inspired by Clojure, Phel brings macros, persistent data structures, and expressive functional idioms to the PHP ecosystem.

  • awesome-backseat-driver 1.0.15 - Plugin marketplace for Clojure AI context in GitHub Copilot: agents, skills, and workflows for REPL-first interactive programming with Calva Backseat Driver

  • diehard 0.12.1 - Clojure resilience library for flexible retry, circuit breaker and rate limiter

  • calva-backseat-driver 0.0.35 - VS Code AI Agent Interactive Programming. Tools for CoPIlot and other assistants. Can also be used as an MCP server.

  • teensyp 0.6.0 - A small, zero-dependency Clojure TCP server that uses Java NIO

  • pg-datahike 0.1.44 - Postgres compatibility layer for Datahike.

  • cli-tools 1.0.0 - CLIs and subcommands for Clojure or Babashka

  • svar 0.7.11 - Type‑safe LLM output for Clojure. Works with any text‑only model.

  • honeysql 2.7.1388 - Turn Clojure data structures into SQL

  • cli 0.11.70 - Turn Clojure functions into CLIs!

  • yggdrasil 0.2.28 - Git-like, causal space-time lattice abstraction over systems supporting this memory model.

  • spindel 0.1.18 - Cross-platform FRP runtime with a git-like memory model.

Permalink

Terraform Is Day 1. Day 2 Needs an Infrastructure Package

Terraform is excellent at standing infrastructure up, but Day 2 work often begins after apply: DNS updates, cache purges, configuration, notifications, and handoffs to other tools. BigConfig packages keep Terraform as a powerful implementation detail while exposing code-first, local-first, and agent-first lifecycle verbs. The BigConfig SDK supports TypeScript, Python, and Clojure.

Permalink

clj.rs Clojure implemented on Rust

Notes

  • clj.rs could be a big thing
    • Clofer collapsed:: true
    • clj.rs
      • Gives me breathing space
      • https://clj.rs
      • Casey Marshall
      • Freedom from Java / Oracle
      • Actively Maintained!!
        • Will Clojure community will ensure active maintenance?
      • Jank
        • Clojure on C++
        • LLVM
        • Pointers
      • The Garbage collection
        • In Ruby it was a big topic
      • Fast
      • Huge set of libraries
      • GPU!
      • No more JavaScript?
        • I hate JavaScript
      • Embedded Systems
      • OS programming
      • Data Science
      • Clojure on Zed editor?
        • Jpyter Lab
        • VSCode is slow
          • Take up resources
        • M$ - common who really trusts them?
      • Less nightmare for Rust dev’s
        • Imagine the productivity gain!!!!
      • Please rally behind this project
      • No more Clofer, grow, and long live https://clj.rs/
      • -ve’s
        • Claude is used
          • A closed model
        • Only 1 person project
        • Maybe the world does not know
        • Underwhelming reaction
      • What I will do
        • Try to write some Clojure programs with clojurust
          • Learn Rust, make it learnable.
        • Watch https://clj.rs , report about it
          • Contribute to it if I can (it’s a big if)
        • Update my Clojure Book
        • Write a Lisp compiler in Rust, just for Hobby
        • Contribute to Stats With Clojure
    • Clojure should keep moving collapsed:: true
      • Rust is a great idea now
      • Who knows what next?
      • Clojure foundation
        • Official Clojure implementation on various runtimes
        • Standard Clojure API
        • Target many platforms
          • Migrate / compile to many platforms

Permalink

How to Scaffold BigConfig Packages with Claude Code Skills

This tutorial shows how to install and try the three Claude Code skills that scaffold minimal, launcher-conformant BigConfig packages in TypeScript, Python, and Clojure. Each starter package includes validate and build verbs, a root run file, packaged resources, tests, and the shape needed to run through bc-pkg.

Permalink

Slowed Down

I slowed down. I don’t know why — maybe I have a lot of work to do, maybe my brain browned out. Maybe it’s because I am leading a team of quants in India, and we are expanding our business. So maybe that slowed me down. Another reason is Rust is proving to be a bit difficult for me. I was reading and I have read till enums, and I couldn’t grasp the concepts as quickly as I thought I would.

I really miss this book, PHP: The Complete Reference. When I read it, I picked up PHP in a jiffy.

Everything had complete examples. Whatever code that was newly added was highlighted and called out, and it was really easy to follow. This was before LLMs, and I think even before Stack Overflow became really popular.

Compared to PHP: The Complete Reference, the Rust book is no match. The Rust community doesn’t want beginners to learn Rust — it wants advanced programmers to learn Rust, which could even be fatal for the language in the long run.

So I am looking for new sources apart from the Rust book, and I have zeroed in on a freeCodeCamp video.

Maybe this video has a practical demonstration of how to code with Rust, and maybe, by watching the video and typing along side by side, I can learn Rust quicker.

So that’s about it for this week. Let me report back next week on what I’m up to. I also miss Clojure. Compared to Clojure, Rust is super difficult — you don’t have a REPL and all those things. I just really miss my Clojure projects, and I want to contribute to them because I love Clojure. Let’s see how things go, and let’s see if I forge ahead or give up. I don’t think I’ll give up before I write a Lisp interpreter in Rust. I really want to learn to write a compiler.

Permalink

Migrating from a Monolithic Orchestrator to Apache Airflow

Photo by Corinne Kutz on Unsplash

Before we knew better

Our orchestration system started as a simple internal solution to manage event pipelines and trigger downstream jobs. Over time, as more workflows and dependencies were added, it gradually evolved into a tightly coupled monolithic scheduler that became increasingly difficult to understand and maintain.

Understanding how a workflow executed often meant looking through multiple files, configurations and database tables.

For newer team members, onboarding into the system took time because much of the workflow context was distributed across different parts of the codebase. Even relatively small changes required careful coordination to ensure existing pipelines continued to work as expected. Similarly, debugging typically involved manually tracing logs and rerunning jobs to better understand execution behavior.

Limitations of our legacy design

We had a monolithic architecture written in Clojure that bundled all our event pipelines together, added dependencies between them and triggered a Lambda function.

Legacy Workflow

This Lambda function added a single monolithic step to the EMR cluster. If there was an issue in any one of the pipelines, the entire flow would fail due to the single step on the cluster.

We did not have step-wise monitoring in the old design, so during on-call situations it became very difficult to identify which part of the pipeline was causing the issue.

Photo by Tim Gouw on Unsplash

There was no single place to answer basic questions like:

  • What runs first?
  • What happens if this step fails?
  • How do I re-run just one part safely?

The scheduler worked, but it was hard to understand, hard to maintain and even harder to explain. That’s when we realized we needed a better way.

What we actually needed

Our aim was less about fancy scheduling features and more about making our daily work easier and more reliable.

  1. Simpler onboarding, less mental overhead

Our existing step scheduler was built in Clojure and came with a steep learning curve. New engineers had to spend a lot of time just understanding how jobs were defined, chained and monitored before they could confidently make changes. In contrast, moving to Apache Airflow and its Python-based DAGs made workflows far more readable and intuitive. The structure of dependencies, retries and scheduling became obvious at a glance and because Python was already familiar to a much broader set of engineers, onboarding became significantly faster. The shift not only improved maintainability but also made the scheduler more accessible to the wider engineering team.

2. One failure shouldn’t kill everything

In our old setup, if one step in a multi-step job failed, the entire job would terminate. This was painful, especially for long-running pipelines. A single transient issue like a network glitch or a temporary dependency failure meant rerunning everything from scratch. We needed better fault isolation, where a failed step doesn’t automatically bring down the whole workflow.

3. Clear visibility and easy recovery from failures

When something failed, it wasn’t always obvious what failed or why. We wanted a clear view of each step or task, with the ability to:

  • Quickly see which step failed
  • Get alerted when failures happen
  • Re-trigger only the failed step instead of restarting the entire job

In short, we were looking for a workflow system that was easier to understand, more forgiving when things go wrong and faster to recover from failures without adding more operational complexity.

Why we chose Apache Airflow

Once we were clear with our needs, Apache Airflow was a good fit. Airflow is designed to act as a robust orchestration platform.

  1. Airflow Was Already Available and Production-Ready

One of the biggest factors was that Apache Airflow was already set up and running in our environment. This allowed us to avoid the upfront cost of evaluating, deploying and writing a new scheduler from scratch. Instead, we could focus on solving workflow problems rather than infrastructure problems.

2. Better Observability and Debugging

Airflow provides rich visibility into workflows:

  • Task-level logs
  • Retry history
  • Execution timelines
  • Clear success/failure states
  • Better dependency management between tasks

This significantly improved our ability to debug failures compared to the limited visibility offered by the step scheduler.

3. Support for Retries, Backfills and Scheduling

Apache Airflow provides core orchestration capabilities such as configurable retries, historical backfills, and flexible time- and event-based scheduling out of the box. These features are built into the platform and require minimal configuration. As a result, we were able to eliminate a large amount of custom logic from our legacy scheduler. This simplified our workflows and reduced long-term maintenance overhead.

4. Separation of Orchestration and Business Logic

With Airflow, orchestration logic lives in DAGs while business logic lives in independent services or jobs. This separation made workflows easier to test, reason about and evolve over time — something that was harder to enforce with a step scheduler.

Migration approach

The first step in the migration was mapping the existing scheduler concepts into Airflow primitives.

  • Each pipeline was converted into an Airflow DAG.
  • Existing steps became Airflow tasks/operators.
  • Dependencies were explicitly defined using DAG relationships instead of implicit execution logic.

This made the execution flow much easier to understand and visualize.

In the new setup, Airflow is responsible for workflow coordination, scheduling DAGs, managing task dependencies, handling retries, monitoring execution and sending notifications. The compute-heavy processing workloads such as Spark, Hive and Presto jobs run independently on transient EMR clusters.

The execution flow starts when a scheduled trigger initiates a DAG run in Airflow. Airflow then creates an EMR cluster and submits the required processing steps using EMR operators. After submitting the processing steps to EMR, Airflow monitors their execution using EMR Sensors such as EmrStepSensor.

During this process the sensor continues polling until the step reaches a terminal state such as COMPLETED, FAILED or CANCELLED. If the step completes successfully, Airflow marks the sensor task as successful and proceeds with downstream tasks. If the step fails, Airflow immediately updates the task status and triggers the configured retry policies, failure callbacks or alerting mechanisms.

This allows Airflow to track the progress of jobs while EMR handles the actual processing and compute resources. The Airflow UI reflects the current state of each task, making it easy to track long-running jobs and troubleshoot failures.

In addition to status monitoring, Airflow captures execution metadata and integrates with EMR logs stored in locations such as S3 and CloudWatch. This allows us to quickly investigate failed jobs by navigating from the Airflow task instance to the corresponding EMR step logs.

Once all submitted steps have completed successfully, Airflow initiates cluster termination using EmrTerminateJobFlowOperator. Since we use transient clusters, this ensures that compute resources exist only for the duration of the workflow, helping optimize infrastructure costs and avoid idle cluster usage.

Finally, Airflow updates the DAG status, records execution metrics and sends notifications through configured channels such as Slack and email. This end-to-end orchestration model provides a clear separation between workflow management and data processing while improving observability, reliability and operational efficiency.

Separating business logic

Previously, orchestration and execution logic were tightly coupled in the same system. During migration:

  • Spark jobs, Hive queries and scripts were extracted into reusable components.
  • Airflow was used purely for orchestration.
  • DAGs became lightweight and focused only on workflow coordination.

This improved maintainability and simplified testing.

Incremental rollout strategy

Instead of migrating everything at once:

  • Low-risk workflows were migrated first
  • Legacy scheduler and Airflow ran in parallel initially
  • Compared workflow executions across both systems over a defined observation window, validating not only data correctness but also runtime behavior and infrastructure efficiency. Key metrics such as total execution time, EMR instance-hours consumed, job completion rates and failure patterns were analyzed across multiple pipeline runs.
  • Critical pipelines were migrated gradually

This minimized operational risk during the transition.

Logging and observability improvements

The previous system required manually tracing logs across multiple components.

With Airflow:

  • Each task had centralized logs
  • DAG execution history became visible through the UI
  • Failures could be traced directly from task graphs

This reduced troubleshooting effort for developers.

Success metrics (post migration)

Conclusion

Migrating from a monolithic orchestration system to Apache Airflow was more than just a technology change, it was an architectural shift toward building a more scalable, maintainable and observable workflow platform.

By decoupling orchestration from business logic, standardizing workflow management and leveraging Airflow’s built-in capabilities such as scheduling, retries and monitoring, we significantly reduced operational complexity and improved developer experience.

While there are still areas we plan to improve, moving to Airflow gave us a solid foundation to scale our data and event-driven workflows more reliably in the future.

Migrating from a Monolithic Orchestrator to Apache Airflow was originally published in helpshift-engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.

Permalink

Putting Code Under a Microscope: Wavelet-Based Context for LLMs

Every developer who has tried an AI coding tool is familiar with the problem of watching the model fumble with the codebase to find relevant sections to edit. Since it's not possible to load an entire codebase into the context for large projects, it greps through a few files to give it some context, and guesses what to do next. But code has a hierarchical structure with layers and boundaries. Functions sit inside classes. Classes live in files. Files make up modules. One 400-line file can contain six different conceptual areas, each with its own distinct purpose.

When a human developer reads the code, we leverage the structure when we try to understand it. We examine the way files and classes are organized, and try to find relevant logic based on that. Wouldn't it be nice if the agent could, like you, zoom in and out of the code so that it could look at the big picture, then jump directly to the exact function it needs without having to open the entire file.

That’s what WaveScope does. It’s an MCP server that uses wavelet transforms to give LLMs a multi-resolution view of the codebase. Imagine something like progressive image loading, but for source files. Even though newer models can handle large contexts on paper, what happens in practice is that they start losing focus. The more context the agent has the harder it is to figure out what it actually needs to work on, and what to prioritize leading to context rot.

Currently, there are two main ways of dealing with the problem. Grep-based search finds exact matches but ignores structure since pattern matching on individual lines can’t tell you where a class boundary is or where an error handling region starts. Embedding-based RAG is another approach which understands semantic meaning but loses position and structure. Neither gives the model a real sense of the architecture of the code.

Code Has a Rhythm

Open a file in any language like Clojure, TypeScript, Rust, or Go and you’ll see repeating common structures throughout the code. Imports sit at the top. Class and function definitions pop up at regular intervals. Indentation also has its own distinct pattern in every language. Comment blocks and blank lines are pauses in between. What if there was some way to extract these patterns, and create a structure similar to an AST without actually having to know the syntax of the language.

Luckily for us, wavelets were made for processing exactly this sort of signal by decomposing it into multiple scales at once. The transform gives you all the fine details along with the large-scale patterns at the same time. This family of algorithms is very versatile and has been used in many different fields. Seismologists use them to spot earthquakes, doctors sharpen MRI scans with them, and audio engineers separate basslines from vocals. Code structure, it turns out, is just another kind of signal that can be decomposed in a similar fashion.

So how does that actually work? Once each line has a score, the file can be treated as a 1D signal representing a sequence of numbers that rises and falls with the density of the structure. The Ricker wavelet is a little template shaped like a bump with a dip on each side. You slide it across the signal one position at a time and, at every position, measure how well the signal underneath matches that shape. A strong match means there's an elevated region sitting between two quieter regions which suggests a structural boundary. The output is a coefficient at every line scored on how much it looks like a boundary.

The trick for encoding different resolutions lies in the width of the template. You can slide the same shape stretched to many widths, each representing a different scale. A narrow wavelet only matches when the elevated region is a line or two wide, so it fires on small, sharp features. A wide one ignores line-level noise and responds to larger regions elevated relative to their surroundings, firing for big structures. So, the next trick is to run multiple widths at once to see boundaries which light up across a band of widths rather than just one. Features consistent across different scales tend to be genuine structural edges that we care about.

Since code structure nests at different sizes, each size shows up at the width that fits it. We can see a concrete example of how this works by running WaveScope on its own src/context.ts. A single line such as a lone import statement, or export class FileContext { declaration stands out over its quieter neighbors making it a sharp one-line spike that the fine, narrow wavelets lock onto. Its coefficient peaks at scale 1 or 2 and fades away at wider scales. At the other extreme is the long keyword-dense cascade inside the inferLabel method, which has a wide run of consecutive if (tokens.includes("class")), interface, enum, and struct branches. That whole region reads as one broad elevated plateau, and its coefficient climbs steadily as the wavelet widens to roughly 0.5 at scale 16, 1.0 at scale 32, 1.3 at scale 64, and about 2.3 at scale 128, where it peaks as the strongest structural response in the entire file. The biggest structure produces the strongest coarse-scale signal, which is exactly the spot you want surfaced first when you zoom out.

The line scores that feed all of this come from the same pass. In our example, export class FileContext { scores 1.6 because the class keyword weight of 1.0 and export at 0.6, the one-line get lineCount() accessor scores about 0.58, the readonly field declarations beneath it score roughly 0.08 each, while comments and blank lines around the class score a flat 0.0. The class declaration towers over its own body, the body towers over the whitespace around it, and the wavelet reads those relative heights at every width.

The intuition above is the front half of the pipeline where you score every line into a 1D signal, then slide the Ricker wavelet across it at eight scales which are 1, 2, 4, 8, 16, 32, 64, and 128 lines giving us a coefficient at every line and width. Next, we need a couple more steps to turn raw coefficient arrays into something the model can use.

First is to do multi-scale peak detection where the arrays are scanned for local maxima and ranked by magnitude. Strongest boundaries represent features such as the beginning of a class or the transition between imports and code. Because a real boundary shows up at several adjacent scales, as we saw above, these repeats can be safely collapsed into a single peak to avoid flooding the ranking with duplicates of the same spot.

The second step is the band assembly, where the peaks are separated into three broad zoom bands. The fine band at scales 1–2 shows raw source lines in a close window around the query center. The medium band at scales 4–16 tracks function and class signatures with some context around them. Finally, the coarse band at scales 32–128 compresses the whole radius into a section-level summary.

All of that processing is handled by the MCP server, and the model simply sees structured JSON with bands and peak positions without having to worry about any of the wavelet math.

What the Model Actually Gets

Let's say an agent calls query_wavelet_context which is centered on line 150 inside a 500 line TypeScript file that has some authentication logic. In this case, the fine band will have the actual lines of code being inspected. The medium band will provide a summary of lines 0–400, guided by peaks such as the imports at top and test helpers at bottom.

The model derives its knowledge of what updateUser does by paying attention to the fine band, but it also knows about authentication context from the coarse band. It's able to jump to related code by recognising class and function boundaries in the wavelet peaks without needing to see all 500 lines of text from the file.

There is also a utility called get_important_positions, which operates on the whole project. It goes through every source file, smooths out the wavelet peaks, and gives you a ranked list of the most important places in the code.

Beyond locating structural boundaries, the server can also measure how complex or irregular those structures are, using a pair of entropy analysis tools driven by a Haar discrete wavelet transform and bit-cost estimation I discussed in my last post. Ricker coefficients can be quantized at each scale using get_entropy_bands which computes their bit-plane counts with higher cost indicating more structural irregularity at that resolution. The original per-line structural signal can also be decomposed through multiple Haar levels using get_complexity_heatmap to project the entropy cost back onto a per-line irregularity score. The model can use these scores as a sort of texture channel to understand where gnarly parts of the code live. Any boilerplate regions will end up with a low score, and so they can be safely summarized or skipped, while high-entropy regions will likely contain dense logic or unusual patterns that warrant extra attention. These tools give the model a data-driven way to triage code at high level and works great for refactoring tasks where the model can easily find sections with tangled logic and break it up.

The key advantage of wavelets is that they focus on the overall structure of the code which is something that can only be inferred using tools like regex. It's worth mentioning that it's possible to get a lot of the same type of analysis using AST parsing which is even more exact. However, AST tools necessitate having a parser for each language while wavelets are completely agnostic of the semantics of the text they're applied to. They simply flag statistical regularities, and that's what makes them such a broadly applicable tool in the first place.

Wavescope’s approach strikes a happy medium between grepping and full blown AST analysis since the transform works on any 1D signal. Language awareness comes from a simple keyword-weighting layer rather than a full parser with each language just needing a 10-line config to describe its core semantics. And the whole thing is cheap to run, able to process an entire file in milliseconds to create hierarchical and multi-scale outputs. The scales happen to be a natural representation of code structure which provides the LLM with a map to navigate it.

Since the wavelet locates edge positions at all levels simultaneously, it trivially locates structural changes that even sophisticated parsers might struggle to detect. For example, a long sequence of if/else blocks looks structurally different from a class with many short methods while regions of documentation appear as valleys between peaks. The wavelet doesn't need to know what these things actually are to perceive that they are structurally different. And this often happens to be just what the model needs to figure out what to do.

By the Numbers

To illustrate the concept, I ran three realistic development tasks against WaveScope's own 14 file codebase containing just under five thousand lines of TypeScript to compare the token cost with the traditional way. "Traditional" here means what LLM coding agents actually do: grep for landmarks, read targeted chunks of code, and skim file headers to get a sample of what the structure looks like.

One common task is to understand the structure of a large file. So, I had the agent analyze index.ts weighing in at 854 lines to see where imports end, to find where the tool registrations cluster, and identify the startup code. These are your typical tasks where the approach would be to grep for export landmarks and section comments, then read the import block, find a registration example, and recognize the startup tail. That costs about 2,000 tokens producing a patchy picture which is just a heuristic. WaveScope's coarse band plus the top 15 important positions give the same structural overview using just 750 tokens with section-level boundaries that grep can't surface because it's only matching lines without the structural context around them.

Now, let's take a look at a different kind of task, which I've alluded to earlier, where we want to find tangled code that needs refactoring. A naive way would be to run wc -l to find the biggest files, then skim large chunks looking for deep nesting and high cyclomatic complexity. Such a task runs at about 5,200 tokens to read 600 lines across two files, and you would still miss tangled code in any files you didn't scan. On the other hand, WaveScope's get_complexity_heatmap flags exact per-line irregularity scores across every file you point it at. Running it on the three largest files, which are index.ts, wce.ts, and context.ts costs a mere 436 tokens. That's a whopping 92% reduction, but also surfaces precise hotspots at line 287 in handleDiffWaveletContext scoring 0.92, line 59 in FileContext with a score of 0.92, and so on. The analysis finds every section of interest across the whole codebase producing a ranked list of spots to look at closely.

Another example would be to identify which files are architecturally core versus peripheral. The traditional approach runs head -25 on every file to read imports, which ends up costing around 2,900 tokens for all 14 files. As a result, you learn what each file imports without actually knowing which ones define the project's architecture, and the model will have to spend more tokens digging deeper based on its best guesses. WaveScope's project-wide get_important_positions returns a structural-density ranking of every file in just 1,700 tokens along with meaningful rankings. Now it's clear that signal.ts tops the list with the highest keyword density per line. The heavyweight algorithmic files are wce.ts, context.ts, and index.ts which have lower average density because their structural features are spread across hundreds of lines of implementation. Other files, such as the haar.ts and file-cache.ts utilities, end up with a low ranking. Now it's clear which files need to be read first in order to understand the conceptual skeleton of the project.

Across all three tasks, WaveScope used significantly fewer tokens while producing meaningful answers that are structural rather than textual which is precisely what we were interested in. The structure provides the model with understanding of relationships within the code base that is simply not possible to do by doing regex based heuristics. On top of that, a 128K token window would burn 8% of its capacity using the traditional approach for these three tasks while WaveScope needs only 2%. And the gap scales proportionally with the size of the codebase, making the tool particularly effective for analyzing large projects.

It's also worth noting that processing all 14 files using signal extraction, Ricker CWT at 8 scales, peak detection, and band assembly all averages at just 3 milliseconds per file. And the entropy heatmap adds about 1ms per file on top. In the context of reasoning time for the model, this is effectively free to do.

Hopefully, the examples above make the value of using structural peak analysis clear already, but we can leverage the information that's already been produced even further by passing it through an entropy encoder to see just how complex those regions actually are. Running get_entropy_bands against index.ts gives a breakdown of the bit-cost estimates per wavelet scale, which is an indirect measure of structural irregularity. Here, it can be seen that the finest detail at scale costs 613 bits while the coarsest costs 322, suggesting that the file is structurally busy at the line level. A high density of handler functions and schema definitions found in a tool registration class is what's responsible for the pattern.

Next, we can call get_complexity_heatmap to take things further by back-projecting entropy onto individual lines via a Haar DWT. Doing that identifies the top irregularity hotspots in index.ts such as handleDiffWaveletContext with a score of 0.92 and handleGetCursorImportantPositions with a score of 0.90. These are non-trivial async handler functions with the most branching logic in them, and the heatmap is able to flag them without knowing anything about TypeScript! The entire heatmap costs 474 tokens as well, giving the model a data-driven triage list to focus on and to safely skip the boilerplate.

WaveScope is open source and can be found at https://github.com/yogthos/wavescope-mcp. Add it as a drop-in MCP server to give your agent a zoom lens for code.

Permalink

Clojure Deref (Jun 2, 2026)

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem (feed: RSS).

Upcoming Events

Podcasts, videos, and media

Blogs, articles, and news

Libraries and Tools

Debut release

  • repl-agent - An MCP server that gives AI agents direct access to a live Clojure nREPL session

  • continuity-auth - Respect-weighted rate limits for the open web

  • lambda-sketch - Probabilistic data structures and sketching algorithms in Clojure

  • pod-babashka-gozxing - A babashka pod for reading and writing QR codes, backed by the Go library gozxing (a port of ZXing).

  • edsger - A reMarkable handwritten almost-Clojure REPL

  • katzen - Generalized algebraic theories and categorical programming for Clojure.

Updates

  • tools.build 0.10.14 - Clojure builds as Clojure programs

  • clojure_cli 1.12.5.1654 - Clojure CLI

  • tools.deps 0.31.1629 - Deps as data and classpath generation

  • deps.clj 1.12.5.1654 - A faithful port of the clojure CLI bash script to Clojure

  • memento-redis 2.0.33 - Memento cache backed by Redis

  • deft 0.1.5 - A collection of macros designed to address issues with objects in Clojure.

  • quiescent 0.2.6 - A Clojure library for composable async tasks with automatic parallelization, structured concurrency, and parent-child and chain cancellation

  • teensyp 0.5.5 - A small, zero-dependency Clojure TCP server that uses Java NIO

  • apie 1.0.2 - Apie 🙈 OpenAPI Service Validator

  • fulcro-rad 1.6.24 - Fulcro Rapid Application Development

  • statecharts 1.4.0-RC18 - A Statechart library for CLJ(S)

  • guardrails 1.3.2 - Efficient, hassle-free function call validation with a concise inline syntax for clojure.spec and Malli

  • clojure-desktop-toolkit 0.7.0 - Create native state-of-the-art desktop applications in Clojure using Eclipse’s SWT graphics toolkit.

  • datalevin 0.10.18 - A simple, fast and versatile Datalog database

  • optimus 2026.05.27 - A Ring middleware for frontend performance optimization.

  • cli 1.30.130 - Opinionated command line argument handling, with excellent support for subcommands

  • clj-oa3 0.4.0 - Clojure client library for OpenADR 3 (Martian HTTP, entity coercion, Malli schemas)

  • svar 0.7.0 - Type‑safe LLM output for Clojure. Works with any text‑only model.

  • portal 0.65.0 - A clojure tool to navigate through your data.

  • partial-cps 0.1.55 - A lean and efficient continuation passing style transform, includes async-await support.

  • spindel 0.1.15 - Cross-platform FRP runtime with a git-like memory model.

  • cljs-ajax 0.9.0-beta1 - simple asynchronous Ajax client for ClojureScript and Clojure

  • deps-new 0.12.2 - Create new projects for the Clojure CLI / deps.edn

  • ghosttyfx 0.1.169.11 - Cljfx wrapper of GhosttyFX

  • pg2 0.1.49 - A fast PostgreSQL driver for Clojure

  • let-go 1.9.0 - Almost Clojure written in Go.

  • gsheetplus 1.1.3 - Low-level and high-level wrapper to work with Google Sheets. Reading, writing and sheet management.

  • phel-lang 0.41.0 - A functional, Lisp-inspired language that compiles to PHP. Inspired by Clojure, Phel brings macros, persistent data structures, and expressive functional idioms to the PHP ecosystem.

  • cryogen 0.7.9 - A simple static site generator written in Clojure

  • Selmer 1.13.4 - A fast, Django inspired template system in Clojure.

  • fulcro-tui 1.0.2 - A JLine-base TUI wrapper, allowing you to write TUI projects in CLJ or Babashka via Fulcro

  • plotje 0.3.1 - simple and easy plotting

  • squint 0.12.193 - Light-weight ClojureScript dialect

  • mokujin 1.0.0.120 - 🪵 Structured logging for Clojure. Thin layer on top of clojure.tools.logging with MDC support

  • edn.c 8c0869c - A fast, zero-copy EDN (Extensible Data Notation) reader written in C11 with SIMD acceleration.

  • nippy 3.7.0-RC2 - Fast serialization library for Clojure

  • tyrell 1.0.0-RC10 - Clojurescript WebComponents library

  • lacinia 1.3.0 - GraphQL implementation in pure Clojure

  • clojure-lint-action 9 - A GitHub Action that lints clojure files with clj-kondo and generates comments with reviewdog on pull requests to improve the code review experience.

  • dataspex 2026.06.1 - See the shape of your data: point-and-click Clojure(Script) data browser

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.