Clojure Developer at 3E nv

Permalink

Building AI agentes in practice with Clojure

During Clojure South, Marlon Silva, Senior Software Engineer at Nubank, shared his perspective on a recurring challenge for software engineers working with AI today: how to move beyond using AI assistants and start engineering reliable, task-oriented AI agents.

According to Marlon, the industry has made it increasingly easy to consume AI — APIs, copilots, and assistants are everywhere — but building AI-powered systems still requires engineers to make a series of low-level, often uncomfortable decisions. In his talk, he focused on demystifying those decisions, framing AI not as a black box, but as infrastructure, and integrations that need to be reasoned about explicitly.

Rather than presenting a new framework or abstraction layer, Marlon walked through the architectural choices he believes matter most when building agents in practice, and explained why Clojure offers a particularly strong foundation for exploring this space.

Infrastructure first: where your models live matters

Marlon started by arguing that any serious AI initiative begins with infrastructure — specifically, how teams access models. While this decision is often treated as an implementation detail, he emphasized that it directly impacts scalability, experimentation, security, and integration with existing systems.

From his perspective, engineers typically face two options:

• Direct AI vendors: Marlon noted that these providers are an excellent entry point for individual developers. Signing up is straightforward, APIs are well-documented, and it is possible to start experimenting almost immediately. For learning and early exploration, this path minimizes friction.
• Cloud providers: For organizations, however, Marlon argued that leveraging existing cloud relationships is usually the better long-term decision. Most companies already have accounts, billing, security controls, and observability in place. Cloud Providers like AWS and GCP make it possible to access models from multiple AI Labs without introducing new suppliers into the stack.

According to Marlon, when teams are operating inside an organization, the most pragmatic default is to use the models already available through their cloud providers. This removes procurement overhead and allows engineers to focus on building systems instead of managing vendors.

The fragmentation problem: too many APIs

Once access to models is established, Marlon pointed out a second, inevitable problem: API fragmentation. Each provider exposes different request formats, parameters, and SDKs, which quickly complicates development and makes experimentation costly.

In his talk, Marlon described this fragmentation as one of the first scaling pain points teams encounter when AI moves beyond a single script or proof of concept.

To address this, he introduced LiteLLM as a practical unification layer. LiteLLM is a proxy that standardizes access to multiple models and providers behind a single API, regardless of where the model is hosted.

Marlon highlighted three concrete benefits of this approach:

• A unified interface, allowing teams to switch models without rewriting integration code.
• Centralized observability, creating a single point for logging, debugging, and auditing model interactions.
• Cost control, which he emphasized as critical. Token usage scales quickly in production, and LiteLLM enables organizations to track and limit usage per team, service, or key.

From Marlon’s perspective, this kind of proxy is not an optimization: it becomes foundational infrastructure as soon as AI is part of a real system.

Why smaller models often work better for agents

Marlon then challenged a common assumption in the AI space: that larger models are always better.

For task-oriented AI agents, he argued, this is rarely true. Citing recent research from Nvidia, Marlon explained that Small Language Models (SLMs) are often a better fit for agents designed to execute specific, well-defined actions.

According to him, the broad generalization capabilities of large LLMs, models capable of writing essays or long-form prose, are unnecessary for most agent workloads. Using them in these contexts leads to wasted capacity, higher costs, and increased complexity.

He outlined several practical advantages of SLMs:

• Cost efficiency: models like Llama 4 Scout on AWS Bedrock cost orders of magnitude less per token than large proprietary models.
• Lower energy consumption: making them a more responsible choice at scale and more eco-friendly.
• Feasible fine-tuning: adapting a 7–10B parameter model to a specific domain is realistic, whereas doing the same with very large models often is not real for most companies.

For Marlon, choosing SLMs is not a compromise, as it is an engineering decision aligned with the actual requirements of agent-based systems.

Why Clojure fits this problem space

From there, Marlon shifted focus to tooling. He explained that AI development is inherently experimental: prompts change, parameters are adjusted, models are swapped, and assumptions are constantly tested. In that context, developer feedback loops matter. This is where Clojure stands out.

Marlon described Clojure as an ergonomic language — not because of syntax alone, but because of its REPL-driven development model. The ability to evaluate functions incrementally, inspect results immediately, and iterate without restarting an application fundamentally changes how engineers explore problem spaces.

In his experience, this interactive workflow aligns closely with how AI systems are built and refined.Beyond the REPL, Marlon highlighted two interoperability advantages:

• Java interoperability: Because Clojure runs on the JVM, it has seamless access to the Java ecosystem. Cloud SDKs, HTTP clients, observability tools, and mature libraries are immediately available.
• Python interoperability: With libraries such as libpython-clj, Clojure can import and execute Python code directly. While not as seamless as Java interop, this capability allows engineers to reuse Python-based AI tooling without abandoning Clojure’s interactive workflow.

For Marlon, this combination makes Clojure a strong orchestration layer for AI systems that need to integrate with multiple ecosystems.

From theory to practice: the live demonstration

To make these ideas concrete, Marlon walked through a live demonstration.

He started with simple Python scripts that sent text, images, and PDFs to Bedrock-hosted models via a local LiteLLM proxy. These examples established a baseline using familiar tooling.

Next, he reproduced the same workflows in Clojure by importing the Python LiteLLM package directly into the Clojure runtime. Python functions were called from Clojure code, with inputs and outputs handled interactively.

According to Marlon, the most important part of the demo was not that this approach works, but how it changes the development experience. Python-based workflows often require frequent context switching — editing files, running scripts, and restarting processes. With Clojure and the REPL, the entire feedback loop stays inside the editor.

For exploratory domains like AI, Marlon argued, this difference directly translates into faster iteration and deeper focus.

Conclusion

Marlon closed by emphasizing that AI agents remain a young and rapidly evolving field. Libraries, architectures, and best practices are still in flux, which makes flexibility a key requirement.

He also offered a note of caution: granting models excessive autonomy without clear boundaries and controls can lead to fragile systems. In his view, frameworks that favour obscure control flow and mostly relies on LLM itself encourages a “ship and pray” approach to AI development. At the same time, Marlon pointed out that new agent architectures are actively emerging from research groups at organizations like Nvidia and DeepMind, signaling that significant changes are still ahead.

His conclusion was pragmatic: combining well-chosen infrastructure, models sized to the problem, and tools that favor exploration creates a solid foundation for building AI systems grounded in engineering discipline. The repository shared during the talk serves as a starting point for engineers interested in continuing that exploration.

References

Small Language Models are the Future of Agentic AI

AlphaEvolve: A coding agent for scientific and algorithmic discovery

GitHub – marlonjsilva/clj-agents

GitHub – clj-python/libpython-clj

The post Building AI agentes in practice with Clojure appeared first on Building Nubank.

Permalink

I am sorry, but everyone is getting syntax highlighting wrong

Translations: Russian

Syntax highlighting is a tool. It can help you read code faster. Find things quicker. Orient yourself in a large file.

Like any tool, it can be used correctly or incorrectly. Let’s see how to use syntax highlighting to help you work.

Christmas Lights Diarrhea

Most color themes have a unique bright color for literally everything: one for variables, another for language keywords, constants, punctuation, functions, classes, calls, comments, etc.

Sometimes it gets so bad one can’t see the base text color: everything is highlighted. What’s the base text color here?

The problem with that is, if everything is highlighted, nothing stands out. Your eye adapts and considers it a new norm: everything is bright and shiny, and instead of getting separated, it all blends together.

Here’s a quick test. Try to find the function definition here:

and here:

See what I mean?

So yeah, unfortunately, you can’t just highlight everything. You have to make decisions: what is more important, what is less. What should stand out, what shouldn’t.

Highlighting everything is like assigning “top priority” to every task in Linear. It only works if most of the tasks have lesser priorities.

If everything is highlighted, nothing is highlighted.

Enough colors to remember

There are two main use-cases you want your color theme to address:

1. Look at something and tell what it is by its color (you can tell by reading text, yes, but why do you need syntax highlighting then?)
2. Search for something. You want to know what to look for (which color).

1 is a direct index lookup: color → type of thing.

2 is a reverse lookup: type of thing → color.

Truth is, most people don’t do these lookups at all. They might think they do, but in reality, they don’t.

Let me illustrate. Before:

After:

Can you see it? I misspelled return for retunr and its color switched from red to purple.

I can’t.

Here’s another test. Close your eyes (not yet! Finish this sentence first) and try to remember what color your color theme uses for class names?

Can you?

If the answer for both questions is “no”, then your color theme is not functional. It might give you comfort (as in—I feel safe. If it’s highlighted, it’s probably code) but you can’t use it as a tool. It doesn’t help you.

What’s the solution? Have an absolute minimum of colors. So little that they all fit in your head at once. For example, my color theme, Alabaster, only uses four:

• Green for strings
• Purple for constants
• Yellow for comments
• Light blue for top-level definitions

That’s it! And I was able to type it all from memory, too. This minimalism allows me to actually do lookups: if I’m looking for a string, I know it will be green. If I’m looking at something yellow, I know it’s a comment.

Limit the number of different colors to what you can remember.

If you swap green and purple in my editor, it’ll be a catastrophe. If somebody swapped colors in yours, would you even notice?

What should you highlight?

Something there isn’t a lot of. Remember—we want highlights to stand out. That’s why I don’t highlight variables or function calls—they are everywhere, your code is probably 75% variable names and function calls.

I do highlight constants (numbers, strings). These are usually used more sparingly and often are reference points—a lot of logic paths start from constants.

Top-level definitions are another good idea. They give you an idea of a structure quickly.

Punctuation: it helps to separate names from syntax a little bit, and you care about names first, especially when quickly scanning code.

Please, please don’t highlight language keywords. class, function, if, elsestuff like this. You rarely look for them: “where’s that if” is a valid question, but you will be looking not at the if the keyword, but at the condition after it. The condition is the important, distinguishing part. The keyword is not.

Highlight names and constants. Grey out punctuation. Don’t highlight language keywords.

Comments are important

The tradition of using grey for comments comes from the times when people were paid by line. If you have something like

of course you would want to grey it out! This is bullshit text that doesn’t add anything and was written to be ignored.

But for good comments, the situation is opposite. Good comments ADD to the code. They explain something that couldn’t be expressed directly. They are important.

So here’s another controversial idea:

Comments should be highlighted, not hidden away.

Use bold colors, draw attention to them. Don’t shy away. If somebody took the time to tell you something, then you want to read it.

Two types of comments

Another secret nobody is talking about is that there are two types of comments:

1. Explanations
2. Disabled code

Most languages don’t distinguish between those, so there’s not much you can do syntax-wise. Sometimes there’s a convention (e.g. -- vs /* */ in SQL), then use it!

Here’s a real example from Clojure codebase that makes perfect use of two types of comments:

Disabled code is gray, explanation is bright yellow

Light or dark?

Per statistics, 70% of developers prefer dark themes. Being in the other 30%, that question always puzzled me. Why?

And I think I have an answer. Here’s a typical dark theme:

and here’s a light one:

On the latter one, colors are way less vibrant. Here, I picked them out for you:

Notice how many colors there are. No one can remember that many.

This is because dark colors are in general less distinguishable and more muddy. Look at Hue scale as we move brightness down:

Basically, in the dark part of the spectrum, you just get fewer colors to play with. There’s no “dark yellow” or good-looking “dark teal”.

Nothing can be done here. There are no magic colors hiding somewhere that have both good contrast on a white background and look good at the same time. By choosing a light theme, you are dooming yourself to a very limited, bad-looking, barely distinguishable set of dark colors.

So it makes sense. Dark themes do look better. Or rather: light ones can’t look good. Science ¯\_(ツ)_/¯

But!

But.

There is one trick you can do, that I don’t see a lot of. Use background colors! Compare:

The first one has nice colors, but the contrast is too low: letters become hard to read.

The second one has good contrast, but you can barely see colors.

The last one has both: high contrast and clean, vibrant colors. Lighter colors are readable even on a white background since they fill a lot more area. Text is the same brightness as in the second example, yet it gives the impression of clearer color. It’s all upside, really.

UI designers know about this trick for a while, but I rarely see it applied in code editors:

If your editor supports choosing background color, give it a try. It might open light themes for you.

Bold and italics

Don’t use. This goes into the same category as too many colors. It’s just another way to highlight something, and you don’t need too many, because you can’t highlight everything.

In theory, you might try to replace colors with typography. Would that work? I don’t know. I haven’t seen any examples.

Using italics and bold instead of colors

Myth of number-based perfection

Some themes pay too much attention to be scientifically uniform. Like, all colors have the same exact lightness, and hues are distributed evenly on a circle.

This could be nice (to know if you have OCD), but in practice, it doesn’t work as well as it sounds:

OkLab l=0.7473 c=0.1253 h=0, 45, 90, 135, 180, 225, 270, 315

The idea of highlighting is to make things stand out. If you make all colors the same lightness and chroma, they will look very similar to each other, and it’ll be hard to tell them apart.

Our eyes are way more sensitive to differences in lightness than in color, and we should use it, not try to negate it.

Let’s design a color theme together

Let’s apply these principles step by step and see where it leads us. We start with the theme from the start of this post:

First, let’s remove highlighting from language keywords and re-introduce base text color:

Next, we remove color from variable usage:

and from function/method invocation:

The thinking is that your code is mostly references to variables and method invocation. If we highlight those, we’ll have to highlight more than 75% of your code.

Notice that we’ve kept variable declarations. These are not as ubiquitous and help you quickly answer a common question: where does thing thing come from?

Next, let’s tone down punctuation:

I prefer to dim it a little bit because it helps names stand out more. Names alone can give you the general idea of what’s going on, and the exact configuration of brackets is rarely equally important.

But you might roll with base color punctuation, too:

Okay, getting close. Let’s highlight comments:

We don’t use red here because you usually need it for squiggly lines and errors.

This is still one color too many, so I unify numbers and strings to both use green:

Finally, let’s rotate colors a bit. We want to respect nesting logic, so function declarations should be brighter (yellow) than variable declarations (blue).

Compare with what we started:

In my opinion, we got a much more workable color theme: it’s easier on the eyes and helps you find stuff faster.

Shameless plug time

I’ve been applying these principles for about 8 years now.

I call this theme Alabaster and I’ve built it a couple of times for the editors I used:

• VS Code
• JetBrains IDEs
• Sublime Text (twice)

It’s also been ported to many other editors and terminals; the most complete list is probably here. If your editor is not on the list, try searching for it by name—it might be built-in already! I always wondered where these color themes come from, and now I became an author of one (and I still don’t know).

Feel free to use Alabaster as is or build your own theme using the principles outlined in the article—either is fine by me.

As for the principles themselves, they worked out fantastically for me. I’ve never wanted to go back, and just one look at any “traditional” color theme gives me a scare now.

I suspect that the only reason we don’t see more restrained color themes is that people never really thought about it. Well, this is your wake-up call. I hope this will inspire people to use color more deliberately and to change the default way we build and use color themes.

Permalink

Statistics made simple

I have a weird relationship with statistics: on one hand, I try not to look at it too often. Maybe once or twice a year. It’s because analytics is not actionable: what difference does it make if a thousand people saw my article or ten thousand?

I mean, sure, you might try to guess people’s tastes and only write about what’s popular, but that will destroy your soul pretty quickly.

On the other hand, I feel nervous when something is not accounted for, recorded, or saved for future reference. I might not need it now, but what if ten years later I change my mind?

Seeing your readers also helps to know you are not writing into the void. So I really don’t need much, something very basic: the number of readers per day/per article, maybe, would be enough.

Final piece of the puzzle: I self-host my web projects, and I use an old-fashioned web server instead of delegating that task to Nginx.

Static sites are popular and for a good reason: they are fast, lightweight, and fulfil their function. I, on the other hand, might have an unfinished gestalt or two: I want to feel the full power of the computer when serving my web pages, to be able to do fun stuff that is beyond static pages. I need that freedom that comes with a full programming language at your disposal. I want to program my own web server (in Clojure, sorry everybody else).

Existing options

All this led me on a quest for a statistics solution that would uniquely fit my needs. Google Analytics was out: bloated, not privacy-friendly, terrible UX, Google is evil, etc.

What is going on?

Some other JS solution might’ve been possible, but still questionable: SaaS? Paid? Will they be around in 10 years? Self-host? Are their cookies GDPR-compliant? How to count RSS feeds?

Nginx has access logs, so I tried server-side statistics that feed off those (namely, Goatcounter). Easy to set up, but then I needed to create domains for them, manage accounts, monitor the process, and it wasn’t even performant enough on my server/request volume!

My solution

So I ended up building my own. You are welcome to join, if your constraints are similar to mine. This is how it looks:

It’s pretty basic, but does a few things that were important to me.

Setup

Extremely easy to set up. And I mean it as a feature.

Just add our middleware to your Ring stack and get everything automatically: collecting and reporting.

(def app
  (-> routes
    ...
    (ring.middleware.params/wrap-params)
    (ring.middleware.cookies/wrap-cookies)
    ...
    (clj-simple-stats.core/wrap-stats))) ;; <-- just add this

It’s zero setup in the best sense: nothing to configure, nothing to monitor, minimal dependency. It starts to work immediately and doesn’t ask anything from you, ever.

See, you already have your web server, why not reuse all the setup you did for it anyway?

Request types

We distinguish between request types. In my case, I am only interested in live people, so I count them separately from RSS feed requests, favicon requests, redirects, wrong URLs, and bots. Bots are particularly active these days. Gotta get that AI training data from somewhere.

RSS feeds are live people in a sense, so extra work was done to count them properly. Same reader requesting feed.xml 100 times in a day will only count as one request.

Hosted RSS readers often report user count in User-Agent, like this:

Feedly/1.0 (+http://www.feedly.com/fetcher.html; 457 subscribers; like FeedFetcher-Google)

Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 6 subscribers)

Feedbin feed-id:1373711 - 142 subscribers

My personal respect and thank you to everybody on this list. I see you.

Graphs

Visualization is important, and so is choosing the correct graph type. This is wrong:

Continuous line suggests interpolation. It reads like between 1 visit at 5am and 11 visits at 6am there were points with 2, 3, 5, 9 visits in between. Maybe 5.5 visits even! That is not the case.

This is how a semantically correct version of that graph should look:

Some attention was also paid to having reasonable labels on axes. You won’t see something like 117, 234, 10875. We always choose round numbers appropriate to the scale: 100, 200, 500, 1K etc.

Goes without saying that all graphs have the same vertical scale and syncrhonized horizontal scroll.

Insights

We don’t offer much (as I don’t need much), but you can narrow reports down by page, query, referrer, user agent, and any date slice.

Not implemented (yet)

It would be nice to have some insights into “What was this spike caused by?”

Some basic breakdown by country would be nice. I do have IP addresses (for what they are worth), but I need a way to package GeoIP into some reasonable size (under 1 Mb, preferably; some loss of resolution is okay).

Finally, one thing I am really interested in is “Who wrote about me?” I do have referrers, only question is how to separate signal from noise.

Performance. DuckDB is a sport: it compresses data and runs column queries, so storing extra columns per row doesn’t affect query performance. Still, each dashboard hit is a query across the entire database, which at this moment (~3 years of data) sits around 600 MiB. I definitely need to look into building some pre-calculated aggregates.

One day.

How to get

Head to github.com/tonsky/clj-simple-stats and follow the instructions:

Let me know what you think! Is it usable to you? What could be improved?

Permalink

FFI with GraalVM Native Image: The Real Work of Maintaining a Library That Crosses Language Boundaries

Exposing a library via FFI seems simple on paper.

You compile your code to .so, export some functions with extern "C", write bindings in the target language. Done, interoperability achieved.

Except that's not how it works in practice. At least not when your lib is a database written in Clojure, compiled with GraalVM native-image, that needs to manage Git and Lucene internally, and will be called from Rust in environments ranging from dev laptops to CI containers with 8MB of stack.

Permalink

Episode 12: It allows a small team to achieve really great goals, with Marcin Maicki, Dentons

Episode 12 of "Clojure in product. Would you do it again?" is live — Marcin Maicki, Global Data Developer & Lead Developer at Dentons, joins Artem Barmin and Vadym Kostiuk to talk about running Clojure inside a large, decentralized enterprise.

Highlights:

- Marcin’s Clojure origin story: started with ClojureScript, moved into Clojure, and found the functional mindset a natural fit coming from React.

- How Clojure landed at Dentons: a conscious choice for a focused referral‑network venture that valued expressiveness and small teams.

- Practical stack and ops: Postgres, Elasticsearch, Reagent/Re‑frame + Material UI, Metabase; Marcin also works on PySpark/Databricks in his global data role.

- Maintenance and risk: why they’re migrating away from old, unmaintained libs; regular security scans and external testing make dependency health a real concern.

- Team, onboarding, and hiring: a small Clojure pod (Marcin + one dev, testers, DevOps); knowledge sharing, docs, and close pairing are the onboarding tools — hiring remains the main practical blocker.

- Enterprise realities: polycentric org structure, integration friction with firm standards (Power BI, Azure), and the tradeoffs that make Clojure a strong fit in some contexts but a harder sell in others.

Watch Episode 12 to hear the full conversation and the nuances of keeping a nine‑year Clojure codebase healthy in a corporate setting.

Permalink

SQLite in Production? Not So Fast for Complex Queries

There is a growing movement to use SQLite for everything. Kent C. Dodds argues for defaulting to SQLite in web development due to its zero-latency reads and minimal operational burden. Wesley Aptekar-Cassels makes a strong case that SQLite works for web apps with large user bases, provided they don't need tens of thousands of writes per second. Discussions on Hacker News and elsewhere cite companies like Apple, Adobe, and Dropbox using SQLite in production. Even the official SQLite documentation encourages its use for most websites with fewer than 100K hits per day.

These points are fair. The overarching theme is a pushback against automatically choosing complex, client-server databases like PostgreSQL when SQLite is often more than sufficient, simpler to manage, and faster for the majority of use cases. I agree with that framing. The debate has settled into a well-understood set of tradeoffs:

These are the terms of the current discussion. But there is an important, often overlooked dimension missing from this framing.

SQLite struggles with complex queries. More specifically, SQLite is not well-suited to handle the kind of multi-join queries that arise naturally in any serious production system. This goes beyond the usual talking points about deployment concerns (write concurrency, distribution, and so on). It points to a system-level limitation: the query optimizer itself. That limitation matters even for read-heavy, single-node deployments, which is exactly the use case where SQLite is supposed to shine.

I have benchmark evidence showing this clearly. This post focuses on join-heavy analytical queries, not on the many workloads where SQLite is already the right choice. But first, let me explain why this matters more than people think.

Multi-join queries are not exotic

A common reaction to discussing multi-join queries is: "I don't write queries with 10 joins." This usually means one of three things: the schema is denormalized, the logic has been moved into application code, or the product is simple. None of these mean the problem goes away.

In any system with many entity types, rich relationships, history or versioning, permissions, and compositional business rules, multi-join queries inevitably appear. They emerge whenever data is normalized and questions are compositional. Here are concrete examples from real production systems.

Enterprise SaaS (CRM / ERP / HR). A query like "show me all open enterprise deals" in a Salesforce-like system touches accounts, contacts, products, pricebooks, territories, users, permissions, and activity logs. Real queries in these systems routinely involve 10-20 joins. Every dimension of the business (customers, ownership, products, pricing, regions, access control, activity statistics) is often normalized into its own table.

Healthcare (EHR). "Patients with condition X, treated by doctors in department Y, prescribed drug Z in the last 6 months, and whose insurance covers that drug" spans patients, visits, diagnoses, providers, departments, prescriptions, drugs, insurance plans, coverage rules, and claims. Exceeding 15 joins is common.

E-commerce and Marketplaces. "Orders in the last 30 days that include products from vendor V, shipped late, refunded, with customers in region R" touches orders, order items, products, vendors, shipments, delivery events, refunds, customers, addresses, regions, and payment methods. Again, 10+ joins.

Authorization and Permission systems. "Which documents can user U see?" requires traversing users, groups, roles, role assignments, resource policies, ACLs, inheritance rules, and organizational hierarchies. This alone can be 12+ joins, sometimes recursive.

Analytics and BI. Star schemas look simple on paper, but real dashboard queries add slowly changing dimensions, hierarchy tables, permission joins, and attribution models. A "simple" dashboard query often hits 6-10 dimension tables plus access control.

Knowledge graphs and semantic systems. "Papers authored by people affiliated with institutions collaborating with company X on topic Y" requires joining papers, authors, affiliations, institutions, collaborations, and topics. Very common in search and recommendation systems.

Event sourcing and temporal queries. Reconstructing the state of an account at a point in time with approval chains requires joining entity tables, event tables, approval tables, history tables, and version joins. Temporal dimensions multiply join counts quickly.

AI / ML feature pipelines. Feature stores generate massive joins. Assembling a feature vector often requires joining user profiles, sessions, events, devices, locations, and historical aggregates. This is why feature stores are expensive.

The pattern is consistent across domains:

Complex joins are not accidental. They emerge from normalized data, explicit relationships, compositional business rules, layered authorization, and historical records. Again, if you don't see many joins in your system, it usually means the schema is denormalized, the logic is in the application layer, or the product hasn't reached sufficient complexity yet. This does not mean the system is better. It often means complexity has been pushed into the application layer, which can add engineering cost without adding real value.

The evidence: JOB benchmark

The Join Order Benchmark (JOB) is a standard benchmark designed specifically to stress database query optimizers on complex multi-join queries [1]. Based on the Internet Movie Database (IMDb), a real-world, highly normalized dataset with over 36 million rows in its largest table, it contains 113 analytical queries with 3 to 16 joins each, averaging 8 joins per query. Unlike synthetic benchmarks like TPC, JOB uses real data with realistic data distributions, making it a much harder test of query optimization.

I ran this benchmark comparing three databases: SQLite (via JDBC), PostgreSQL 18, and Datalevin (an open-source database I build). All were tested in default configurations with no tuning, on a MacBook Pro M3 Pro with 36GB RAM. This is not a tuning shootout, but a look at out-of-the-box optimizer behavior. Details of the benchmark methodology can be found here.

Overall wall clock time

SQLite needed a 60-second timeout per query, and 9 queries failed to complete within that limit. The actual total time for SQLite would be substantially higher if these were included. For example, query 10c, when allowed to run to completion, took 446.5 seconds.

Execution time statistics (milliseconds)

The median tells the story: SQLite's median is nearly 3x worse than the other two.

Per-query speedup: Datalevin vs. SQLite

The chart at the top of this post shows the speedup ratio (SQLite time / Datalevin time) for each of the queries on a logarithmic scale (excluding 9 timeouts). Points above the 1x line (10^0) mean Datalevin is faster; points below mean SQLite is faster. The horizontal lines mark 1x, 10x, and 100x speedups.

Several patterns stand out:

• The vast majority of points are above the 1x line, often by 10x or more.
• For the hardest queries, Datalevin achieves 100x+ speedups. These are precisely the complex multi-join queries where SQLite's optimizer breaks down.
• SQLite is rarely faster, and when it is, the margin is small.
• The 9 timed-out queries (not shown) would push the ratio even higher.

Where SQLite breaks down

Timeouts. Queries 8c, 8d, 10c, 15c, 15d, 23a, 23b, 23c, and 28c all timed out at the 60-second limit during the benchmark runs. These represent queries with higher join counts where SQLite's optimizer failed to find an efficient plan.

Extreme slowdowns. Even among queries that completed, SQLite was often dramatically slower. Query 9d took 37.8 seconds on SQLite versus 1.6 seconds on Datalevin (24x). Query 19d took 20.8 seconds versus 5.7 seconds. Query families 9, 10, 12, 18, 19, 22, and 30 all show SQLite performing significantly worse, often by 10-50x.

Why SQLite falls behind

SQLite's query optimizer has fundamental limitations for complex joins:

1. Limited join order search. SQLite uses exhaustive search for join ordering only up to a limited number of tables. Beyond that threshold, it falls back to heuristics that produce poor plans for complex queries.

2. Weak statistics model. SQLite's cardinality estimation is simpler than PostgreSQL's, which itself has well-documented weaknesses [1]. With fewer statistics to guide optimization, SQLite makes worse choices about which tables to join first and which access methods to use.

3. No cost-based plan selection for complex cases. For queries with many tables, SQLite's planner cannot explore enough of the plan space to find good join orderings. The result is plans that process orders of magnitude more intermediate rows than necessary.

These limitations are architectural; they are not bugs likely to be fixed in a near-term release. They reflect design tradeoffs inherent in SQLite's goal of being a lightweight, embedded database.

What this means for "SQLite in production"

SQLite is excellent for what it was designed to be: an embedded database for applications with simple query patterns. It excels as a local data store, a file format, and a cache. For read-heavy workloads with straightforward queries touching a few tables, it works extremely well.

But the production systems described above, e.g. CRM, EHR, e-commerce, authorization, analytics, are precisely where SQLite's query optimizer becomes a bottleneck. These are not hypothetical workloads, but the day-to-day reality of systems that serve businesses and users.

The "SQLite in production" advocates often benchmark simple cases: key-value lookups, single-table scans, basic CRUD operations. On those workloads, SQLite does extremely well. But production systems grow. Schemas become more normalized as data integrity requirements increase. Questions become more compositional as business logic matures. And at that point, the query optimizer becomes the bottleneck, not the network round trip to a database server.

Before choosing SQLite for a production system, ask: will our queries stay simple forever? If the answer is no, and it usually is, the savings in deployment simplicity may not be worth the cost in query performance as the system grows.

An alternative approach

In a previous post, I described how Datalevin, a triplestore using Datalog, handles these complex queries effectively. Its query optimizer uses counting and sampling on its triple indices to produce accurate cardinality estimates, resulting in better execution plans. Unlike row stores, where cardinality estimation is notoriously difficult due to bundled storage, a triplestore can count and sample individual data atoms directly.

This approach yields plans that are not only better than SQLite's, but consistently better than PostgreSQL's across the full range of JOB queries. Despite Datalevin being written in Clojure on the JVM rather than optimized C code, it still managed to halve the total query time in the JOB benchmark. The quality of the optimizer's decisions matters more than the raw execution speed of the engine.

For systems that need both deployment simplicity (Datalevin works as an embedded database too) and the ability to handle complex queries as they inevitably arise, a triplestore with a cost-based optimizer offers a practical alternative to either SQLite or a full client-server RDBMS. It is not a silver bullet, but it can deliver SQLite-like operational simplicity without giving up complex-query performance.

If you have different results or have tuned SQLite to handle these queries well, I would love to compare notes. The goal here is not to dunk on SQLite, but to surface a missing dimension in a discussion that often defaults to deployment tradeoffs alone.

References

[1] Leis, V., et al. "How good are query optimizers, really?" VLDB Endowment. 2015.

Permalink

Gaiwan: Hire Us!

Hire Us!

We are actively pursuing new work with clients old and new. We&aposre in the first place interested in Clojure/ClojureScript gigs, but happy to chat about any potential leads even if you yourself are not hiring.

Things we can do for you:

• Long- or short-term team augmentation
• Programming new features
• Modernizing old codebases
• Clojure / Clojurescript coaching
• Make your team more productive by improving dev tooling
• Consulting on how to manage YOUR open source and community
• Feature work on OUR open source (all lambdaisland/Gaiwan libraries)
• Architecture review
• Cleaning up vibe-coded code bases that have become unmaintainable

Mitesh Shah profile photo with wild hair and a blue suit.

Our colleague Mitesh Shah is open to new work. He&aposs a talented engineer, practical and enterpreneurial with a lot of experience at startups. He knows how to ship, he&aposs a great communicator, and above all he&aposs a great person to work with. He&aposs truly full stack, from UI/UX to the database and back.

Bettina Shzu-Juraschek, who covers legal, tax, HR, and operations at Gaiwan, can create financial overviews and close money leaks for small businesses.

2026 Conferences Preview

This year we&aposre looking forward to attending the Babashka conference on May 8, 2026, and Dutch Clojure Days on May 9, 2026, in Amsterdam. It&aposs always great to get together with the Clojure community in real life! We&aposre booking a sailboat in Weesp with accommodations for up to 16 people; if you want to book a berth either in your own room or in a shared room, sign up here.

Bettina will be at FOSS Backstage and FOSS Backstage Design in Berlin from March 16-18, 2026, to push forward our open source projects. Reach out if you want to connect there.

#tea-break

At Gaiwan we share interesting reads in our #tea-break channel. Here&aposs a selection:

Why there’s no European Google? "Can we just stop equating success with short-term economic growth? What if we used usefulness and longevity? What if we gave more value to the fundamental technological infrastructure instead of the shiny new marketing gimmick used to empty naive wallets?"

The Office according to "The Office." by Ribbonfarm. An oldie but goodie. Are you a sociopath, a loser, or clueless?

The Bitter Lesson by Rich Sutton. In the long run, general-purpose methods that leverage massive computation—specifically search and learning—consistently outperform specialized systems built on human domain expertise.

Yayyay events shares the financial statements from their events. "Organizing Lambda World this year came with a price tag of €52,000,...with a €4,000 deficit." If we had included the costs of our salaries in the budget for the Heart of Clojure conference we organized in 2024, we would have also been 5 figures in the red. 🙁

What are you reading these days?

Permalink

Just what IS Python, anyway?

A mental model for understanding Python’s role

Anyone who started programming in the early 1980s might have started with Apple II BASIC, BBC BASIC or Sinclair BASIC (ZX-81 or ZX Spectrum) - and 6502 or Z80 assembler.

Those early environments were all defined by immediacy. You typed something in; the machine did something. There was no ambiguity about what the language was for.

Since then a professional programmer might have ventured through FORTRAN, C, C++, UNIX shell, Visual Basic, VBA, VB.NET, C#, F#, JavaScript and more recently Rust, Zig, Nim and Odin. Every one of those fits elegantly into a mental slot: Systems language, Application language, Functional language, Runtime language or Tooling language.

Python, oddly, doesn’t. It can be tricky, for an experienced programmer, to grasp what it was and how it related to the other languages or which slot to put it in. Often, they will conclude "I don't like Python" and express confusion at its vast popularity - and even primacy - in the 2020s.

A slippery language

Traditionally we’re taught to classify languages along a few axes:

• compiled vs interpreted
• scripting vs “real” languages
• imperative vs OO vs functional

Python fits poorly into all of them.

It isn't a compiled language in the C or Rust sense: it doesn't result in a standalone executable. But it isn't purely interpreted either, since it must be processed before execution. It supports imperative, object-oriented and functional styles but isn’t optimized for any of them. It began as a scripting language, but today it’s used to build large, long-running systems.

So just what IS Python?

Python is not a binary-producing language

The turning point is to realize that Python is not defined by the artefact it produces.

C, C++, Rust, Zig and Fortran produce binaries that can be directly run. The output is the thing. Once compiled, the language more or less disappears.

Python doesn’t work like that.

Python source code is compiled to bytecode, and that bytecode is executed by a virtual machine. The VM, the object model, the garbage collector and the standard library are not incidental. They are Python. A Python program needs this ecosystem to run; it can't run standalone, unless they are all bundled in with it.

In structural terms, Python sits alongside languages with runtimes and ecosystems:

• .NET (C#, F#, VB.NET)
• the JVM (Java, Scala, Kotlin, Clojure)

In all three cases, the runtime is the unit of execution, not the compiled artefact.

“Interpreted vs compiled” is therefore a false dichotomy. CPython parses Python source-code to an Abstract Syntax Tree (AST), compiles it to bytecode and then executes that bytecode on a VM. That’s not conceptually different from Java or .NET — just simpler and often slower, while the runtime startup overhead persists.

Python’s real role: orchestration

The solution to the puzzle of "What IS Python?" is to realize that Python is a runtime-centric language, whereupon its real role becomes obvious.

Python is not primarily about doing work. It’s about controlling work. It's exceptional good and for quickly lashing stuff together: like Lego, Meccano or snap-on tooling than traditional software construction.

The most important Python libraries — NumPy, SciPy, Pandas, PyTorch, TensorFlow — are not written in Python in any meaningful sense. Python provides the API, the glue and the control flow. The heavy lifting happens in underlying libraries written in C, C++, Fortran or CUDA - anything that can expose a C ABI (Application Binary Interface).

Python performs the same role over its libraries as:

• SQL over databases
• shell over Unix
• VBA over Office

It is an orchestration language sitting above high-performance systems. That’s why it thrives in scientific computing, data pipelines and machine learning. It lets you build rapidly and easily, with simply syntax, whilst the underlying libraries deliver the performance. So long as orchestration overhead is low, Pythobn-based systems can scale surprisingly far.

Why Python still feels slippery

Even with this framing, Python can still feel oddly unsatisfying if you come from strongly structured languages.

Compared with .NET or the JVM, Python has:

• weak static guarantees
• loose module boundaries
• a simpler, leakier object model

If you’re used to the discipline of C#, F# or Rust, Python can feel vague. Things work — until they don’t — and the language often declines to help you reason about correctness ahead of time.

It turns out that being able to throw things together quickly, in easy-to-understand code, and ecosystem breadth are far more important for mass adoption than type-safety, compilation, raw-performance or architectural rigidity. Make something easy, and more people will do it, more often.

Python's winning formula is to lower the barrier-to-entry for proof-of-concept and prototype stage projects - much like Visual BASIC and VBA did in the 1990s - and can even get to MVP (Minimum Viable Product). You can always make it faster, later, by translating critical paths into a compiled language.

Getting something working, at all and quickly, turns out to be hugely more important than getting it working fast or elegantly - something shell scripting showed us as far back as the 1970s.

Clearing up potential misunderstandings

Common misconceptions are worth addressing:

• “Python is slow”
  Python orchestrates underlying code. In most applications, performance-critical paths live in native libraries. Only in a small number of domains — such as ultra-low-latency systems — does Python itself become the limiting factor.

• “Python is a scripting language”
  Historically true, as that's how it originated, but it has evolved vastly since then.

• “Python is interpreted”
  Better to say it is pre-compiled to bytecode that is then executed by a virtual machine.

A better language classification

A proper taxonomy therefore looks like this:

1. Standalone native languages
   C, C++, Rust, Zig, Fortran
   → the binary is the product

2. Runtime ecosystems
   Python, JVM languages, .NET languages
   → the runtime is the product

3. Host-bound scripting languages
   Bash, PowerShell, VBA
   → the host environment is the product

Python belongs firmly in the second group.

A brief note for Rust and Go proponents

A common challenge from Rust or Go developers is that Python’s role is better served by “doing it properly” in a compiled language from the start.

That view makes sense — if your problem is well-specified, stable, performance-critical, and worth committing to upfront architectural constraints. In those cases, Rust or Go are often excellent choices - although these languages are more specialized than, say, C#, F# or JavaScript.

But many real-world problems do not start that way. They begin as ill-defined, exploratory or evolving systems: data pipelines, research code, internal tools, integration glue. A research-team needs to test an idea quickly in a small-scale way, rather than performantly on terabytes of data. A business-development team needs to solve a problem quickly and tactically, because the business needs a solution "yesterday". Some problems move to fast to wait for a strategic solution, or the cost of a strategic solution cannot yet be justified as too much is unknown. In those contexts, early commitment to strict typing, memory models or concurrency primitives can slow learning rather than accelerate it.

Python’s advantage is not that it replaces C#, Java, Rust or Go. It is that it defers commitment. You can explore the problem space quickly, validate assumptions, and only later decide which parts deserve the cost of rewriting in a compiled language. Your Proof-of-Concept or Prototype written in Python becomes your teacher and learning exercise.

In practice, Python and Rust and Go are not competitors but complements: Python for orchestration and discovery; Rust or Go for stabilised, performance-critical components where very specific issues need to be solved. Rust eliminates entire classes of bug to do with memory-management and threading; Go is superb at server-side services. These are not everyday programming needs.

Summary

Python isn’t confused, incoherent or a "toy" language. It simply departs from the mental models of earlier generations of languages and fulfills a unique role that no other language can quite match.

Python is not any of compiled, interpreted or “just a scripting language”. It's a runtime-centric orchestration layer and a complete ecosystem of its own. It's the Visual Basic and VBA of the internet era: ideal for rapid assembly, experimentation and leverage rather than purity or control.

And that makes it incredibly useful - and wildly popular.

Permalink

JAVA REALITY !!

▶ What Java Is ?

• A verbose, boilerplate-heavy, object-oriented language.
• Created during the dot-com bubble by Sun Microsystems.
• Originally meant for TV remotes, released publicly in 1996.
• Runs on billions of devices for decades.

▶ Java Philosophy

• Motto: "Write once, run everywhere"
• Reality: "Write once, debug everywhere"
• Known for long logs, heavy configuration, and legacy systems.

▶ Ecosystem & Influence

• Inspired JVM languages: Groovy, Clojure, Scala, Kotlin.
• Also indirectly inspired JavaScript.
• Failed attempt at the web: Java Applets (thankfully dead).

▶ Enterprise Reality

• Often paired with Oracle Database.
• Companies spend years talking about migrating away.
• Expensive licenses, long-term enterprise lock-in.

▶ Getting Started Pain

• Install JDK, JRE, JVM.
• Prepare for massive error logs.
• Memorize: public static void main(String[] args)

▶ Java Coding Style

• Forced Object-Oriented Programming.
• Requires classes even for Hello World.
• Much more boilerplate than languages like Python.
• Encourages deep inheritance and complex class hierarchies.

▶ Typical Java Project Lifecycle

1. Write one giant class.
2. Boss complains.
3. Split into deeply nested subclasses.
4. Code becomes impossible to refactor.
5. Developer questions life choices.

▶ Final Truth

• Java is widely hated yet widely used.
• Two types of languages: 1) Those people complain about 2) Those nobody uses
• Love it or hate it, Java gets real work done.

Permalink

Volumetric Clouds with Clojure and LWJGL

Procedural generation of volumetric clouds using different types of noise

(Cross posting article published at Clojure Civitas)

Dependencies

To download the required libraries, we use a deps.edn file with the following content:

{:deps
 {
  org.clojure/clojure {:mvn/version "1.12.3"}
  org.scicloj/noj     {:mvn/version "2-beta18"}
  midje/midje         {:mvn/version "1.10.10"}
  generateme/fastmath {:mvn/version "3.0.0-alpha4"}
  comb/comb           {:mvn/version "1.0.0"}
  }
}

We are going to import the following methods and namespaces:

(require '[clojure.math :refer (PI sqrt cos sin tan to-radians pow floor)]
         '[midje.sweet :refer (fact facts tabular => roughly)]
         '[fastmath.vector :refer (vec2 vec3 add mult sub div mag dot normalize)]
         '[fastmath.matrix :refer (mat->float-array mulm
                                   rotation-matrix-3d-x rotation-matrix-3d-y)]
         '[tech.v3.datatype :as dtype]
         '[tech.v3.tensor :as tensor]
         '[tech.v3.datatype.functional :as dfn]
         '[tablecloth.api :as tc]
         '[scicloj.tableplot.v1.plotly :as plotly]
         '[tech.v3.libs.buffered-image :as bufimg]
         '[comb.template :as template])
(import '[org.lwjgl.opengl GL11]
        '[org.lwjgl BufferUtils]
        '[org.lwjgl.glfw GLFW]
        '[org.lwjgl.opengl GL GL11 GL12 GL13 GL15 GL20 GL30 GL32 GL42])

Worley noise

Worley noise is a type of structured noise which is defined for each pixel using the distance to the nearest seed point.

Noise parameters

First we define a function to create parameters of the noise.

• size is the size of each dimension of the noise array
• divisions is the number of subdividing cells in each dimension
• dimensions is the number of dimensions

(defn make-noise-params
  [size divisions dimensions]
  {:size size :divisions divisions :cellsize (/ size divisions) :dimensions dimensions})

Here is a corresponding Midje test. Note that ideally you practise Test Driven Development (TDD), i.e. you start with writing one failing test. Because this is a Clojure notebook, the unit tests are displayed after the implementation.

(fact "Noise parameter initialisation"
      (make-noise-params 256 8 2) => {:size 256 :divisions 8 :cellsize 32 :dimensions 2})

2D and 3D vectors

Next we need a function which allows us to create 2D or 3D vectors depending on the number of input parameters.

(defn vec-n
  ([x y] (vec2 x y))
  ([x y z] (vec3 x y z)))

(facts "Generic vector function for creating 2D and 3D vectors"
       (vec-n 2 3) => (vec2 2 3)
       (vec-n 2 3 1) => (vec3 2 3 1))

Random points

The following method generates a random point in a cell specified by the cell indices.

(defn random-point-in-cell
  [{:keys [cellsize]} & indices]
  (let [random-seq (repeatedly #(rand cellsize))
        dimensions (count indices)]
    (add (mult (apply vec-n (reverse indices)) cellsize)
         (apply vec-n (take dimensions random-seq)))))

We test the method by replacing the random function with a deterministic function.

(facts "Place random point in a cell"
       (with-redefs [rand (fn [s] (* 0.5 s))]
         (random-point-in-cell {:cellsize 1} 0 0) => (vec2 0.5 0.5)
         (random-point-in-cell {:cellsize 2} 0 0) => (vec2 1.0 1.0)
         (random-point-in-cell {:cellsize 2} 0 3) => (vec2 7.0 1.0)
         (random-point-in-cell {:cellsize 2} 2 0) => (vec2 1.0 5.0)
         (random-point-in-cell {:cellsize 2} 2 3 5) => (vec3 11.0 7.0 5.0)))

We can now use the random-point method to generate a grid of random points. The grid is represented using a tensor from the dtype-next library.

(defn random-points
  [{:keys [divisions dimensions] :as params}]
  (tensor/clone
    (tensor/compute-tensor (repeat dimensions divisions)
                           (partial random-point-in-cell params))))

(facts "Greate grid of random points"
       (let [params-2d (make-noise-params 32 8 2)
             params-3d (make-noise-params 32 8 3)]
         (with-redefs [rand (fn [s] (* 0.5 s))]
           (dtype/shape (random-points params-2d)) => [8 8]
           ((random-points params-2d) 0 0) => (vec2 2.0 2.0)
           ((random-points params-2d) 0 3) => (vec2 14.0 2.0)
           ((random-points params-2d) 2 0) => (vec2 2.0 10.0)
           (dtype/shape (random-points params-3d)) => [8 8 8]
           ((random-points params-3d) 2 3 5) => (vec3 22.0 14.0 10.0))))

Here is a scatter plot showing one random point placed in each cell.

(let [points  (tensor/reshape (random-points (make-noise-params 256 8 2)) [(* 8 8)])
      scatter (tc/dataset {:x (map first points) :y (map second points)})]
  (-> scatter
      (plotly/base {:=title "Random points"})
      (plotly/layer-point {:=x :x :=y :y})))

Modular distance

In order to get a periodic noise array, we need to component-wise wrap around distance vectors.

(defn mod-vec
  [{:keys [size]} v]
  (let [size2 (/ size 2)
        wrap  (fn [x] (-> x (+ size2) (mod size) (- size2)))]
    (apply vec-n (map wrap v))))

(facts "Wrap around components of vector to be within -size/2..size/2"
       (mod-vec {:size 8} (vec2 2 3)) => (vec2 2 3)
       (mod-vec {:size 8} (vec2 5 2)) => (vec2 -3 2)
       (mod-vec {:size 8} (vec2 2 5)) => (vec2 2 -3)
       (mod-vec {:size 8} (vec2 -5 2)) => (vec2 3 2)
       (mod-vec {:size 8} (vec2 2 -5)) => (vec2 2 3)
       (mod-vec {:size 8} (vec3 2 3 1)) => (vec3 2 3 1)
       (mod-vec {:size 8} (vec3 5 2 1)) => (vec3 -3 2 1)
       (mod-vec {:size 8} (vec3 2 5 1)) => (vec3 2 -3 1)
       (mod-vec {:size 8} (vec3 2 3 5)) => (vec3 2 3 -3)
       (mod-vec {:size 8} (vec3 -5 2 1)) => (vec3 3 2 1)
       (mod-vec {:size 8} (vec3 2 -5 1)) => (vec3 2 3 1)
       (mod-vec {:size 8} (vec3 2 3 -5)) => (vec3 2 3 3))

Using the mod-dist function we can calculate the distance between two points in the periodic noise array.

(defn mod-dist
  [params a b]
  (mag (mod-vec params (sub b a))))

The tabular macro implemented by Midje is useful for running parametrized tests.

(tabular "Wrapped distance of two points"
         (fact (mod-dist {:size 8} (vec2 ?ax ?ay) (vec2 ?bx ?by)) => ?result)
         ?ax ?ay ?bx ?by ?result
         0   0   0   0   0.0
         0   0   2   0   2.0
         0   0   5   0   3.0
         0   0   0   2   2.0
         0   0   0   5   3.0
         2   0   0   0   2.0
         5   0   0   0   3.0
         0   2   0   0   2.0
         0   5   0   0   3.0)

Modular lookup

We also need to lookup elements with wrap around. We recursively use tensor/select and then finally the tensor as a function to lookup along each axis.

(defn wrap-get
  [t & args]
  (if (> (count (dtype/shape t)) (count args))
    (apply tensor/select t (map mod args (dtype/shape t)))
    (apply t (map mod args (dtype/shape t)))))

A tensor with index vectors is used to test the lookup.

(facts "Wrapped lookup of tensor values"
       (let [t (tensor/compute-tensor [4 6] vec2)]
         (wrap-get t 2 3) => (vec2 2 3)
         (wrap-get t 2 7) => (vec2 2 1)
         (wrap-get t 5 3) => (vec2 1 3)
         (wrap-get (wrap-get t 5) 3) => (vec2 1 3)))

The following function converts a noise coordinate to the index of a cell in the random point array.

(defn division-index
  [{:keys [cellsize]} x]
  (int (floor (/ x cellsize))))

(facts "Convert coordinate to division index"
       (division-index {:cellsize 4} 3.5)  => 0
       (division-index {:cellsize 4} 7.5)  => 1
       (division-index {:cellsize 4} -0.5) => -1)

Getting indices of Neighbours

The following function determines the neighbouring indices of a cell recursing over each dimension.

(defn neighbours
  [& args]
  (if (seq args)
    (mapcat (fn [v] (map (fn [delta] (into [(+ (first args) delta)] v)) [-1 0 1]))
            (apply neighbours (rest args)) )
    [[]]))

(facts "Get neighbouring indices"
       (neighbours) => [[]]
       (neighbours 0) => [[-1] [0] [1]]
       (neighbours 3) => [[2] [3] [4]]
       (neighbours 1 10) => [[0 9] [1 9] [2 9] [0 10] [1 10] [2 10] [0 11] [1 11] [2 11]])

Sampling Worley noise

Using above functions one can now implement Worley noise. For each pixel the distance to the closest seed point is calculated. This is achieved by determining the distance to each random point in all neighbouring cells and then taking the minimum.

(defn worley-noise
  [{:keys [size dimensions] :as params}]
  (let [random-points (random-points params)]
    (tensor/clone
      (tensor/compute-tensor
        (repeat dimensions size)
        (fn [& coords]
            (let [center   (map #(+ % 0.5) coords)
                  division (map (partial division-index params) center)]
              (apply min
                     (for [neighbour (apply neighbours division)]
                          (mod-dist params (apply vec-n (reverse center))
                                    (apply wrap-get random-points neighbour))))))
        :double))))

Here a 256 × 256 Worley noise tensor is created.

(def worley (worley-noise (make-noise-params 256 8 2)))

The values are inverted and normalised to be between 0 and 255.

(def worley-norm
  (dfn/* (/ 255 (- (dfn/reduce-max worley) (dfn/reduce-min worley)))
         (dfn/- (dfn/reduce-max worley) worley)))

Finally one can display the noise.

(bufimg/tensor->image worley-norm)

Perlin noise

Perlin noise is generated by choosing a random gradient vector at each cell corner. The noise tensor’s intermediate values are interpolated with a continuous function, utilizing the gradient at the corner points.

Random gradients

The 2D or 3D gradients are generated by creating a vector where each component is set to a random number between -1 and 1. Random vectors are generated until the vector length is greater 0 and lower or equal to 1. The vector then is normalized and returned. Random vectors outside the unit circle or sphere are discarded in order to achieve a uniform distribution on the surface of the unit circle or sphere.

(defn random-gradient
  [& args]
  (loop [args args]
        (let [random-vector (apply vec-n (map (fn [_x] (- (rand 2.0) 1.0)) args))
              vector-length (mag random-vector)]
          (if (and (> vector-length 0.0) (<= vector-length 1.0))
            (div random-vector vector-length)
            (recur args)))))

The function below serves as a Midje checker for a vector with an approximate expected value.

(defn roughly-vec
  [expected error]
  (fn [actual]
      (<= (mag (sub actual expected)) error)))

In the following tests, the random function is again replaced with a deterministic function.

(facts "Create unit vector with random direction"
       (with-redefs [rand (constantly 0.5)]
         (random-gradient 0 0)
         => (roughly-vec (vec2 (- (sqrt 0.5)) (- (sqrt 0.5))) 1e-6))
       (with-redefs [rand (constantly 1.5)]
         (random-gradient 0 0)
         => (roughly-vec (vec2 (sqrt 0.5) (sqrt 0.5)) 1e-6)))

The random gradient function is then used to generate a field of random gradients.

(defn random-gradients
 [{:keys [divisions dimensions]}]
 (tensor/clone (tensor/compute-tensor (repeat dimensions divisions) random-gradient)))

The function is verified to correctly generate 2D and 3D random gradient fields.

(facts "Random gradients"
       (with-redefs [rand (constantly 1.5)]
         (dtype/shape (random-gradients {:divisions 8 :dimensions 2}))
         => [8 8]
         ((random-gradients {:divisions 8 :dimensions 2}) 0 0)
         => (roughly-vec (vec2 (sqrt 0.5) (sqrt 0.5)) 1e-6)
         (dtype/shape (random-gradients {:divisions 8 :dimensions 3})) => [8 8 8]
         ((random-gradients {:divisions 8 :dimensions 3}) 0 0 0)
         => (vec3 (/ 1 (sqrt 3)) (/ 1 (sqrt 3)) (/ 1 (sqrt 3)))))

The gradient field can be plotted with Plotly as a scatter plot of disconnected lines.

(let [gradients (tensor/reshape (random-gradients (make-noise-params 256 8 2))
                                [(* 8 8)])
      points    (tensor/reshape (tensor/compute-tensor [8 8] (fn [y x] (vec2 x y)))
                                [(* 8 8)])
      scatter   (tc/dataset {:x (mapcat (fn [point gradient]
                                            [(point 0)
                                             (+ (point 0) (* 0.5 (gradient 0)))
                                             nil])
                                        points gradients)
                             :y (mapcat (fn [point gradient]
                                            [(point 1)
                                             (+ (point 1) (* 0.5 (gradient 1)))
                                             nil])
                                        points gradients)})]
  (-> scatter
      (plotly/base {:=title "Random gradients" :=mode "lines"})
      (plotly/layer-point {:=x :x :=y :y})))

Corner vectors

The next step is to determine the vectors to the corners of the cell for a given point. First we define a function to determine the fractional part of a number.

(defn frac
  [x]
  (- x (Math/floor x)))

(facts "Fractional part of floating point number"
       (frac 0.25) => 0.25
       (frac 1.75) => 0.75
       (frac -0.25) => 0.75)

This function can be used to determine the relative position of a point in a cell.

(defn cell-pos
  [{:keys [cellsize]} point]
  (apply vec-n (map frac (div point cellsize))))

(facts "Relative position of point in a cell"
       (cell-pos {:cellsize 4} (vec2 2 3)) => (vec2 0.5 0.75)
       (cell-pos {:cellsize 4} (vec2 7 5)) => (vec2 0.75 0.25)
       (cell-pos {:cellsize 4} (vec3 7 5 2)) => (vec3 0.75 0.25 0.5))

A 2 × 2 tensor of corner vectors can be computed by subtracting the corner coordinates from the point coordinates.

(defn corner-vectors
  [{:keys [dimensions] :as params} point]
  (let [cell-pos (cell-pos params point)]
    (tensor/compute-tensor
      (repeat dimensions 2)
      (fn [& args] (sub cell-pos (apply vec-n (reverse args)))))))

(facts "Compute relative vectors from cell corners to point in cell"
       (let [corners2 (corner-vectors {:cellsize 4 :dimensions 2} (vec2 7 6))
             corners3 (corner-vectors {:cellsize 4 :dimensions 3} (vec3 7 6 5))]
         (corners2 0 0) => (vec2 0.75 0.5)
         (corners2 0 1) => (vec2 -0.25 0.5)
         (corners2 1 0) => (vec2 0.75 -0.5)
         (corners2 1 1) => (vec2 -0.25 -0.5)
         (corners3 0 0 0) => (vec3 0.75 0.5 0.25)))

Extract gradients of cell corners

The function below retrieves the gradient values at a cell’s corners, utilizing wrap-get for modular access. The result is a 2 × 2 tensor of gradient vectors.

(defn corner-gradients
  [{:keys [dimensions] :as params} gradients point]
  (let [division (map (partial division-index params) point)]
    (tensor/compute-tensor
      (repeat dimensions 2)
      (fn [& coords] (apply wrap-get gradients (map + (reverse division) coords))))))

(facts "Get 2x2 tensor of gradients from a larger tensor using wrap around"
       (let [gradients2 (tensor/compute-tensor [4 6] (fn [y x] (vec2 x y)))
             gradients3 (tensor/compute-tensor [4 6 8] (fn [z y x] (vec3 x y z))) ]
         ((corner-gradients {:cellsize 4 :dimensions 2} gradients2 (vec2 9 6)) 0 0)
         => (vec2 2 1)
         ((corner-gradients {:cellsize 4 :dimensions 2} gradients2 (vec2 9 6)) 0 1)
         => (vec2 3 1)
         ((corner-gradients {:cellsize 4 :dimensions 2} gradients2 (vec2 9 6)) 1 0)
         => (vec2 2 2)
         ((corner-gradients {:cellsize 4 :dimensions 2} gradients2 (vec2 9 6)) 1 1)
         => (vec2 3 2)
         ((corner-gradients {:cellsize 4 :dimensions 2} gradients2 (vec2 23 15)) 1 1)
         => (vec2 0 0)
         ((corner-gradients {:cellsize 4 :dimensions 3} gradients3 (vec3 9 6 3)) 0 0 0)
         => (vec3 2 1 0)))

Influence values

The influence value is the function value of the function with the selected random gradient at a corner.

(defn influence-values
  [gradients vectors]
  (tensor/compute-tensor
    (repeat (count (dtype/shape gradients)) 2)
    (fn [& args] (dot (apply gradients args) (apply vectors args)))
    :double))

(facts "Compute influence values from corner vectors and gradients"
       (let [gradients2 (tensor/compute-tensor [2 2] (fn [_y x] (vec2 x 10)))
             vectors2   (tensor/compute-tensor [2 2] (fn [y _x] (vec2 1 y)))
             influence2 (influence-values gradients2 vectors2)
             gradients3 (tensor/compute-tensor [2 2 2] (fn [z y x] (vec3 x y z)))
             vectors3   (tensor/compute-tensor [2 2 2] (fn [_z _y _x] (vec3 1 10 100)))
             influence3 (influence-values gradients3 vectors3)]
         (influence2 0 0) => 0.0
         (influence2 0 1) => 1.0
         (influence2 1 0) => 10.0
         (influence2 1 1) => 11.0
         (influence3 1 1 1) => 111.0))

Interpolating the influence values

For interpolation the following “ease curve” is used.

(defn ease-curve
  [t]
  (-> t (* 6.0) (- 15.0) (* t) (+ 10.0) (* t t t)))

(facts "Monotonously increasing function with zero derivative at zero and one"
       (ease-curve 0.0) => 0.0
       (ease-curve 0.25) => (roughly 0.103516 1e-6)
       (ease-curve 0.5) => 0.5
       (ease-curve 0.75) => (roughly 0.896484 1e-6)
       (ease-curve 1.0) => 1.0)

The ease curve monotonously increases in the interval from zero to one.

(-> (tc/dataset {:t (range 0.0 1.025 0.025)
                 :ease (map ease-curve (range 0.0 1.025 0.025))})
    (plotly/base {:=title "Ease Curve"})
    (plotly/layer-line {:=x :t :=y :ease}))

The interpolation weights are recursively calculated from the ease curve and the coordinate distances of the point to upper and lower cell boundary.

(defn interpolation-weights
  ([params point]
   (interpolation-weights (cell-pos params point)))
  ([pos]
   (if (seq pos)
     (let [w1   (- 1.0 (last pos))
           w2   (last pos)
           elem (interpolation-weights (butlast pos))]
       (tensor/->tensor [(dfn/* (ease-curve w1) elem) (dfn/* (ease-curve w2) elem)]))
     1.0)))

(facts "Interpolation weights"
       (let [weights2 (interpolation-weights {:cellsize 8} (vec2 2 7))
             weights3 (interpolation-weights {:cellsize 8} (vec3 2 7 3))]
         (weights2 0 0) => (roughly 0.014391 1e-6)
         (weights2 0 1) => (roughly 0.001662 1e-6)
         (weights2 1 0) => (roughly 0.882094 1e-6)
         (weights2 1 1) => (roughly 0.101854 1e-6)
         (weights3 0 0 0) => (roughly 0.010430 1e-6)))

Sampling Perlin noise

A Perlin noise sample is computed by

• Getting the random gradients for the cell corners.
• Getting the corner vectors for the cell corners.
• Computing the influence values which have the desired gradients.
• Determining the interpolation weights.
• Computing the weighted sum of the influence values.

(defn perlin-sample
  [params gradients point]
  (let [gradients (corner-gradients params gradients point)
        vectors   (corner-vectors params point)
        influence (influence-values gradients vectors)
        weights   (interpolation-weights params point)]
    (dfn/reduce-+ (dfn/* weights influence))))

Now one can sample the Perlin noise by performing above computation for the center of each pixel.

(defn perlin-noise
  [{:keys [size dimensions] :as params}]
  (let [gradients (random-gradients params)]
    (tensor/clone
      (tensor/compute-tensor
        (repeat dimensions size)
        (fn [& args]
            (let [center (apply vec-n (map #(+ % 0.5) (reverse args)))]
              (perlin-sample params gradients center)))
        :double))))

Here a 256 × 256 Perlin noise tensor is created.

(def perlin (perlin-noise (make-noise-params 256 8 2)))

The values are normalised to be between 0 and 255.

(def perlin-norm
  (dfn/* (/ 255 (- (dfn/reduce-max perlin) (dfn/reduce-min perlin)))
         (dfn/- perlin (dfn/reduce-min perlin))))

Finally one can display the noise.

(bufimg/tensor->image perlin-norm)

Mixing noise values

Combination of Worley and Perlin noise

You can blend Worley and Perlin noise by performing a linear combination of both.

(def perlin-worley-norm (dfn/+ (dfn/* 0.3 perlin-norm) (dfn/* 0.7 worley-norm)))

Here for example is the average of Perlin and Worley noise.

(bufimg/tensor->image (dfn/+ (dfn/* 0.5 perlin-norm) (dfn/* 0.5 worley-norm)))

Interpolation

One can linearly interpolate tensor values by recursing over the dimensions as follows.

(defn interpolate
  [tensor & args]
  (if (seq args)
    (let [x  (first args)
          xc (- x 0.5)
          xf (frac xc)
          x0 (int (Math/floor xc))]
      (+ (* (- 1.0 xf) (apply interpolate (wrap-get tensor      x0 ) (rest args)))
         (*        xf  (apply interpolate (wrap-get tensor (inc x0)) (rest args)))))
    tensor))

Here x-, y-, and z-ramps are used to test that interpolation works.

(facts "Interpolate values of tensor"
       (let [x2 (tensor/compute-tensor [4 6] (fn [_y x] x))
             y2 (tensor/compute-tensor [4 6] (fn [y _x] y))
             x3 (tensor/compute-tensor [4 6 8] (fn [_z _y x] x))
             y3 (tensor/compute-tensor [4 6 8] (fn [_z y _x] y))
             z3 (tensor/compute-tensor [4 6 8] (fn [z _y _x] z))]
         (interpolate x2 2.5 3.5) => 3.0
         (interpolate y2 2.5 3.5) => 2.0
         (interpolate x2 2.5 4.0) => 3.5
         (interpolate y2 3.0 3.5) => 2.5
         (interpolate x2 0.0 0.0) => 2.5
         (interpolate y2 0.0 0.0) => 1.5
         (interpolate x3 2.5 3.5 5.5) => 5.0
         (interpolate y3 2.5 3.5 3.0) => 3.0
         (interpolate z3 2.5 3.5 5.5) => 2.0))

Octaves of noise

Fractal Brownian Motion is implemented by computing a weighted sum of the same base noise function using different frequencies.

(defn fractal-brownian-motion
  [base octaves & args]
  (let [scales (take (count octaves) (iterate #(* 2 %) 1))]
    (reduce + 0.0
            (map (fn [amplitude scale] (* amplitude (apply base (map #(* scale %) args))))
                 octaves scales))))

Here the Fractal Brownian Motion is tested using an alternating 1D function and later a 2D checkboard function.

(facts "Fractal Brownian motion"
       (let [base1 (fn [x] (if (>= (mod x 2.0) 1.0) 1.0 0.0))
             base2 (fn [y x] (if (= (Math/round (mod y 2.0)) (Math/round (mod x 2.0)))
                               0.0 1.0))]
         (fractal-brownian-motion base2 [1.0] 0 0) => 0.0
         (fractal-brownian-motion base2 [1.0] 0 1) => 1.0
         (fractal-brownian-motion base2 [1.0] 1 0) => 1.0
         (fractal-brownian-motion base2 [1.0] 1 1) => 0.0
         (fractal-brownian-motion base2 [0.5] 0 1) => 0.5
         (fractal-brownian-motion base2 [] 0 1) => 0.0
         (fractal-brownian-motion base2 [0.0 1.0] 0 0) => 0.0
         (fractal-brownian-motion base2 [0.0 1.0] 0.0 0.5) => 1.0
         (fractal-brownian-motion base2 [0.0 1.0] 0.5 0.0) => 1.0
         (fractal-brownian-motion base2 [0.0 1.0] 0.5 0.5) => 0.0
         (fractal-brownian-motion base1 [1.0] 0) => 0.0
         (fractal-brownian-motion base1 [1.0] 1) => 1.0
         (fractal-brownian-motion base1 [0.0 1.0] 0.0) => 0.0
         (fractal-brownian-motion base1 [0.0 1.0] 0.5) => 1.0))

Remapping and clamping

The remap function is used to map a range of values of an input tensor to a different range.

(defn remap
  [value low1 high1 low2 high2]
  (dfn/+ low2 (dfn/* (dfn/- value low1) (/ (- high2 low2) (- high1 low1)))))

(tabular "Remap values of tensor"
       (fact ((remap (tensor/->tensor [?value]) ?low1 ?high1 ?low2 ?high2) 0)
             => ?expected)
       ?value ?low1 ?high1 ?low2 ?high2 ?expected
       0      0     1      0     1      0
       1      0     1      0     1      1
       0      0     1      2     3      2
       1      0     1      2     3      3
       2      2     3      0     1      0
       3      2     3      0     1      1
       1      0     2      0     4      2)

The clamp function is used to element-wise clamp values to a range.

(defn clamp
  [value low high]
  (dfn/max low (dfn/min value high)))

(tabular "Clamp values of tensor"
       (fact ((clamp (tensor/->tensor [?value]) ?low ?high) 0) => ?expected)
       ?value ?low ?high ?expected
       2      2    3      2
       3      2    3      3
       0      2    3      2
       4      2    3      3)

Generating octaves of noise

The octaves function is used to create a series of decreasing weights and normalize them so that they add up to 1.

(defn octaves
  [n decay]
  (let [series (take n (iterate #(* % decay) 1.0))
        sum    (apply + series)]
    (mapv #(/ % sum) series)))

Here is an example of noise weights decreasing by 50% at each octave.

(octaves 4 0.5)
; [0.5333333333333333
;  0.26666666666666666
;  0.13333333333333333
;  0.06666666666666667]

Now a noise array can be generated using octaves of noise.

(defn noise-octaves
  [tensor octaves low high]
  (tensor/clone
    (clamp
      (remap
        (tensor/compute-tensor (dtype/shape tensor)
                               (fn [& args]
                                   (apply fractal-brownian-motion
                                     (partial interpolate tensor)
                                     octaves
                                     (map #(+ % 0.5) args)))
                               :double)
        low high 0 255)
      0 255)))

2D examples

Here is an example of 4 octaves of Worley noise.

(bufimg/tensor->image (noise-octaves worley-norm (octaves 4 0.6) 120 230))

Here is an example of 4 octaves of Perlin noise.

(bufimg/tensor->image (noise-octaves perlin-norm (octaves 4 0.6) 120 230))

Here is an example of 4 octaves of mixed Perlin and Worley noise.

(bufimg/tensor->image (noise-octaves perlin-worley-norm (octaves 4 0.6) 120 230))

OpenGL rendering

OpenGL initialization

In order to render the clouds we create a window and an OpenGL context. Note that we need to create an invisible window to get an OpenGL context, even though we are not going to draw to the window

(GLFW/glfwInit)

(def window-width 640)
(def window-height 480)

(GLFW/glfwDefaultWindowHints)
(GLFW/glfwWindowHint GLFW/GLFW_VISIBLE GLFW/GLFW_FALSE)
(def window (GLFW/glfwCreateWindow window-width window-height "Invisible Window" 0 0))

(GLFW/glfwMakeContextCurrent window)
(GL/createCapabilities)

Compiling and linking shader programs

The following method is used to compile a shader.

(defn make-shader [source shader-type]
  (let [shader (GL20/glCreateShader shader-type)]
    (GL20/glShaderSource shader source)
    (GL20/glCompileShader shader)
    (when (zero? (GL20/glGetShaderi shader GL20/GL_COMPILE_STATUS))
      (throw (Exception. (GL20/glGetShaderInfoLog shader 1024))))
    shader))

The different shaders are then linked to become a program using the following method.

(defn make-program [& shaders]
  (let [program (GL20/glCreateProgram)]
    (doseq [shader shaders]
           (GL20/glAttachShader program shader)
           (GL20/glDeleteShader shader))
    (GL20/glLinkProgram program)
    (when (zero? (GL20/glGetProgrami program GL20/GL_LINK_STATUS))
      (throw (Exception. (GL20/glGetProgramInfoLog program 1024))))
    program))

This method is used to perform both compilation and linking of vertex shaders and fragment shaders.

(defn make-program-with-shaders
  [vertex-sources fragment-sources]
  (let [vertex-shaders   (map #(make-shader % GL20/GL_VERTEX_SHADER) vertex-sources)
        fragment-shaders (map #(make-shader % GL20/GL_FRAGMENT_SHADER) fragment-sources)
        program          (apply make-program (concat vertex-shaders fragment-shaders))]
    program))

In order to pass data to LWJGL methods, we need to be able to convert arrays to Java buffer objects.

(defmacro def-make-buffer [method create-buffer]
  `(defn ~method [data#]
     (let [buffer# (~create-buffer (count data#))]
       (.put buffer# data#)
       (.flip buffer#)
       buffer#)))

Setup of vertex data

Above macro is used to define methods for creating float, int, and byte buffer objects.

(def-make-buffer make-float-buffer BufferUtils/createFloatBuffer)
(def-make-buffer make-int-buffer BufferUtils/createIntBuffer)
(def-make-buffer make-byte-buffer BufferUtils/createByteBuffer)

We implement a method to create a vertex array object (VAO) with a vertex buffer object (VBO) and an index buffer object (IBO).

(defn setup-vao [vertices indices]
  (let [vao (GL30/glGenVertexArrays)
        vbo (GL15/glGenBuffers)
        ibo (GL15/glGenBuffers)]
    (GL30/glBindVertexArray vao)
    (GL15/glBindBuffer GL15/GL_ARRAY_BUFFER vbo)
    (GL15/glBufferData GL15/GL_ARRAY_BUFFER (make-float-buffer vertices)
                       GL15/GL_STATIC_DRAW)
    (GL15/glBindBuffer GL15/GL_ELEMENT_ARRAY_BUFFER ibo)
    (GL15/glBufferData GL15/GL_ELEMENT_ARRAY_BUFFER (make-int-buffer indices)
                       GL15/GL_STATIC_DRAW)
    {:vao vao :vbo vbo :ibo ibo}))

We also define the corresponding destructor for the vertex data.

(defn teardown-vao [{:keys [vao vbo ibo]}]
  (GL15/glBindBuffer GL15/GL_ELEMENT_ARRAY_BUFFER 0)
  (GL15/glDeleteBuffers ibo)
  (GL15/glBindBuffer GL15/GL_ARRAY_BUFFER 0)
  (GL15/glDeleteBuffers vbo)
  (GL30/glBindVertexArray 0)
  (GL15/glDeleteBuffers vao))

Offscreen rendering to a texture

The following method is used to create an empty 2D RGBA floating point texture

(defn make-texture-2d
  [width height]
  (let [texture (GL11/glGenTextures)]
    (GL11/glBindTexture GL11/GL_TEXTURE_2D texture)
    (GL11/glTexParameteri GL12/GL_TEXTURE_2D GL11/GL_TEXTURE_MIN_FILTER GL11/GL_LINEAR)
    (GL11/glTexParameteri GL12/GL_TEXTURE_2D GL11/GL_TEXTURE_MAG_FILTER GL11/GL_LINEAR)
    (GL11/glTexParameteri GL12/GL_TEXTURE_2D GL11/GL_TEXTURE_WRAP_S GL11/GL_REPEAT)
    (GL11/glTexParameteri GL12/GL_TEXTURE_2D GL11/GL_TEXTURE_WRAP_T GL11/GL_REPEAT)
    (GL42/glTexStorage2D GL11/GL_TEXTURE_2D 1 GL30/GL_RGBA32F width height)
    texture))

We define a method to convert a Java buffer object to a floating point array.

(defn float-buffer->array
  "Convert float buffer to float array"
  [buffer]
  (let [result (float-array (.limit buffer))]
    (.get buffer result)
    (.flip buffer)
    result))

The following method copies texture data into a Java buffer and then converts it to a floating point array.

(defn read-texture-2d
  [texture width height]
  (let [buffer (BufferUtils/createFloatBuffer (* height width 4))]
    (GL11/glBindTexture GL11/GL_TEXTURE_2D texture)
    (GL11/glGetTexImage GL11/GL_TEXTURE_2D 0 GL12/GL_RGBA GL11/GL_FLOAT buffer)
    (float-buffer->array buffer)))

This method sets up rendering using a specified texture as a framebuffer and then executes the body.

(defmacro framebuffer-render
  [texture width height & body]
  `(let [fbo# (GL30/glGenFramebuffers)]
     (try
       (GL30/glBindFramebuffer GL30/GL_FRAMEBUFFER fbo#)
       (GL11/glBindTexture GL11/GL_TEXTURE_2D ~texture)
       (GL32/glFramebufferTexture GL30/GL_FRAMEBUFFER GL30/GL_COLOR_ATTACHMENT0
                                  ~texture 0)
       (GL20/glDrawBuffers (make-int-buffer
                             (int-array [GL30/GL_COLOR_ATTACHMENT0])))
       (GL11/glViewport 0 0 ~width ~height)
       ~@body
       (finally
         (GL30/glBindFramebuffer GL30/GL_FRAMEBUFFER 0)
         (GL30/glDeleteFramebuffers fbo#)))))

We also create a method to set up the layout of the vertex buffer. Our vertex data is only going to contain 3D coordinates of points.

(defn setup-point-attribute
  [program]
  (let [point-attribute (GL20/glGetAttribLocation program "point")]
    (GL20/glVertexAttribPointer point-attribute 3 GL11/GL_FLOAT false
                                (* 3 Float/BYTES) (* 0 Float/BYTES))
    (GL20/glEnableVertexAttribArray point-attribute)))

We are going to use a simple background quad to perform volumetric rendering.

(defn setup-quad-vao
  []
  (let [vertices (float-array [ 1.0  1.0 0.0,
                               -1.0  1.0 0.0,
                                1.0 -1.0 0.0,
                               -1.0 -1.0 0.0])
        indices  (int-array [0 1 3 2])]
    (setup-vao vertices indices)))