Development Diary #11
Description:
In this blog post, we'll explore the fascinating world of programming by learning how to write the classic "Hello, World!" program in 50 different programming languages. From the simplicity of Python to the elegance of Haskell, we'll take a brief tour through a diverse range of languages, each with its own unique syntax and approach to programming. Whether you're a seasoned developer looking to expand your repertoire or a curious beginner eager to dip your toes into the vast ocean of code, this journey promises to be both educational and entertaining.
Now, let's dive into writing "Hello, World!" in 50 different languages:
1 Python:
print("Hello, World!")
2 Java:
public class HelloWorld {
public static void main(String[] args) {
System out println("Hello, World!");
}
}
3 C:
#include <stdio h>
int main() {
printf("Hello, World!\n");
return 0;
}
4 C++:
#include <iostream>
int main() {
std::cout << "Hello, World!" << std::endl;
return 0;
}
5 JavaScript:
console log("Hello, World!");
6 Ruby:
puts "Hello, World!"
7 Swift:
print("Hello, World!")
8 Go:
package main
import "fmt"
func main() {
fmt Println("Hello, World!")
}
9 Rust:
fn main() {
println!("Hello, World!");
}
10 PHP:
<?php
echo "Hello, World!";
?>
11 Perl:
print "Hello, World!\n";
12 Kotlin:
fun main() {
println("Hello, World!")
}
13 Scala:
object HelloWorld {
def main(args: Array[String]): Unit = {
println("Hello, World!")
}
}
14 Lua:
print("Hello, World!")
15 Haskell:
main :: IO ()
main = putStrLn "Hello, World!"
16 Dart:
void main() {
print('Hello, World!');
}
17 Shell:
echo "Hello, World!"
18 Batch:
@echo off
echo Hello, World!
19 PowerShell:
Write-Output "Hello, World!"
20 VBScript:
MsgBox "Hello, World!"
21 Objective-C:
#import <Foundation/Foundation h>
int main(int argc, const char * argv[]) {
@autoreleasepool {
NSLog(@"Hello, World!");
}
return 0;
}
22 Assembly:
section data
hello db 'Hello, World!',10
len equ $ - hello
section text
global _start
_start:
; write our string to stdout
mov eax, 4 ; sys_write
mov ebx, 1 ; file descriptor 1 (stdout)
mov ecx, hello ; message to write
mov edx, len ; message length
int 0x80 ; syscall
; exit
mov eax, 1 ; sys_exit
xor ebx, ebx ; exit status 0
int 0x80 ; syscall
23 VBA (Visual Basic for Applications):
Sub HelloWorld()
MsgBox "Hello, World!"
End Sub
24 Tcl:
puts "Hello, World!"
25 COBOL:
IDENTIFICATION DIVISION
PROGRAM-ID HELLO-WORLD
PROCEDURE DIVISION
DISPLAY "Hello, World!"
STOP RUN
26 F#:
printfn "Hello, World!"
27 Elixir:
IO puts "Hello, World!"
28 SQL (MySQL):
SELECT 'Hello, World!';
29 SQL (SQLite):
SELECT 'Hello, World!';
30 SQL (PostgreSQL):
SELECT 'Hello, World!';
31 SQL (Oracle):
SELECT 'Hello, World!' FROM DUAL;
32 SQL (SQL Server):
PRINT 'Hello, World!';
33 Smalltalk:
Transcript show: 'Hello, World!'; cr
34 R:
cat("Hello, World!\n")
35 Bash:
echo "Hello, World!"
36 Erlang:
-module(hello)
-export([hello_world/0])
hello_world() ->
io:fwrite("Hello, World!~n")
37 Julia:
println("Hello, World!")
38 MATLAB:
disp('Hello, World!');
39 AutoHotkey:
MsgBox, Hello, World!
40 Clojure:
(println "Hello, World!")
41 Groovy:
println "Hello, World!"
42 OCaml:
print_endline "Hello, World!"
43 D:
import std stdio;
void main()
{
writeln("Hello, World!");
}
44 Crystal:
puts "Hello, World!"
45 Nim:
echo "Hello, World!"
46 Common Lisp:
(format t "Hello, World!~%")
47 Scheme:
(display "Hello, World!")
(newline)
48 Prolog:
:- initialization(main)
main :-
write('Hello, World!'), nl,
halt
49 ABAP:
REPORT ZHELLO_WORLD
WRITE: / 'Hello, World!'
50 VB NET:
vb net
Module HelloWorld
Sub Main()
Console WriteLine("Hello, World!")
End Sub
End Module
That concludes our journey through "Hello, World!" in 50 different programming languages Whether you're just starting or have been programming for years, exploring different
Did you know that when you write a form-1, form-2 or form-3 Reagent component they all default to becoming React class components
?
For example, if you were to write this form-1
Reagent component:
(defn welcome []
[:h1 "Hello, friend"])
By the time Reagent passes it to React it would be the equivalent of you writing this:
class Welcome extends React.Component {
render() {
return <h1>Hello, friend</h1>
}
}
Okay, so, Reagent components become React Class Components
. Why do we care? This depth of understanding is valuable because it means we can better understand:
The result of all of this "fundamental" learning is we can more effectively harness JavaScript from within ClojureScript.
The reason all of your Reagent components become class components
is because
all of the code you pass to Reagent is run through an internal Reagent function
called create-class.
create-class
is interesting because of how it uses JavaScript to
transform a Reagent component into something that is recognized as a React
class component. Before we look into what create-class
is doing, it's
helpful to review how "classes" work in JavaScript.
Prior to ES6, JavaScript did not have classes. and this made some JS developers sad because classes are a common pattern used to structure code and provide support for:
But as I said, prior to ES6, JavaScript didn't have a formal syntax for "classes". To compensate for the lack of classes, the JavaScript community got creative and developed a series of instantiation patterns to help simulate classes.
Of all of these patterns, the pseudoclassical instantiation pattern
became one of the most popular ways to simulate a class in JavaScript. This is evidenced by the fact that many of the "first generation" JavaScript libraries and frameworks, like google closure library and backbone, are written in this style.
The reason we are going over this history is because the thing about a programming language is there are "patterns" and "syntax". The challenge with "patterns" is:
The last point in praticular is relevant to our conversation because patterns live in a context and assume prior knowledge. Knowledge like how well we know the context of a problem, the alternative approaches to addressing a problem, advancements in a language and so on.
The end result is that a pattern can just become a thing we do. We can forget or never know why it started in the first place or what the world could look like if we chose a different path.
For example, the most common way of writing a React class component is to use ES6 class syntax. But did you know that ES6 class syntax is little more than syntactic sugar around the pseudoclassical instantiation pattern
?
For example, you can write a valid React class component using the pseudoclassical instantiation pattern
like this:
// 1. define a function (component) called `Welcome`
function Welcome(props, context, updater) {
React.Component.call(this, props, context, updater)
return this
}
// 2. connect `Welcome` to the `React.Component` prototype
Welcome.prototype = Object.create(React.Component.prototype)
// 3. re-define the `constructor`
Object.defineProperty(Welcome.prototype, 'constructor', {
enumerable: false,
writable: true,
configurable: true,
value: Welcome,
})
// 4. define your React components `render` method
Welcome.prototype.render = function render() {
return <h2>Hello, Reagent</h2>
}
While the above is a valid React Class Component
, it's also verbose and error prone. For these reasons JavaScript introduced ES6 classes to the language:
class Welcome extends React.Component {
render() {
return <h1>Hello, Reagent</h1>
}
}
For those looking for further evidence, we can support our claim that ES6 Classes
result in same thing as what the pseudoclassical instantiation pattern
produces by using JavaScript's built-in introspection tools to compare the pseudoclassical instantiation pattern
to the ES6 class
syntax.
pseudoclassical instantiation pattern:
function Welcome(props, context, updater) {
React.Component.call(this, props, context, updater)
return this
}
// ...repeat steps 2 - 4 from above before completing the rest
var welcome = new Welcome()
Welcome.prototype instanceof React.Component
// => true
Object.getPrototypeOf(Welcome.prototype) === React.Component.prototype
// => true
welcome instanceof React.Component
// => true
welcome instanceof Welcome
// => true
Object.getPrototypeOf(welcome) === Welcome.prototype
// => true
React.Component.prototype.isPrototypeOf(welcome)
// => true
Welcome.prototype.isPrototypeOf(welcome)
// => true
ES6 class
class Welcome extends React.Component {
render() {
console.log('ES6 Inheritance')
}
}
var welcome = new Welcome()
Welcome.prototype instanceof React.Component
// => true
Object.getPrototypeOf(Welcome.prototype) === React.Component.prototype
// => true
welcome instanceof React.Component
// => true
welcome instanceof Welcome
// => true
Object.getPrototypeOf(welcome) === Welcome.prototype
// => true
React.Component.prototype.isPrototypeOf(welcome)
// => true
Welcome.prototype.isPrototypeOf(welcome)
// => true
What does all of this mean? As far as JavaScript and React are concerned, both definions of the Welcome
component are valid React Class Components
.
With this in mind, lets look at Reagent's create-class
function and see what it does.
The history lesson from the above section is important because create-class
uses a modified version of the pseudoclassical instantiation pattern
. Let's take a look at what we mean.
The following code sample is a simplified version of Reagent's create-class
function:
function cmp(props, context, updater) {
React.Component.call(this, props, context, updater)
return this
}
goog.extend(cmp.prototype, React.Component.prototype, classMethods)
goog.extend(cmp, React.Component, staticMethods)
cmp.prototype.constructor = cmp
What we have above is Reagents take on the pseudoclassical instantiation pattern
with a few minor tweaks:
// 1. we copy to properties + methods of React.Component
goog.extend(cmp.prototype, React.Component.prototype, classMethods)
goog.extend(cmp, React.Component, staticMethods)
// 2. the constructor is not as "thorough"
cmp.prototype.constructor = cmp
Exploring point 1 we see that Reagent has opted to copy the properties
and methods
of React.Component
directly to the Reagent compnents we write. That is what's happening here:
goog.extend(cmp.prototype, React.Component.prototype, classMethods)
If we were using the the traditional pseudoclassical
approach we would instead do this:
cmp.prototype = Object.create(React.Component.prototype)
Thus, the difference is that Reagent's approach copies all the methods and properties from React.Component
to the cmp
prototype where as the second approach is going to link
the cmp
prototype to React.component
prototype. The benefit of linking is that each time you instantiate a Welcome
component, the Welcome
component does not need to re-create all of the React.components
methods and properties.
Exploring the second point, Reagent is doing this:
cmp.prototype.constructor = cmp
whereas with the traditional pseudoclassical
approach we would instead do this:
Object.defineProperty(Welcome.prototype, 'constructor', {
enumerable: false,
writable: true,
configurable: true,
value: Welcome,
})
The difference in the above approaches is that if we just use =
as we are doing in the Reagent version we create an enumerable
constructor. This can have an implication depending on who consumes our classes, but in our case we know that only React is going to be consuming our class components, so we can do this with relative confidence.
What is one of the more interesting results of the above two Reagent modifications? First, if React depended on JavaScript introspection to tell whether or not a component is a child of React.Component
we would not be happy campers:
Welcome.prototype instanceof React.Component
// => false...Welcome is not a child of React.Component
Object.getPrototypeOf(Welcome.prototype) === React.Component.prototype
// => false...React.component is not part of Welcomes prototype chain
welcome instanceof React.Component
// => false...Welcome is not an instance of React.Component
welcome instanceof Welcome
// => true...welcome is a child of Welcome
Object.getPrototypeOf(welcome) === Welcome.prototype
// => true...welcome is linked to Welcome prototype
console.log(React.Component.prototype.isPrototypeOf(welcome))
// => false...React.Component not linked to the prototype of React.Component
console.log(Welcome.prototype.isPrototypeOf(welcome))
// is Welcome is the ancestory?
What the above shows is that Welcome
is not a child of React.component
even though it has all the properties and methods that React.Component
has. This is why were lucky that React is smart about detecting class vs. function components.
Second, by copying
rather than linking
prototypes we could inccur a
performance cost. How much of a performance hit? In our case this cost is likely
negligible.
In my experience, digging into the weeds and going on these detours has been an important part of my growth as a developer. The weeds have allowed me to be a better programmer because I'm honing my ability to understand challenging topics and find answers. The result is a strange feeling of calm and comfort.
This calm and comfort shouldn't be overlooked. So much of our day-to-day is left unquestioned and unanalyzed. We let knowledge become "cultural" or "tribal". This is scary. It's scary because it leads to bad decisions because no one around us knows the whys or wherefores. Ultimately, it's a bad habit. A bad habit which is seen by some as a virtue because it would simply take too much time for to learn things ourselves. That's until you actually start doing this kind of work and spend time learning and observing and seeing that these "new things" we're seeing all the time aren't really new, but just another example of that old thing back.
The Clojure Tools
are a group of convenience tools which currently consist of:
Clojure CLI
tools.build
The Clojure Tools
.
were designed to answer some of the following questions:
Clojure CLI
)Clojure CLI
)Clojure CLI
)Clojure CLI
)tools.build
)The rest of this post will dig into each of these tools.
The Clojure CLI
is a CLI program. Here is what it looks like to use the
Clojure CLI
and some of the things it can do:
Run a Clojure repl
clj
Run a Clojure program
clj -M -m your-clojure-program
manage Clojure dependencies
clj -Sdeps '{:deps {bidi/bidi {:mvn/version "2.1.6"}}}'
Like all Clojure
programs, the Clojure CLI
is built on a few libraries:
deps
)The following sections will provide overviews of each of the above tools.
The Clojure CLI
is invoked by calling either clj
or clojure
shell commands:
# clj
clj -M -m your-clojure-program
# clojure
clojure -M -m your-clojure-program
Under the hood, clj
actually calls clojure
. The difference is that clj
wraps the clojure
command with a tool called rlwrap. rlwrap
improves the developer experience by making it easier for you, a human, to type
in the terminal while you're running your Clojure REPL. However, even though
it's easier for you to type, rlwrap
can make it hard to compose the clj
command with other tools. As a result, it's a common practice to use clojure
in production/ci environments .
Additionally, not all environments have access to rlwrap
so it's another
dependency you have to install.
Okay, so they do the same thing. What do they do? clj/clojure
has one job:
run Clojure programs against a classpath.
The next sections will outline the tools that make up the Clojure CLI
tool.
If you dig into the clj/clojure
is just a bash script which ultimatley calls
a command like this:
java [java-opt*] -cp classpath clojure.main [init-opt*] [main-opt] [arg*]
Thus, the Clojure CLI
tool makes it easier to run Clojure programs. It saves
you having to type out a gnarly Java
command and make it work on different
environments (windows, linux, mac etc). However, it orchestrates the building
of the classpath by calling out to tools.deps
.
tools.deps
is a Clojure libary responsible for managing your dependencies.
It does the following things:
deps.edn
fileWhat's interesting about this program is that it's just a Clojure library.
This means that you can use it outside of the Clojure CLI
.
The other thing that makes tools.deps
great is that it's a small and
focused library. Why this is great is that if something goes wrong it's easy
to read and learn the library in a short period of time.
deps.edn
is just an edn file where you configure your project and specify
project dependencies. You can think of it like Clojure's version of
package.json
. The deps.edn
file is a Clojure map
with a specific structure. Here's an example of some of the properties of
a deps.edn
file:
{:deps {...}
:paths [...]
:aliases {...}}
As you can see, we use the keywords :deps
, :paths
and :aliases
and more
to start to describe your project and the dependencies it requires.
As we noted above, deps.edn
is read in when you run clj/clojure
and tells clj/clojure
which dependencies are requires to run your project.
tools.build
is a Clojure library with functions for building clojure projects.
For example, build a jar
or uberjar
.
The way you would use tools.build
is by writing a separate program inside
your app which knows how to build your app. The convention is to create a
build.clj
file in the root of your project. Import tools.build
and use
the functions provides by tools.build
to build your program.
The 3 main types of Clojure programs one might build into 3 sub categories:
tool
library
app
When you run your build.clj
file, you will use Clojure CLI
's -T
switch.
The -T
switch is meant to run general clojure programs via the Clojure CLI
and since build.clj
is a separate program, distinct form the app you are
writing, you would run it via the -T
switch.
You would use -T
for Clojure programs that you want to run as a "tool". For
example, deps-new is a Clojure library which creates new Clojure projects
based on a template you provide. This is a great example of a Clojure project
which is built to be a "tool".
I don't want to go into more detail about -T
now because that means we would
have to dive into other Clojure CLI
switches like -X
and -M
. That's for
another post. On to the Installer!
The "Clojure CLI Installer" is a fancy way of referring to the brew tap
used
to install Clojure on mac and linux machines. As of February 2020, Clojure
started maintaining their own brew tap. Thus, if you installed the
Clojure CLI
via
brew install clojure
you will likely want to uninstall clojure
and install the following:
brew install clojure/tools/clojure
In all likelihood, you would probably be fine with brew install clojure
as it
will recieve updates. However, while brew install clojure
will still see some
love, it won't be as active as the clojure/tools/clojure
tap.
This section will provide a quick comparison of clj
, lein
and boot
.
Firstly, all of the above tools are more or less addressing the same problems in their own way. Your job is to choose the one you like best.
If you're curious which to choose, my answer is the Clojure CLI
.
The reason I like the Clojure CLI
is because the tool is simple.
You can read through clj
and tools.deps
in an afternoon and understand
what they are doing. The same (subjectively of course) cannot be
said for lein
or boot
. I will note that Clojure CLI
's API is not
straightforward and can be confusing.
Secondly, the Clojure Tools
promote libraries over frameworks. This is
important when working with a language like Clojure because it really does
reward you for breaking down your thinking.
Finally, the Clojure community is really leaning into
building tools for Clojure CLI
. For example, where lein
used to have significantly
more functionality, the community has built a ton of incredible tools that
will cover many of your essential requirements.
RSS is a trendy topic among tech circles, it’s coming back from time to time without a breakthrough but still, it’s a technology used. Unfortunately, a lot of websites don’t offer RSS feeds anymore.
Big web players like Facebook, Twitter, and LinkedIn are promoting web silos. They want us to stay on their website as long as possible. They design their news feed for that purpose and not for the user best interest.
Thanks to WordPress and the technology entropy millions of websites still offer a RSS feed. What to do for websites that don’t?
I built it with remus, a library for parsing RSS feed, and added functions giving similar output using alternative mechanisms to build the feed.
I initially started the project with the idea that when a website does not offer a RSS feed, it is still most likely offering a sitemap to be easily accessible by search engines. Websites still want their content to be found by Google, right?
To my surprise, that often does not work. Some websites have no sitemap at all (I guess they are not doing really well on SEO…), or the file is broken for various reasons: the XML is invalid, the date is in the wrong format or missing, the sitemap has thousands or URLs with no grouping of any sort… I get some feeds from that but globally that often did not allow me to create a news feed and that requires a lot of resources to crawl the URL and see what they are about. I’m running a side project on a single machine, so I don’t want to set up a piece of heavy machinery [1] to basically create an XML summary of a webpage.
A second quick solution was to collect all the links of a webpage. It was super easy to implement, thanks to enlive. The algorithm:
Since the previous solution, I have learned about zipper. This is an extremely powerful library to recursively parse a tree in a state machine. It was part of Clojure 1.0 and was initially designed by Gérard Huet. The library is somehow not well documented and known (?). Here are some good materials to see how to use it:
Even if it’s not a well-known library, many tools of the Clojure day-to-day developer use it through rewrite-clj
Since an HTML page is a tree, that seems like a perfect match! I just need the links that look legit and “look around” to extract the feed entry.
Here is the code.
The simplified algorithm for the “looking around” part:
Visually it goes like this:
The block parsing consists of finding the most likely link to the article, the title, the optional description, and the optional date. Code here.
Basically, the algo searches a few levels up the link itself, assuming that the title/description/date are located in a nearby HTML element. 4 levels and 2 children are numbers I found by experimentation.
It seems to work okay on the few websites I added to my news reader. Let’s hope it is 🤞🏻
The state of information systems on the open Internet:
What it should be: Database → News feed
What it is (yolo): Database → … → HTML → educated guess extraction
What a waste of CPU cycle 🤦
Of course, it is not the quality of a real RSS feed: information is extracted on educated guesses where somewhere they are lying properly organized in a relational database. It’s a waste of resources compared to the time were RSS feeds were a hype technology, but a good workaround to create my own news feed instead of consuming the dubious one built by the big tech company.
[1] Something you can run and operate as a single individual, or small team. Not something requiring an elephant cluster or hundreds of machines.
The post What’s the personality behind Clojure? appeared first on Flexiana.
Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem (feed: RSS). Thanks to Anton Fonarev for link aggregation.
Bringing Real-Time AI to Phone Calls using core.async (by Ovi Stoica) - London Clojurians
Learn Clojure – Recursion - Clojure Diary
Seajure March 2024 - YAMLScript - Amperity
Clojure visual-tools meeting 22: ggplot study session 2 - Sci Cloj
January & February 2024 Long-Term Project Updates - Kathy Davis (Clojurists Together)
Zero User’s Guide - Ray Stubbs
Rama: How to achieve a transactional update across partitions - Jakub Holý
Hands on Rama, day 3: Foreign keys and data integrity, macros, queries - Jakub Holý
A demo about being cautious with clj→js and js→clj - Peter Strömberg
Named capturing groups in Clojure - Gert Goet
New releases and tools this week:
muuntaja 0.6.10 - Clojure library for fast http api format negotiation, encoding and decoding
next-jdbc 1.3.925 - A modern low-level Clojure wrapper for JDBC-based access to databases
flow-storm-debugger 3.13.1 - A debugger for Clojure and ClojureScript with some unique features
contajners 1.0.4 - An idiomatic, data-driven, REPL friendly clojure client for OCI container engines
typedclojure 1.2.1 - An optional type system for Clojure
clay 2-alpha87 - A tiny Clojure tool for dynamic workflow of data visualization and literate programming
calva 2.0.425 - Clojure & ClojureScript Interactive Programming for VS Code
metamorph.ml 0.7.8 - Machine learning functions for metamorph based on machine learning pipelines. Part of scicloj.ml
overarch 0.13.0 - A data driven description of software architecture based on UML and the C4 model
Ok, so I've been doing more intermittent work on catwalk
. There's a bunch of other stuff happening in life, so the project isn't going exactly as well as I wanted, but I'm making progress.
As of this writing, the latest updates to the catwalk
server have to do with adding CORS
headers, and the next couple are going to have to do with removing the blacklist system and adding a fairly general, API-key based user
system. The medium term plan is to get a bit more elaborate and allow external sign-ups in case someone else wants to use this to make their own blogs into podcasts.
The immediate next steps are all going to have to do with making the client a lot more performant. The problem is that, for some reason, the blogcast
page works really poorly. In particular, when I go to delete certain lines, or edit and re-record them, there's frequent, massive UI pauses that prevent me from moving around the page. There's also some issues I'm still working through regarding keyboard navigation, although as you'll understand in a moment, that's going to be less relevant.
I pushed a few commits recently that I hoped would make the situation more bearable, and let me catch up with my casting work, but they did no such thing. Instead slowing the client to even more of a crawl. I'm not entirely sure why this is happening, and it's entirely possible that it's a direct result of me setting up the state graph for this application in a moronic way. Still, I was hoping to get half-way decent performance out of it and didn't.
You probably didn't even realize I was trying for this. Ok, maybe that series of android pokes tipped my hand, but I don't think I explicitly mentioned what the goal is. What I'd really like with Catwalk is for it to become a web-and-or-mobile-based blogcast tool, not just a web one. I recognize that I could trivially accomplish both, in a sense, by making the web side of it reactive, but it's also sometimes nice to have a native app when you're dealing with a phone screen.
I'd been trying to figure out a way to get both of these things at once, in a nice, enjoyable clojure
context. This doesn't seem to be in the cards, unfortunately. Every attempt I've made to get a react-native-based CLJS project off the ground has ended up with me hitting roadblocks in the form of stale instructions and/or tooling. The latest one I've tried out is Expo. Which on its' own looks pretty snazzy, builds trivially on Android, seems to allow for separate web-deployment, and has a really nice experimental remote REPL on-device.
If I have to code in Javascript directly in order to use this, then so be it.
I've tried out some basics, and no big complaints so far. It looks like there are going to be some finer points regarding what elements specifically to use for native/web display, in particular SectionList
vs ScrollView
with a map
call has significantly different interactions, but no big roadblocks. Setting it up is as easy as
~$ yarn create expo-cljs-app your-app-name
...
~$ cd your-app-name
~/your-app-name$ yarn start
It then displays a QR code you can scan from the Anrdoid app to start an interactively updating dev preview of your application and go from there. Getting the web client out of it is then as easy as
~/your-app-name$ yarn expo export -p web
It looks like getting the android version is going to involve changing the value passed to that -p
flag, and also setting up an eas.json
file and doing some more stuff through EAS. I only skimmed the intro, but it doesn't look too intimidating.
As always, I'll let you know how it goes.
Do you love interactive development? Although Clojure is set up perfectly for that, evaluating buffers one at a time can only get you so far.
Once you start dealing with the state, you get data dependencies, and with them, evaluation order starts to matter, and now you change one line but have to re-eval half of your application to see the change.
But how do you know which half?
Clj-reload to the rescue!
Clj-reload scans your source dir, figures out the dependencies, tracks file modification times, and when you are finally ready to reload, it carefully unloads and loads back only the namespaces that you touched and the ones that depend on those. In the correct dependency order, too.
Let’s do a simple example.
a.clj:
(ns a
(:require b))
b.clj:
(ns b
(:require c))
c.clj:
(ns c)
Imagine you change something in b.clj
and want to see these changes in your current REPL. What do you do?
If you call
(clj-reload.core/reload)
it will notice that
b.clj
was changed,a.clj
depends on b.clj
,c.clj
but it doesn’t depend on a.clj
or b.clj
and wasn’t changed.Then the following will happen:
Unloading a
Unloading b
Loading b
Loading a
So:
c
wasn’t touched — no reason to,b
was reloaded because it was changed,a
was loaded after the new version of b
was in place. Any dependencies a
had will now point to the new versions of b
.That’s the core proposition of clj-reload
.
Here, I recorded a short video:
But if you prefer text, then start with:
(require '[clj-reload.core :as reload])
(reload/init
{:dirs ["src" "dev" "test"]})
:dirs
are relative to the working directory.
Use:
(reload/reload)
; => {:unloaded [a b c], :loaded [c b a]}
reload
can be called multiple times. If reload fails, fix the error and call reload
again.
Works best if assigned to a shortcut in your editor.
reload
returns a map of namespaces that were reloaded:
{:unloaded [<symbol> ...]
:loaded [<symbol> ...]}
By default, reload
throws if it can’t load a namespace. You can change it to return exception instead:
(reload/reload {:throw false})
; => {:unloaded [a b c]
; :loaded [c b]
; :failed b
; :exception <Throwable>}
By default, clj-reload will only reload namespaces that were both:
If you pass :only :loaded
option to reload
, it will reload all currently loaded namespaces, no matter if they were changed or not.
If you pass :only :all
option to reload
, it will reload all namespaces it can find in the specified :dirs
, no matter whether loaded or changed.
Some namespaces contain state you always want to persist between reloads. E.g. running web-server, UI window, etc. To prevent these namespaces from reloading, add them to :no-reload
during init
:
(reload/init
{:dirs ...
:no-reload '#{user myapp.state ...}})
Sometimes your namespace contains stateful resource that requires proper shutdown before unloading. For example, if you have a running web server defined in a namespace and you unload that namespace, it will just keep running in the background.
To work around that, define an unload hook:
(def my-server
(server/start app {:port 8080}))
(defn before-ns-unload []
(server/stop my-server))
before-ns-unload
is the default name for the unload hook. If a function with that name exists in a namespace, it will be called before unloading.
You can change the name (or set it to nil
) during init
:
(reload/init
{:dirs [...]
:unload-hook 'my-unload})
This is a huge improvement over tools.namespace
. tools.namespace
doesn’t report which namespaces it’s going to reload, so your only option is to stop everything before reload and start everything after, no matter what actually changed.
One of the main innovations of clj-reload
is that it can keep selected variables between reloads.
To do so, just add ^:clj-reload/keep
to the form:
(ns test)
(defonce x
(rand-int 1000))
^:clj-reload/keep
(def y
(rand-int 1000))
^:clj-reload/keep
(defrecord Z [])
and then reload:
(let [x test/x
y test/y
z (test/->Z)]
(reload/reload)
(let [x' test/x
y' test/y
z' (test/->Z)]
(is (= x x'))
(is (= y y'))
(is (identical? (class z) (class z')))))
Here’s how it works:
defonce
works out of the box. No need to do anything.def
/defn
/deftype
/defrecord
/defprotocol
can be annotated with ^:clj-reload/keep
and can be persistet too.clj-reload.core/keep-methods
multimethod.Why is this important? With tools.namespace
you will structure your code in a way that will work with its reload implementation. For example, you’d probably move persistent state and protocols into separate namespaces, not because logic dictates it, but because reload library will not work otherwise.
clj-reload
allows you to structure the code the way business logic dictates it, without the need to adapt to developer workflow.
Simply put: the fact that you use clj-reload
during development does not spill into your production code.
The simplest way to reload Clojure code is just re-evaluating an entire buffer.
It works for simple cases but fails to account for dependencies. If something depends on your buffer, it won’t see these changes.
The second pitfall is removing/renaming vars or functions. If you had:
(def a 1)
(def b (+ a 1))
and then change it to just
(def b (+ a 1))
it will still compile! New code is evaluated “on top” of the old one, without unloading the old one first. The definition of a
will persist in the namespace and let b
compile.
It might be really hard to spot these errors during long development sessions.
(require ... :reload-all)
Clojure has :reload
and :reload-all
options for require
. They do track upstream dependencies, but that’s about it.
In our original example, if we do
(require 'a :reload-all)
it will load both b
and c
. This is excessive (b
or c
might not have changed), doesn’t keep track of downstream dependencies (if we reload b
, it will not trigger a
, only c
) and it also “evals on top”, same as with buffer eval.
tools.namespace is a tool originally written by Stuart Sierra to work around the same problems. It’s a fantastic tool and the main inspiration for clj-reload
. I’ve been using it for years and loving it, until I realized I wanted more.
So the main proposition of both tools.namespace
and clj-reload
is the same: they will track file modification times and reload namespaces in the correct topological order.
This is how clj-reload
is different:
tools.namespace
reloads every namespace it can find. clj-reload
only reloads the ones that were already loaded. This allows you to have broken/experimental/auxiliary files lie around without breaking your workflow TNS-65tools.namespace
always reloads everything. In clj-reload
, even the very first reload only reloads files that were actually changed TNS-62clj-reload
supports namespaces split across multiple files (like core_deftype.clj
, core_defprint.clj
in Clojure) TNS-64clj-reload
can see dependencies in top-level standalone require
and use
forms TNS-64clj-reload
supports load and unload hooks per namespace TNS-63clj-reload
can specify exclusions during configuration, without polluting the source code of those namespaces.clj-reload
can keep individual vars around and restore previous values after reload. E.g. defonce
doesn’t really work with tools.namespace
, but it does with clj-reload
.clj-reload
has 2× smaller codebase and 0 runtime dependencies.clj-reload
doesn’t support ClojureScript. Patches welcome.Clj-reload grew from my personal needs on Humble UI project. But I hope other people will find it useful, too.
Let me know what works for you and what doesn’t! I’ll try to at least be on par with tools.namespace
.
And of course, here’s the link:
PG2 version 0.1.6 is out, and it ships various improvements to JSON(b) handling.
Postgres is amazing when dealing with JSON. There hardly can be a database that
serves it better. Unfortunately, Postgres clients never respect the JSON
feature, which is horrible. Take JDBC, for example: when querying a JSON(b)
value, you’ll get a dull PGObject
which should be decoded manually. The same
applies to insertion: one cannot just pass a Clojure map or a vector. It should
be packed into the PGObject
as well.
Of course, this can be automated by extending certain protocols. But it’s still slow as it’s done on Clojure level (not Java), and it forces you to copy the same code across projects.
Fortunately, PG2 supports JSON out from the box. If you query a JSON value, you’ll get its Clojure counter-part: a map, a vector, etc. To insert a JSON value to a table, you pass either a Clojure map or a vector. No additional steps are required.
PG2 relies on jsonista library to handle JSON. At the moment of writing, this is the fastest JSON library for Clojure. Jsonista uses a concept of object mappers: objects holding custom rules to encode and decode values. You can compose your own mapper with custom rules and pass it into the connection config.
Let’s prepare a connection and a test table with a jsonb column:
(def config
{:host "127.0.0.1"
:port 10140
:user "test"
:password "test"
:dbname "test"})
(def conn
(jdbc/get-connection config))
(pg/query conn "create table test_json (
id serial primary key,
data jsonb not null
)")
Now insert a row:
(pg/execute conn
"insert into test_json (data) values ($1)"
{:params [{:some {:nested {:json 42}}}]})
No need to encode a map manually nor wrap it into a sort of PGObject
. Let’s
fetch the new row by id:
(pg/execute conn
"select * from test_json where id = $1"
{:params [1]
:first? true})
{:id 1 :data {:some {:nested {:json 42}}}}
Again, the JSON data returns as a Clojure map with no wrappers.
When using JSON with HoneySQL though, some circs are still needed. Namely, you
have to wrap a value with [:lift ...]
as follows:
(pgh/insert-one conn
:test_json
{:data [:lift {:another {:json {:value [1 2 3]}}}]})
{:id 2, :data {:another {:json {:value [1 2 3]}}}}
Without the [:lift ...]
tag, HoneySQL will treat the value as a nested SQL map
and try to render it as a string, which will fail of course or lead to a SQL
injection.
Another way is to use HoneySQL parameters conception:
(pgh/insert-one conn
:test_json
{:data [:param :data]}
{:honey {:params {:data {:some [:json {:map [1 2 3]}]}}}})
For details, see the “HoneySQL Integration” section.
PG2 supports not only Clojure maps but vectors, sets, and lists. Here is an example with with a vector:
(pg/execute conn
"insert into test_json (data) values ($1)"
{:params [[:some :vector [:nested :vector]]]})
{:id 3, :data ["some" "vector" ["nested" "vector"]]}
In rare cases you might store a string or a number in a JSON field. Say, 123 is
a valid JSON value but it’s treated as a number. To tell Postgres it’s a JSON
indeed, wrap the value with pg/json-wrap
:
(pgh/insert-one conn
:test_json
{:data (pg/json-wrap 42)})
{:id 4, :data 42}
The wrapper is especially useful to store a “null” JSON value: not the standard
NULL
but "null"
which, when parsed, becomes nil
. For this, pass
(pg/json-wrap nil)
as follows:
(pgh/insert-one conn
:test_json
{:data (pg/json-wrap nil)})
{:id 5, :data nil} ;; "null" in the database
One great thing about Jsonista is a conception of mapper objects. A mapper is a set of rules how to encode and decode data. Jsonista provides a way to build a custom mapper. Once built, it can be passed to a connection config so the JSON data is written and read back in a special way.
Let’s assume you’re going to tag JSON sub-parts to track their types. For
example, if encoding a keyword :foo
, you’ll get a vector of ["!kw",
"foo"]
. When decoding that vector, by the "!kw"
string, the mapper
understands it a keyword and coerces "foo"
to :foo
.
Here is how you create a mapper with Jsonista:
(ns ...
(:import
clojure.lang.Keyword
clojure.lang.PersistentHashSet)
(:require
[jsonista.core :as j]
[jsonista.tagged :as jt]))
(def tagged-mapper
(j/object-mapper
{:encode-key-fn true
:decode-key-fn true
:modules
[(jt/module
{:handlers
{Keyword {:tag "!kw"
:encode jt/encode-keyword
:decode keyword}
PersistentHashSet {:tag "!set"
:encode jt/encode-collection
:decode set}}})]}))
The object-mapper
function accepts even more options but we skip them for now.
Now that you have a mapper, pass it into a config:
(def config
{:host "127.0.0.1"
:port 10140
:user "test"
:password "test"
:dbname "test"
:object-mapper tagged-mapper})
(def conn
(jdbc/get-connection config))
All the JSON operations made by this connection will use the passed object mapper. Let’s insert a set of keywords:
(pg/execute conn
"insert into test_json (data) values ($1)"
{:params [{:object #{:foo :bar :baz}}]})
When read back, the JSON value is not a vector of strings any longer but a set of keywords:
(pg/execute conn "select * from test_json")
[{:id 1, :data {:object #{:baz :bar :foo}}}]
To peek a raw JSON value, select it as a plain text and print (just to avoid escaping quotes):
(printl (pg/execute conn "select data::text json_raw from test_json where id = 10"))
;; [{:json_raw {"object": ["!set", [["!kw", "baz"], ["!kw", "bar"], ["!kw", "foo"]]]}}]
If you read that row using another connection with a default object mapper, the data is returned without expanding tags.
PG2 provides an utility namespace for JSON encoding and decoding. You can use it
for files, HTTP API, etc. If you already have PG2 in the project, there is no
need to plug in Cheshire or another JSON library. The namespace is pg.json
:
(ns ...
(:require
[pg.json :as json]))
The read-string
function reads a value from a JSON string:
(json/read-string "[1, 2, 3]")
[1 2 3]
The first argument might be an object mapper:
(json/read-string tagged-mapper "[\"!kw\", \"hello\"]")
:hello
The functions read-stream
and read-reader
act the same but accept either an
InputStream
or a Reader
object:
(let [in (-> "[1, 2, 3]" .getBytes io/input-stream)]
(json/read-stream tagged-mapper in))
(let [in (-> "[1, 2, 3]" .getBytes io/reader)]
(json/read-reader tagged-mapper in))
The write-string
function dumps an value into a JSON string:
(json/write-string {:test [:hello 1 true]})
;; "{\"test\":[\"hello\",1,true]}"
The first argument might be a custom object mapper. Let’s reuse our tagger mapper:
(json/write-string tagged-mapper {:test [:hello 1 true]})
;; "{\"test\":[[\"!kw\",\"hello\"],1,true]}"
The functions write-stream
and write-writer
act the same. The only
difference is, they accept either an OutputStream
or Writer
objects. The
first argument might be a mapper as well:
(let [out (new ByteArrayOutputStream)]
(json/write-stream tagged-mapper {:foo [:a :b :c]} out))
(let [out (new StringWriter)]
(json/write-writer tagged-mapper {:foo [:a :b :c]} out))
PG2 provides an HTTP Ring middleware for JSON. It acts like wrap-json-request
and wrap-json-response
middleware from the ring-json
library. Comparing to it, the PG2 stuff has the following advantages:
Imagine you have a Ring handler that reads JSON body and returns a JSON map. Something like this:
(defn api-handler [request]
(let [user-id (-> request :data :user_id)
user (get-user-by-id user-id)]
{:status 200
:body {:user user}}))
Here is how you wrap it:
(ns ...
(:require
[pg.ring.json :refer [wrap-json
wrap-json-response
wrap-json-request]]))
(def app
(-> api-handler
(wrap-this-foo)
(wrap-json <opt>)
(wrap-that-bar)))
Above, the wrap-json
wrapper is a combination of wrap-json-request
and
wrap-json-response
. You can apply them both explicitly:
(def app
(-> api-handler
(wrap-this-foo)
(wrap-json-request <opt>)
(wrap-json-response <opt>)
(wrap-that-bar)))
All the three wrap-json...
middleware accept a handler to wrap and a map of
options. Here is the options supported:
Name | Direction | Description |
---|---|---|
:object-mapper |
request, response | An custom instance of ObjectMapper |
:slot |
request | A field to assoc the parsed JSON data (1) |
:malformed-response |
request | A ring response returned when payload cannot be parsed (2) |
Notes:
The default slot name is :json
. Please avoid using :body
or :params
to
prevent overriding existing request fields. This is especially important for
:body
! Often, you need the origin input stream to calculate an MD5 or
SHA-256 hash-sum of the payload. If you overwrite the :body
field, you
cannot do that.
The default malformed response is something like 400 “Malformed JSON” (plain text).
A full example:
(def json-opt
{:slot :data
:object-mapper tagged-mapper ;; see above
:malformed-response {:status 404
:body "<h1>Bad JSON</h1>"
:headers {"content-type" "text/html"}}})
(def app
(-> api-handler
(wrap-this-foo)
(wrap-json json-opt)
(wrap-that-bar)))
How to make progress without making a mess.
Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem (feed: RSS). Thanks to Anton Fonarev for link aggregation.
Brimm: graph backend with a Notion-inspired UI (by Filip Juruković) - London Clojurians
Better Sieve - Finding first n unique characters in Clojure - Clojure Diary
Let’s talk about Clojure editors and IDEs: Emacs, IntelliJ IDEA Cursive, VSCode Calva, Vim - Andrey Fadeev
REPL-driving the browser by Josh Glover - Func Prog Sweden
Heart of Clojure - Heart of Clojure conference announcement
With YAMLScript, YAML Becomes a Proper Programming Language - Joab Jackson
New releases and tools this week:
pretty 2.3 - Library for helping print things prettily, in Clojure - ANSI fonts, formatted exceptions
hirundo 0.1.33 - Helidon 4.x RING adapter - using loom/java21+
clojure-lsp-intellij 2.1.0 - Intellij Plugin for Clojure & ClojureScript development via Language Server (LSP) made in Clojure
datalevin 0.9.3 - A simple, fast and versatile Datalog database
Clojure-Sublimed 3.7.0 - Clojure support for Sublime Text 4
calva 2.0.422 - Clojure & ClojureScript Interactive Programming for VS Code
timbre-json-appender 0.2.12 - JSON appender for Timbre
trace 1.1 - Better REPL based debugging output
vl-gallery-edn - vega-lite gallery in EDN format
deps.clj 1.11.2.1446 - A faithful port of the clojure CLI bash script to Clojure
guardrails 1.2.4 - Efficient, hassle-free function call validation with a concise inline syntax for clojure.spec and Malli
metamorph 0.2.4 - A Clojure library designed to providing pipelining operations
metamorph.ml 0.7.3 - Machine learning functions for metamorph based on machine learning pipelines. Part of scicloj.ml
clj-kondo 2024.03.13 - Static analyzer and linter for Clojure code that sparks joy
clojure-lsp 2024.03.13-13.11.00 - Clojure & ClojureScript Language Server (LSP) implementation
clay 2-alpha82 - A tiny Clojure tool for dynamic workflow of data visualization and literate programming
bubble-config - An aero powered config with environments aimed at Babashka tasks
yamlscript 0.1.42 - Programming in YAML
shadow-cljs-ext - Loading the shadow-cljs UI in browser devtools
easyreagent - Easy React Components for Reagent
PG2 version 0.1.5 ships its own migration engine through the
pg2-migration
package. Like Migratus or Ragtime, it allows to grow the
database schema continuously, track changes and apply them with care.
;; lein
[com.github.igrishaev/pg2-migration "0.1.5]
;; deps
com.github.igrishaev/pg2-migration {:mvn/version "0.1.5"}
Migrations are SQL files that are applied to the database in certain order. A
migration has an id and a direction: next/up or prev/down. Usually it’s split on
two files called <id>.up.sql
and <id>.down.sql
holding SQL commands. Say,
the -up file creates a table with an index, and the -down one drops the index
first, and then the table.
Migrations might have a slug: a short human-friendly text describing
changes. For example, in a file called 002.create-users-table.up.sql
, the slug
is “Create users table”.
In PG2, the migration framework looks for files matching the following pattern:
<id>.<slug>.<direction>.sql
where:
id
is a Long number, for example 12345 (a counter), or 20240311 (date
precision), or 20240311235959 (date & time precision);
slug
is an optional word or group of words joined with -
or _
, for
example create-users-table-and-index
or remove_some_view
. When rendered,
both -
and _
are replaced with spaces, and the phrase is capitalized.
direction
is either prev/down
or next/up
. Internally, down
and up
are transformed to prev
and next
because these two have the same amount of
characters and files look better.
Examples:
001-create-users.next.sql
012-next-only-migration.up.sql
153-add-some-table.next.sql
Above, the leading zeroes in ids are used for better alignment only. Infernally
they are transferred into 1, 12 and 153 Long numbers. Thus, 001
, 01
and 1
become the same id 1 after parsing.
Each id has at most two directions: prev/down and next/up. On bootstrap, the engine checks it to prevent weird behaviour. The table below shows there are two rows which, after parsing, have the same (id, direction) pair. The bootstrap step will end up with an exception saying which files duplicate each other.
Filename | Parsed |
---|---|
001-some-trivial-slug.next.sql |
(1, next) |
001-some-simple-slug.next.sql |
(1, next) |
A migration might have only one direction, e.g. next/up or prev/down file only.
When parsing, the registry is ignored meaning that both
001-Create-Users.NEXT.sql
and 001-CREATE-USERS.next.SQL
files produce the
same map.
The files hold SQL expressions to be evaluated by the engine. Here is the
content of the 001-create-users.next.sql
file:
create table IF NOT EXISTS test_users (
id serial primary key,
name text not null
);
BEGIN;
insert into test_users (name) values ('Ivan');
insert into test_users (name) values ('Huan');
insert into test_users (name) values ('Juan');
COMMIT;
Pay attention to the following points.
A single file might have as many SQL expressions as you want. There is no need
to separate them with magic comments like --;;
as Migratus requires. The
whole file is executed in a single query. Use the standard semicolon at the
end of each expression.
There is no a hidden transaction management. Transactions are up to you: they
are explicit! Above, we wrap tree INSERT
queries into a single
transaction. You can use save-points, rollbacks, or whatever you want. Note
that not all expressions can be in a transaction. Say, the CREATE TABLE
one
cannot and thus is out from the transaction scope.
For granular transaction control, split your complex changes on two or three files named like this:
# direct parts
001-huge-update-step-1.next.sql
002-huge-update-step-2.next.sql
003-huge-update-step-3.next.sql
# backward counterparts
003-huge-update-step-3.prev.sql
002-huge-update-step-2.prev.sql
001-huge-update-step-1.prev.sql
At the moment, neither .edn
nor .clj
migrations are supported. This is by
design because personally I’m highly against mixing SQL and Clojure. Every time
I see an EDN transaction, I get angry. Mixing these two for database management
is the worst idea one can come up with. If you’re thinking about migrating a
database with Clojure, please close you laptop and have a walk to the nearest
park.
Migration files are stored in project resources. The default search path is
migrations
. Thus, their physical location is resources/migrations
. The
engine scans the migrations
resource for children files. Files from nested
directories are also taken into account. The engine supports Jar resources when
running the code from an uberjar.
The resource path can be overridden with settings.
All the applied migrations are tracked in a database table called migrations
by default. The engine saves the id and the slug or a migration applied as well
as the current timestamp of the event. The timestamp field has a time zone. Here
is the structure of the table:
CREATE TABLE IF NOT EXISTS migrations (
id BIGINT PRIMARY KEY,
slug TEXT,
created_at timestamp with time zone not null default current_timestamp
)
Every time you apply a migration, a new record is inserted into the table. On rollback, a corresponding migration is deleted.
You can override the name of the table in settings (see below).
The migration engine is controlled with both API and CLI interface. Let’s review CLI first.
The pg.migration.cli
namespaces acts like the main entry point. It accepts
general options, a command, and command-specific options:
<global options> <command> <command options>
General options are:
-c, --config CONNFIG migration.config.edn Path to the .edn config file
-p, --port PORT 5432 Port number
-h, --host HOST localhost Host name
-u, --user USER The current USER env var User
-w, --password PASSWORD <empty string> Password
-d, --database DATABASE The current USER env var Database
--table TABLE :migrations Migrations table
--path PATH migrations Migrations path
Most of the options have default values. Both user and database names come from
the USER
environment variable. The password is an empty string by default. For
local trusted connections, the password might not be required.
The list of the commands:
Name | Meaning |
---|---|
create | Create a pair of blank up & down migration files |
help | Print a help message |
list | Show all the migrations and their status (applied or not) |
migrate | Migrate forward (everything, next only, or up to a certain ID) |
rollback | Rollback (the current one, everything, or down to a certain ID) |
Each command has its own sub-options which we will describe below.
Here is how you review the migrations:
<lein or deps preamble> \
-h 127.0.0.1 \
-p 10150 \
-u test \
-w test \
-d test \
--table migrations_test \
--path migrations \
list
| ID | Applied? | Slug
| ----- | -------- | --------
| 1 | true | create users
| 2 | false | create profiles
| 3 | false | next only migration
| 4 | false | prev only migration
| 5 | false | add some table
Every command has its own arguments and help message. For example, to review the
create
command, run:
lein with-profile +migrations run -m pg.migration.cli -c config.example.edn create --help
Syntax:
--id ID The id of the migration (auto-generated if not set)
--slug SLUG Optional slug (e.g. 'create-users-table')
--help false Show help message
Passing -u
, -h
, and other arguments all the time is inconvenient. The engine
can read them at once from a config file. The default config location is
migration.config.edn
. Override the path to the config using the -c
parameter:
<lein/deps> -c config.edn list
The config file has the following structure:
{:host "127.0.0.1"
:port 10150
:user "test"
:password #env PG_PASSWORD
:database "test"
:migrations-table :migrations_test
:migrations-path "migrations"}
The :migrations-table
field must be a keyword because it takes place in a
HoneySQL map.
The :migrations-path
field is a string referencing a resource with migrations.
Pay attention to the #env
tag. The engine uses custom readers when loading a
config. The tag reads the actual value from an environment variable. Thus, the
database password won’t be exposed to everyone. When the variable is not set, an
exception is thrown.
The create
command makes a pair of two blank migration files. If not set, the
id is generated automatically using the YYYYmmddHHMMSS
pattern.
lein with-profile +migration run -m pg.migration.cli \
-c config.example.edn \
create
ls -l migrations
20240312074156.next.sql
20240312074156.prev.sql
You can also provide a custom id and a slug as well:
lein with-profile +migration run -m pg.migration.cli \
-c config.example.edn \
create \
--id 100500 \
--slug 'some huge changes in tables'
ll migrations
100500.some-huge-changes-in-tables.next.sql
100500.some-huge-changes-in-tables.prev.sql
20240312074156.next.sql
20240312074156.prev.sql
The list
command renders all the migrations and their status: whether they are
applied or not.
lein with-profile +migration run -m pg.migration.cli -c config.example.edn list
| ID | Applied? | Slug
| ----- | -------- | --------
| 1 | true | create users
| 2 | true | create profiles
| 3 | true | next only migration
| 4 | false | prev only migration
| 5 | false | add some table
The migrate
command applies migrations to the database. By default, all the
pending migrations are processed. You can change this behaviour using these
flags:
... migrate --help
Syntax:
--all Migrate all the pending migrations
--one Migrate next a single pending migration
--to ID Migrate next to certain migration
--help false Show help message
With the --one
flag set, only one next migration will be applied. If --to
parameter is set, only migrations up to this given ID are processed. Examples:
... migrate # all migrations
... migrate --all # all migrations
... migrate --one # next only
... migrate --to 123 # all that <= 123
The rollback
command reverses changes in the database and removes
corresponding records from the migration table. By default, only the current
migration is rolled back. Syntax:
... rollback --help
Syntax:
--all Rollback all the previous migrations
--one Rollback to the previous migration
--to ID Rollback to certain migration
--help false Show help message
The --one
argument is the default behaviour. When --all
is passed, all the
backward migrations are processed. To rollback to a certain migration, pass
--to ID
. Examples:
... rollback # current only
... rollback --one # current only
... rollback --to 20240515 # down to 20240515
... rollback --all # down to the very beginning
Lein preamble looks usually something like this:
> lein run -m pg.migration.cli <ARGS>
The pg2-migration
library must be in dependencies. Since migrations are
managed aside from the main application, they’re put into a separate profile,
for example:
:profiles
{:migrations
{:main pg.migration.cli
:resource-paths ["path/to/resources"]
:dependencies
[[com.github.igrishaev/pg2-core ...]]}}
Above, the migrations
profile has the dependency and the :main
attribute. Now run lein run
with migration arguments:
> lein with-profile +migrations run -c migration.config.edn migrate --to 100500
Here is an example of an alias in deps.edn
that prints pending migrations:
{:aliases
{:migrations-list
{:extra-deps
{com.github.igrishaev/pg2-migration {:mvn/version "..."}}
:extra-paths
["test/resources"]
:main-opts
["-m" "pg.migration.cli"
"-h" "127.0.0.1"
"-p" "10150"
"-u" "test"
"-w" "test"
"-d" "test"
"--table" "migrations_test"
"--path" "migrations"
"list"]}}}
Run it as follows:
> clj -M:migrations-list
You can shorten it by using the config file. Move all the parameters into the
migration.config.edn
file, and keep only a command with its sub-arguments in
the :main-opts
vector:
{:aliases
{:migrations-migrate
{:extra-deps
{com.github.igrishaev/pg2-migration {:mvn/version "..."}}
:extra-paths
["test/resources"]
:main-opts ["migrate" "--all"]}}}
To migrate:
> clj -M:migrations-migrate
There is a way to manage migrations through code. The pg.migration.core
namespace provides basic functions to list, create, migrate, and rollback
migrations.
To migrate, call one of the following functions: migrate-to
, migrate-all
,
and migrate-one
. All of them accept a config map:
(ns demo
(:require
[pg.migration.core :as mig]))
(def CONFIG
{:host "127.0.0.1"
:port 5432
:user "test"
:password "secret"
:database "test"
:migrations-table :test_migrations
:migrations-path "migrations"})
;; migrate all pinding migrations
(mig/migrate-all CONFIG)
;; migrate only one next migration
(mig/migrate-one CONFIG)
;; migrate to a certain migration
(mig/migrate-to CONFIG 20240313)
The same applies to rollback:
;; rollback all previously applied migrations
(mig/rollback-all CONFIG)
;; rollback the current migration
(mig/migrate-one CONFIG)
;; rollback to the given migration
(mig/rollback-to CONFIG 20230228)
The read-disk-migrations
function reads migrations from disk. It returns a
sorted map without information about whether migrations have been applied:
(mig/read-disk-migrations "migrations")
{1
{:id 1
:slug "create users"
:url-prev #object[java.net.URL "file:/.../migrations/001-create-users.prev.sql"]
:url-next #object[java.net.URL "file:/.../migrations/001-create-users.next.sql"]}
2
{:id 2
:slug "create profiles"
:url-prev #object[java.net.URL "file:/.../migrations/foobar/002-create-profiles.prev.sql"]
:url-next #object[java.net.URL "file:/.../migrations/foobar/002-create-profiles.next.sql"]}
...}
The make-scope
function accepts a config map and returns a scope map. The
scope map knows everything about the state of migrations, namely: which of them
have been applied, what is the current migration, the table name, the resource
path, and more.
The function create-migration-files
creates and returns a pair of empty SQL
files. By default, the id is generated from the current date & time, and the
slug is missing:
(create-migration-files "migrations")
[#object[java.io.File "migrations/20240313120122.prev.sql"]
#object[java.io.File "migrations/20240313120122.next.sql"]]
Pass id and slug in options if needed:
(create-migration-files "migrations" {:id 12345 :slug "Hello migration"})
[#object[java.io.File "migrations/12345.hello-migration.prev.sql"]
#object[java.io.File "migrations/12345.hello-migration.next.sql"]]
On bootstrap, the engine checks migrations for conflicts. A conflict is a situation when a migration with less id has been applied before a migration with greater id. Usually it happens when two developers create migrations in parallel and merge them in a wrong order. For example:
When you try to apply migration 20240315, the engine will check if 20240316 has already been applied. If yes, an exception pops up saying which migration cause the problem (in our case, these are 20240316 and 20240315). To recover from the conflict, rename 20240315 to 20240317.
In other words: this is a conflict:
id applied?
20240312 true
20240315 false
20240316 true ;; applied before 20240315
And this is a solution:
id applied?
20240312 true
20240316 true
20240317 false ;; 20240315 renamed to 20240317
A meandering path to a definition I can live with.
Code
Notes