Say Hello to Programming: Writing "Hello, World!" in 50 Different Languages

Description:
In this blog post, we'll explore the fascinating world of programming by learning how to write the classic "Hello, World!" program in 50 different programming languages. From the simplicity of Python to the elegance of Haskell, we'll take a brief tour through a diverse range of languages, each with its own unique syntax and approach to programming. Whether you're a seasoned developer looking to expand your repertoire or a curious beginner eager to dip your toes into the vast ocean of code, this journey promises to be both educational and entertaining.

Now, let's dive into writing "Hello, World!" in 50 different languages:

1 Python:

print("Hello, World!")

2 Java:

public class HelloWorld {
    public static void main(String[] args) {
        System out println("Hello, World!");
    }
}

3 C:

#include <stdio h>

int main() {
    printf("Hello, World!\n");
    return 0;
}

4 C++:

#include <iostream>

int main() {
    std::cout << "Hello, World!" << std::endl;
    return 0;
}

5 JavaScript:

console log("Hello, World!");

6 Ruby:

puts "Hello, World!"

7 Swift:

print("Hello, World!")

8 Go:

package main

import "fmt"

func main() {
    fmt Println("Hello, World!")
}

9 Rust:

fn main() {
    println!("Hello, World!");
}

10 PHP:

<?php
echo "Hello, World!";
?>

11 Perl:

print "Hello, World!\n";

12 Kotlin:

fun main() {
    println("Hello, World!")
}

13 Scala:

object HelloWorld {
  def main(args: Array[String]): Unit = {
    println("Hello, World!")
  }
}

14 Lua:

print("Hello, World!")

15 Haskell:

main :: IO ()
main = putStrLn "Hello, World!"

16 Dart:

void main() {
  print('Hello, World!');
}

17 Shell:

echo "Hello, World!"

18 Batch:

@echo off
echo Hello, World!

19 PowerShell:

Write-Output "Hello, World!"

20 VBScript:

MsgBox "Hello, World!"

21 Objective-C:

#import <Foundation/Foundation h>

int main(int argc, const char * argv[]) {
    @autoreleasepool {
        NSLog(@"Hello, World!");
    }
    return 0;
}

22 Assembly:

section  data
    hello db 'Hello, World!',10
    len equ $ - hello

section  text
    global _start

_start:
    ; write our string to stdout
    mov eax, 4         ; sys_write
    mov ebx, 1         ; file descriptor 1 (stdout)
    mov ecx, hello     ; message to write
    mov edx, len       ; message length
    int 0x80           ; syscall
    ; exit
    mov eax, 1         ; sys_exit
    xor ebx, ebx       ; exit status 0
    int 0x80           ; syscall

23 VBA (Visual Basic for Applications):

Sub HelloWorld()
    MsgBox "Hello, World!"
End Sub

24 Tcl:

puts "Hello, World!"

25 COBOL:

       IDENTIFICATION DIVISION 
       PROGRAM-ID  HELLO-WORLD 
       PROCEDURE DIVISION 
           DISPLAY "Hello, World!" 
           STOP RUN 

26 F#:

printfn "Hello, World!"

27 Elixir:

IO puts "Hello, World!"

28 SQL (MySQL):

SELECT 'Hello, World!';

29 SQL (SQLite):

SELECT 'Hello, World!';

30 SQL (PostgreSQL):

SELECT 'Hello, World!';

31 SQL (Oracle):

SELECT 'Hello, World!' FROM DUAL;

32 SQL (SQL Server):

PRINT 'Hello, World!';

33 Smalltalk:

Transcript show: 'Hello, World!'; cr 

34 R:

cat("Hello, World!\n")

35 Bash:

echo "Hello, World!"

36 Erlang:

-module(hello) 
-export([hello_world/0]) 

hello_world() ->
    io:fwrite("Hello, World!~n") 

37 Julia:

println("Hello, World!")

38 MATLAB:

disp('Hello, World!');

39 AutoHotkey:

MsgBox, Hello, World!

40 Clojure:

(println "Hello, World!")

41 Groovy:

println "Hello, World!"

42 OCaml:

print_endline "Hello, World!"

43 D:

import std stdio;

void main()
{
    writeln("Hello, World!");
}

44 Crystal:

puts "Hello, World!"

45 Nim:

echo "Hello, World!"

46 Common Lisp:

(format t "Hello, World!~%")

47 Scheme:

(display "Hello, World!")
(newline)

48 Prolog:

:- initialization(main) 

main :-
    write('Hello, World!'), nl,
    halt 

49 ABAP:

REPORT ZHELLO_WORLD 

WRITE: / 'Hello, World!' 

50 VB NET:

vb net
Module HelloWorld
    Sub Main()
        Console WriteLine("Hello, World!")
    End Sub
End Module 

That concludes our journey through "Hello, World!" in 50 different programming languages Whether you're just starting or have been programming for years, exploring different

Permalink

What the Reagent Component?!

Did you know that when you write a form-1, form-2 or form-3 Reagent component they all default to becoming React class components?

For example, if you were to write this form-1 Reagent component:

(defn welcome []
  [:h1 "Hello, friend"])

By the time Reagent passes it to React it would be the equivalent of you writing this:

class Welcome extends React.Component {
  render() {
    return <h1>Hello, friend</h1>
  }
}

Okay, so, Reagent components become React Class Components. Why do we care? This depth of understanding is valuable because it means we can better understand:

The result of all of this "fundamental" learning is we can more effectively harness JavaScript from within ClojureScript.

A Pseudoclassical Pattern

The reason all of your Reagent components become class components is because all of the code you pass to Reagent is run through an internal Reagent function called create-class.

create-class is interesting because of how it uses JavaScript to transform a Reagent component into something that is recognized as a React class component. Before we look into what create-class is doing, it's helpful to review how "classes" work in JavaScript.

Prior to ES6, JavaScript did not have classes. and this made some JS developers sad because classes are a common pattern used to structure code and provide support for:

  • instantiation
  • inheritance
  • polymorphism

But as I said, prior to ES6, JavaScript didn't have a formal syntax for "classes". To compensate for the lack of classes, the JavaScript community got creative and developed a series of instantiation patterns to help simulate classes.

Of all of these patterns, the pseudoclassical instantiation pattern became one of the most popular ways to simulate a class in JavaScript. This is evidenced by the fact that many of the "first generation" JavaScript libraries and frameworks, like google closure library and backbone, are written in this style.

The reason we are going over this history is because the thing about a programming language is there are "patterns" and "syntax". The challenge with "patterns" is:

  • They're disseminated culturally (tribal knowledge)
  • They're difficult to identify
  • They're often difficult to search
  • They often require a deeper knowledge to understand how and why to use a pattern.

The last point in praticular is relevant to our conversation because patterns live in a context and assume prior knowledge. Knowledge like how well we know the context of a problem, the alternative approaches to addressing a problem, advancements in a language and so on.

The end result is that a pattern can just become a thing we do. We can forget or never know why it started in the first place or what the world could look like if we chose a different path.

For example, the most common way of writing a React class component is to use ES6 class syntax. But did you know that ES6 class syntax is little more than syntactic sugar around the pseudoclassical instantiation pattern?

For example, you can write a valid React class component using the pseudoclassical instantiation pattern like this:

// 1. define a function (component) called `Welcome`
function Welcome(props, context, updater) {
  React.Component.call(this, props, context, updater)

  return this
}

// 2. connect `Welcome` to the `React.Component` prototype
Welcome.prototype = Object.create(React.Component.prototype)

// 3. re-define the `constructor`
Object.defineProperty(Welcome.prototype, 'constructor', {
  enumerable: false,
  writable: true,
  configurable: true,
  value: Welcome,
})

// 4. define your React components `render` method
Welcome.prototype.render = function render() {
  return <h2>Hello, Reagent</h2>
}

While the above is a valid React Class Component, it's also verbose and error prone. For these reasons JavaScript introduced ES6 classes to the language:

class Welcome extends React.Component {
  render() {
    return <h1>Hello, Reagent</h1>
  }
}

For those looking for further evidence, we can support our claim that ES6 Classes result in same thing as what the pseudoclassical instantiation pattern produces by using JavaScript's built-in introspection tools to compare the pseudoclassical instantiation pattern to the ES6 class syntax.

pseudoclassical instantiation pattern:

function Welcome(props, context, updater) {
  React.Component.call(this, props, context, updater)

  return this
}

// ...repeat steps 2 - 4 from above before completing the rest

var welcome = new Welcome()

Welcome.prototype instanceof React.Component
// => true

Object.getPrototypeOf(Welcome.prototype) === React.Component.prototype
// => true

welcome instanceof React.Component
// => true

welcome instanceof Welcome
// => true

Object.getPrototypeOf(welcome) === Welcome.prototype
// => true

React.Component.prototype.isPrototypeOf(welcome)
// => true

Welcome.prototype.isPrototypeOf(welcome)
// => true

ES6 class

class Welcome extends React.Component {
  render() {
    console.log('ES6 Inheritance')
  }
}

var welcome = new Welcome()

Welcome.prototype instanceof React.Component
// => true

Object.getPrototypeOf(Welcome.prototype) === React.Component.prototype
// => true

welcome instanceof React.Component
// => true

welcome instanceof Welcome
// => true

Object.getPrototypeOf(welcome) === Welcome.prototype
// => true

React.Component.prototype.isPrototypeOf(welcome)
// => true

Welcome.prototype.isPrototypeOf(welcome)
// => true

What does all of this mean? As far as JavaScript and React are concerned, both definions of the Welcome component are valid React Class Components.

With this in mind, lets look at Reagent's create-class function and see what it does.

What Reagent Does

The history lesson from the above section is important because create-class uses a modified version of the pseudoclassical instantiation pattern. Let's take a look at what we mean.

The following code sample is a simplified version of Reagent's create-class function:

function cmp(props, context, updater) {
  React.Component.call(this, props, context, updater)

  return this
}

goog.extend(cmp.prototype, React.Component.prototype, classMethods)

goog.extend(cmp, React.Component, staticMethods)

cmp.prototype.constructor = cmp

What we have above is Reagents take on the pseudoclassical instantiation pattern with a few minor tweaks:

// 1. we copy to properties + methods of React.Component
goog.extend(cmp.prototype, React.Component.prototype, classMethods)

goog.extend(cmp, React.Component, staticMethods)

// 2. the constructor is not as "thorough"
cmp.prototype.constructor = cmp

Exploring point 1 we see that Reagent has opted to copy the properties and methods of React.Component directly to the Reagent compnents we write. That is what's happening here:

goog.extend(cmp.prototype, React.Component.prototype, classMethods)

If we were using the the traditional pseudoclassical approach we would instead do this:

cmp.prototype = Object.create(React.Component.prototype)

Thus, the difference is that Reagent's approach copies all the methods and properties from React.Component to the cmp prototype where as the second approach is going to link the cmp prototype to React.component prototype. The benefit of linking is that each time you instantiate a Welcome component, the Welcome component does not need to re-create all of the React.components methods and properties.

Exploring the second point, Reagent is doing this:

cmp.prototype.constructor = cmp

whereas with the traditional pseudoclassical approach we would instead do this:

Object.defineProperty(Welcome.prototype, 'constructor', {
  enumerable: false,
  writable: true,
  configurable: true,
  value: Welcome,
})

The difference in the above approaches is that if we just use = as we are doing in the Reagent version we create an enumerable constructor. This can have an implication depending on who consumes our classes, but in our case we know that only React is going to be consuming our class components, so we can do this with relative confidence.

What is one of the more interesting results of the above two Reagent modifications? First, if React depended on JavaScript introspection to tell whether or not a component is a child of React.Component we would not be happy campers:

Welcome.prototype instanceof React.Component
// => false...Welcome is not a child of React.Component

Object.getPrototypeOf(Welcome.prototype) === React.Component.prototype
// => false...React.component is not part of Welcomes prototype chain

welcome instanceof React.Component
// => false...Welcome is not an instance of React.Component

welcome instanceof Welcome
// => true...welcome is a child of Welcome

Object.getPrototypeOf(welcome) === Welcome.prototype
// => true...welcome is linked to Welcome prototype

console.log(React.Component.prototype.isPrototypeOf(welcome))
// => false...React.Component not linked to the prototype of React.Component

console.log(Welcome.prototype.isPrototypeOf(welcome))
// is Welcome is the ancestory?

What the above shows is that Welcome is not a child of React.component even though it has all the properties and methods that React.Component has. This is why were lucky that React is smart about detecting class vs. function components.

Second, by copying rather than linking prototypes we could inccur a performance cost. How much of a performance hit? In our case this cost is likely negligible.

Conclusion

In my experience, digging into the weeds and going on these detours has been an important part of my growth as a developer. The weeds have allowed me to be a better programmer because I'm honing my ability to understand challenging topics and find answers. The result is a strange feeling of calm and comfort.

This calm and comfort shouldn't be overlooked. So much of our day-to-day is left unquestioned and unanalyzed. We let knowledge become "cultural" or "tribal". This is scary. It's scary because it leads to bad decisions because no one around us knows the whys or wherefores. Ultimately, it's a bad habit. A bad habit which is seen by some as a virtue because it would simply take too much time for to learn things ourselves. That's until you actually start doing this kind of work and spend time learning and observing and seeing that these "new things" we're seeing all the time aren't really new, but just another example of that old thing back.

Permalink

What are the Clojure Tools?

The Clojure Tools are a group of convenience tools which currently consist of:

  • Clojure CLI
  • tools.build

The Clojure Tools. were designed to answer some of the following questions:

  • How do I install Clojure? (Clojure CLI)
  • How do I run a Clojure program? (Clojure CLI)
  • How do I manage Clojure packages (dependencies)? (Clojure CLI)
  • How do I configure a Clojure project? (Clojure CLI)
  • How do I build Clojure for production? (tools.build)

The rest of this post will dig into each of these tools.

Clojure CLI

The Clojure CLI is a CLI program. Here is what it looks like to use the Clojure CLI and some of the things it can do:

Run a Clojure repl

clj

Run a Clojure program

clj -M -m your-clojure-program

manage Clojure dependencies

clj -Sdeps '{:deps {bidi/bidi {:mvn/version "2.1.6"}}}'

Like all Clojure programs, the Clojure CLI is built on a few libraries:

The following sections will provide overviews of each of the above tools.

The Clojure CLI is invoked by calling either clj or clojure shell commands:

# clj
clj -M -m your-clojure-program

# clojure
clojure -M -m your-clojure-program

Under the hood, clj actually calls clojure. The difference is that clj wraps the clojure command with a tool called rlwrap. rlwrap improves the developer experience by making it easier for you, a human, to type in the terminal while you're running your Clojure REPL. However, even though it's easier for you to type, rlwrap can make it hard to compose the clj command with other tools. As a result, it's a common practice to use clojure in production/ci environments . Additionally, not all environments have access to rlwrap so it's another dependency you have to install.

Okay, so they do the same thing. What do they do? clj/clojure has one job: run Clojure programs against a classpath.

The next sections will outline the tools that make up the Clojure CLI tool.

clj/clojure

If you dig into the clj/clojure is just a bash script which ultimatley calls a command like this:

java [java-opt*] -cp classpath clojure.main [init-opt*] [main-opt] [arg*]

Thus, the Clojure CLI tool makes it easier to run Clojure programs. It saves you having to type out a gnarly Java command and make it work on different environments (windows, linux, mac etc). However, it orchestrates the building of the classpath by calling out to tools.deps.

tools.deps

tools.deps is a Clojure libary responsible for managing your dependencies. It does the following things:

  • reads in dependencies from a deps.edn file
  • resolves the dependencies and their transitive dependencies
  • builds a classpath

What's interesting about this program is that it's just a Clojure library. This means that you can use it outside of the Clojure CLI.

The other thing that makes tools.deps great is that it's a small and focused library. Why this is great is that if something goes wrong it's easy to read and learn the library in a short period of time.

deps.edn

deps.edn is just an edn file where you configure your project and specify project dependencies. You can think of it like Clojure's version of package.json. The deps.edn file is a Clojure map with a specific structure. Here's an example of some of the properties of a deps.edn file:

{:deps    {...}
 :paths   [...]
 :aliases {...}}

As you can see, we use the keywords :deps, :paths and :aliases and more to start to describe your project and the dependencies it requires.

As we noted above, deps.edn is read in when you run clj/clojure and tells clj/clojure which dependencies are requires to run your project.

Tools.Build

tools.build is a Clojure library with functions for building clojure projects. For example, build a jar or uberjar.

The way you would use tools.build is by writing a separate program inside your app which knows how to build your app. The convention is to create a build.clj file in the root of your project. Import tools.build and use the functions provides by tools.build to build your program.

The 3 main types of Clojure programs one might build into 3 sub categories:

  • A tool
  • A library
  • An app

When you run your build.clj file, you will use Clojure CLI's -T switch. The -T switch is meant to run general clojure programs via the Clojure CLI and since build.clj is a separate program, distinct form the app you are writing, you would run it via the -T switch.

You would use -T for Clojure programs that you want to run as a "tool". For example, deps-new is a Clojure library which creates new Clojure projects based on a template you provide. This is a great example of a Clojure project which is built to be a "tool".

I don't want to go into more detail about -T now because that means we would have to dive into other Clojure CLI switches like -X and -M. That's for another post. On to the Installer!

Installer

The "Clojure CLI Installer" is a fancy way of referring to the brew tap used to install Clojure on mac and linux machines. As of February 2020, Clojure started maintaining their own brew tap. Thus, if you installed the Clojure CLI via

brew install clojure

you will likely want to uninstall clojure and install the following:

brew install clojure/tools/clojure

In all likelihood, you would probably be fine with brew install clojure as it will recieve updates. However, while brew install clojure will still see some love, it won't be as active as the clojure/tools/clojure tap.

clj v lein v boot

This section will provide a quick comparison of clj, lein and boot.

Firstly, all of the above tools are more or less addressing the same problems in their own way. Your job is to choose the one you like best.

If you're curious which to choose, my answer is the Clojure CLI. The reason I like the Clojure CLI is because the tool is simple. You can read through clj and tools.deps in an afternoon and understand what they are doing. The same (subjectively of course) cannot be said for lein or boot. I will note that Clojure CLI's API is not straightforward and can be confusing.

Secondly, the Clojure Tools promote libraries over frameworks. This is important when working with a language like Clojure because it really does reward you for breaking down your thinking.

Finally, the Clojure community is really leaning into building tools for Clojure CLI. For example, where lein used to have significantly more functionality, the community has built a ton of incredible tools that will cover many of your essential requirements.

Permalink

Getting an RSS feed from any website

© Rob McDonald

RSS is a trendy topic among tech circles, it’s coming back from time to time without a breakthrough but still, it’s a technology used. Unfortunately, a lot of websites don’t offer RSS feeds anymore.

Big web players like Facebook, Twitter, and LinkedIn are promoting web silos. They want us to stay on their website as long as possible. They design their news feed for that purpose and not for the user best interest.

Thanks to WordPress and the technology entropy millions of websites still offer a RSS feed. What to do for websites that don’t?

The beginning of super-rss.

I built it with remus, a library for parsing RSS feed, and added functions giving similar output using alternative mechanisms to build the feed.

I initially started the project with the idea that when a website does not offer a RSS feed, it is still most likely offering a sitemap to be easily accessible by search engines. Websites still want their content to be found by Google, right?

To my surprise, that often does not work. Some websites have no sitemap at all (I guess they are not doing really well on SEO…), or the file is broken for various reasons: the XML is invalid, the date is in the wrong format or missing, the sitemap has thousands or URLs with no grouping of any sort… I get some feeds from that but globally that often did not allow me to create a news feed and that requires a lot of resources to crawl the URL and see what they are about. I’m running a side project on a single machine, so I don’t want to set up a piece of heavy machinery [1] to basically create an XML summary of a webpage.

A second quick solution was to collect all the links of a webpage. It was super easy to implement, thanks to enlive. The algorithm:

  • Fetch the page
  • Use enlive to get all the anchors with href
  • Filter the result for links that look obviously wrong
  • Crawl each link to figure out the title description and date (this part was not great: resource intensive and lackluster results)

Current solution: what you see is what you get

Since the previous solution, I have learned about zipper. This is an extremely powerful library to recursively parse a tree in a state machine. It was part of Clojure 1.0 and was initially designed by Gérard Huet. The library is somehow not well documented and known (?). Here are some good materials to see how to use it:

Even if it’s not a well-known library, many tools of the Clojure day-to-day developer use it through rewrite-clj

Since an HTML page is a tree, that seems like a perfect match! I just need the links that look legit and “look around” to extract the feed entry.

Here is the code.

The simplified algorithm for the “looking around” part:

  • For each link on the page
  • Look up from 2 to 4 levels, the first level that finds at least 2 blocks is probably a list of links or feed entries, return these blocks as entry feed
  • process the links until they all have been visited.

Visually it goes like this:

1 level up1 level up
2 level up: bingo this is a list of links

The block parsing consists of finding the most likely link to the article, the title, the optional description, and the optional date. Code here.

Basically, the algo searches a few levels up the link itself, assuming that the title/description/date are located in a nearby HTML element. 4 levels and 2 children are numbers I found by experimentation.

It seems to work okay on the few websites I added to my news reader. Let’s hope it is 🤞🏻

The state of information systems on the open Internet:

What it should be: Database → News feed
What it is (yolo): Database → … → HTML → educated guess extraction
What a waste of CPU cycle 🤦

Of course, it is not the quality of a real RSS feed: information is extracted on educated guesses where somewhere they are lying properly organized in a relational database. It’s a waste of resources compared to the time were RSS feeds were a hype technology, but a good workaround to create my own news feed instead of consuming the dubious one built by the big tech company.

[1] Something you can run and operate as a single individual, or small team. Not something requiring an elephant cluster or hundreds of machines.

Permalink

Clojure Deref (Mar 22, 2024)

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem (feed: RSS). Thanks to Anton Fonarev for link aggregation.

Libraries and Tools

New releases and tools this week:

  • muuntaja 0.6.10 - Clojure library for fast http api format negotiation, encoding and decoding

  • next-jdbc 1.3.925 - A modern low-level Clojure wrapper for JDBC-based access to databases

  • flow-storm-debugger 3.13.1 - A debugger for Clojure and ClojureScript with some unique features

  • contajners 1.0.4 - An idiomatic, data-driven, REPL friendly clojure client for OCI container engines

  • typedclojure 1.2.1 - An optional type system for Clojure

  • clay 2-alpha87 - A tiny Clojure tool for dynamic workflow of data visualization and literate programming

  • calva 2.0.425 - Clojure & ClojureScript Interactive Programming for VS Code

  • metamorph.ml 0.7.8 - Machine learning functions for metamorph based on machine learning pipelines. Part of scicloj.ml

  • trace 1.2 - Better REPL based debugging output

  • overarch 0.13.0 - A data driven description of software architecture based on UML and the C4 model

Permalink

Catwalk and Expo

Ok, so I've been doing more intermittent work on catwalk. There's a bunch of other stuff happening in life, so the project isn't going exactly as well as I wanted, but I'm making progress.

Basic Server Progress

As of this writing, the latest updates to the catwalk server have to do with adding CORS headers, and the next couple are going to have to do with removing the blacklist system and adding a fairly general, API-key based user system. The medium term plan is to get a bit more elaborate and allow external sign-ups in case someone else wants to use this to make their own blogs into podcasts.

Web Client Progress

The immediate next steps are all going to have to do with making the client a lot more performant. The problem is that, for some reason, the blogcast page works really poorly. In particular, when I go to delete certain lines, or edit and re-record them, there's frequent, massive UI pauses that prevent me from moving around the page. There's also some issues I'm still working through regarding keyboard navigation, although as you'll understand in a moment, that's going to be less relevant.

I pushed a few commits recently that I hoped would make the situation more bearable, and let me catch up with my casting work, but they did no such thing. Instead slowing the client to even more of a crawl. I'm not entirely sure why this is happening, and it's entirely possible that it's a direct result of me setting up the state graph for this application in a moronic way. Still, I was hoping to get half-way decent performance out of it and didn't.

Mobile Client Progress

You probably didn't even realize I was trying for this. Ok, maybe that series of android pokes tipped my hand, but I don't think I explicitly mentioned what the goal is. What I'd really like with Catwalk is for it to become a web-and-or-mobile-based blogcast tool, not just a web one. I recognize that I could trivially accomplish both, in a sense, by making the web side of it reactive, but it's also sometimes nice to have a native app when you're dealing with a phone screen.

I'd been trying to figure out a way to get both of these things at once, in a nice, enjoyable clojure context. This doesn't seem to be in the cards, unfortunately. Every attempt I've made to get a react-native-based CLJS project off the ground has ended up with me hitting roadblocks in the form of stale instructions and/or tooling. The latest one I've tried out is Expo. Which on its' own looks pretty snazzy, builds trivially on Android, seems to allow for separate web-deployment, and has a really nice experimental remote REPL on-device.

If I have to code in Javascript directly in order to use this, then so be it.

Expo

I've tried out some basics, and no big complaints so far. It looks like there are going to be some finer points regarding what elements specifically to use for native/web display, in particular SectionList vs ScrollView with a map call has significantly different interactions, but no big roadblocks. Setting it up is as easy as

~$ yarn create expo-cljs-app your-app-name
...
~$ cd your-app-name
~/your-app-name$ yarn start

It then displays a QR code you can scan from the Anrdoid app to start an interactively updating dev preview of your application and go from there. Getting the web client out of it is then as easy as

~/your-app-name$ yarn expo export -p web

It looks like getting the android version is going to involve changing the value passed to that -p flag, and also setting up an eas.json file and doing some more stuff through EAS. I only skimmed the intro, but it doesn't look too intimidating.

As always, I'll let you know how it goes.

Permalink

New Library: clj-reload

The problem

Do you love interactive development? Although Clojure is set up perfectly for that, evaluating buffers one at a time can only get you so far.

Once you start dealing with the state, you get data dependencies, and with them, evaluation order starts to matter, and now you change one line but have to re-eval half of your application to see the change.

But how do you know which half?

The solution

Clj-reload to the rescue!

Clj-reload scans your source dir, figures out the dependencies, tracks file modification times, and when you are finally ready to reload, it carefully unloads and loads back only the namespaces that you touched and the ones that depend on those. In the correct dependency order, too.

Let’s do a simple example.

a.clj:

(ns a
  (:require b))

b.clj:

(ns b
  (:require c))

c.clj:

(ns c)

Imagine you change something in b.clj and want to see these changes in your current REPL. What do you do?

If you call

(clj-reload.core/reload)

it will notice that

  • b.clj was changed,
  • a.clj depends on b.clj,
  • there’s c.clj but it doesn’t depend on a.clj or b.clj and wasn’t changed.

Then the following will happen:

Unloading a
Unloading b
Loading b
Loading a

So:

  • c wasn’t touched — no reason to,
  • b was reloaded because it was changed,
  • a was loaded after the new version of b was in place. Any dependencies a had will now point to the new versions of b.

That’s the core proposition of clj-reload.

Usage

Here, I recorded a short video:

But if you prefer text, then start with:

(require '[clj-reload.core :as reload])

(reload/init
  {:dirs ["src" "dev" "test"]})

:dirs are relative to the working directory.

Use:

(reload/reload)
; => {:unloaded [a b c], :loaded [c b a]}

reload can be called multiple times. If reload fails, fix the error and call reload again.

Works best if assigned to a shortcut in your editor.

Usage: Return value

reload returns a map of namespaces that were reloaded:

{:unloaded [<symbol> ...]
 :loaded   [<symbol> ...]}

By default, reload throws if it can’t load a namespace. You can change it to return exception instead:

(reload/reload {:throw false})

; => {:unloaded  [a b c]
;     :loaded    [c b]
;     :failed    b
;     :exception <Throwable>}

Usage: Choose what to reload

By default, clj-reload will only reload namespaces that were both:

  • Already loaded
  • Changed on disk

If you pass :only :loaded option to reload, it will reload all currently loaded namespaces, no matter if they were changed or not.

If you pass :only :all option to reload, it will reload all namespaces it can find in the specified :dirs, no matter whether loaded or changed.

Usage: Skipping reload

Some namespaces contain state you always want to persist between reloads. E.g. running web-server, UI window, etc. To prevent these namespaces from reloading, add them to :no-reload during init:

(reload/init
  {:dirs ...
   :no-reload '#{user myapp.state ...}})

Usage: Unload hooks

Sometimes your namespace contains stateful resource that requires proper shutdown before unloading. For example, if you have a running web server defined in a namespace and you unload that namespace, it will just keep running in the background.

To work around that, define an unload hook:

(def my-server
  (server/start app {:port 8080}))

(defn before-ns-unload []
  (server/stop my-server))

before-ns-unload is the default name for the unload hook. If a function with that name exists in a namespace, it will be called before unloading.

You can change the name (or set it to nil) during init:

(reload/init
  {:dirs [...]
   :unload-hook 'my-unload})

This is a huge improvement over tools.namespace. tools.namespace doesn’t report which namespaces it’s going to reload, so your only option is to stop everything before reload and start everything after, no matter what actually changed.

Usage: Keeping vars between reloads

One of the main innovations of clj-reload is that it can keep selected variables between reloads.

To do so, just add ^:clj-reload/keep to the form:

(ns test)

(defonce x
  (rand-int 1000))

^:clj-reload/keep
(def y
  (rand-int 1000))

^:clj-reload/keep
(defrecord Z [])

and then reload:

(let [x test/x
      y test/y
      z (test/->Z)]
  
  (reload/reload)
  
  (let [x' test/x
        y' test/y
        z' (test/->Z)]
    (is (= x x'))
    (is (= y y'))
    (is (identical? (class z) (class z')))))

Here’s how it works:

  • defonce works out of the box. No need to do anything.
  • def/defn/deftype/defrecord/defprotocol can be annotated with ^:clj-reload/keep and can be persistet too.
  • Project-specific forms can be added by extending clj-reload.core/keep-methods multimethod.

Why is this important? With tools.namespace you will structure your code in a way that will work with its reload implementation. For example, you’d probably move persistent state and protocols into separate namespaces, not because logic dictates it, but because reload library will not work otherwise.

clj-reload allows you to structure the code the way business logic dictates it, without the need to adapt to developer workflow.

Simply put: the fact that you use clj-reload during development does not spill into your production code.

Comparison: Evaluating buffer

The simplest way to reload Clojure code is just re-evaluating an entire buffer.

It works for simple cases but fails to account for dependencies. If something depends on your buffer, it won’t see these changes.

The second pitfall is removing/renaming vars or functions. If you had:

(def a 1)

(def b (+ a 1))

and then change it to just

(def b (+ a 1))

it will still compile! New code is evaluated “on top” of the old one, without unloading the old one first. The definition of a will persist in the namespace and let b compile.

It might be really hard to spot these errors during long development sessions.

Comparison: (require ... :reload-all)

Clojure has :reload and :reload-all options for require. They do track upstream dependencies, but that’s about it.

In our original example, if we do

(require 'a :reload-all)

it will load both b and c. This is excessive (b or c might not have changed), doesn’t keep track of downstream dependencies (if we reload b, it will not trigger a, only c) and it also “evals on top”, same as with buffer eval.

Comparison: tools.namespace

tools.namespace is a tool originally written by Stuart Sierra to work around the same problems. It’s a fantastic tool and the main inspiration for clj-reload. I’ve been using it for years and loving it, until I realized I wanted more.

So the main proposition of both tools.namespace and clj-reload is the same: they will track file modification times and reload namespaces in the correct topological order.

This is how clj-reload is different:

  • tools.namespace reloads every namespace it can find. clj-reload only reloads the ones that were already loaded. This allows you to have broken/experimental/auxiliary files lie around without breaking your workflow TNS-65
  • First reload in tools.namespace always reloads everything. In clj-reload, even the very first reload only reloads files that were actually changed TNS-62
  • clj-reload supports namespaces split across multiple files (like core_deftype.clj, core_defprint.clj in Clojure) TNS-64
  • clj-reload can see dependencies in top-level standalone require and use forms TNS-64
  • clj-reload supports load and unload hooks per namespace TNS-63
  • clj-reload can specify exclusions during configuration, without polluting the source code of those namespaces.
  • clj-reload can keep individual vars around and restore previous values after reload. E.g. defonce doesn’t really work with tools.namespace, but it does with clj-reload.
  • clj-reload has 2× smaller codebase and 0 runtime dependencies.
  • clj-reload doesn’t support ClojureScript. Patches welcome.

That’s it!

Clj-reload grew from my personal needs on Humble UI project. But I hope other people will find it useful, too.

Let me know what works for you and what doesn’t! I’ll try to at least be on par with tools.namespace.

And of course, here’s the link:

Permalink

PG2 release 0.1.6: rich JSON capabilities

PG2 version 0.1.6 is out, and it ships various improvements to JSON(b) handling.

Table of Content

Postgres is amazing when dealing with JSON. There hardly can be a database that serves it better. Unfortunately, Postgres clients never respect the JSON feature, which is horrible. Take JDBC, for example: when querying a JSON(b) value, you’ll get a dull PGObject which should be decoded manually. The same applies to insertion: one cannot just pass a Clojure map or a vector. It should be packed into the PGObject as well.

Of course, this can be automated by extending certain protocols. But it’s still slow as it’s done on Clojure level (not Java), and it forces you to copy the same code across projects.

Fortunately, PG2 supports JSON out from the box. If you query a JSON value, you’ll get its Clojure counter-part: a map, a vector, etc. To insert a JSON value to a table, you pass either a Clojure map or a vector. No additional steps are required.

PG2 relies on jsonista library to handle JSON. At the moment of writing, this is the fastest JSON library for Clojure. Jsonista uses a concept of object mappers: objects holding custom rules to encode and decode values. You can compose your own mapper with custom rules and pass it into the connection config.

Basic usage

Let’s prepare a connection and a test table with a jsonb column:

(def config
  {:host "127.0.0.1"
   :port 10140
   :user "test"
   :password "test"
   :dbname "test"})

(def conn
  (jdbc/get-connection config))

(pg/query conn "create table test_json (
  id serial primary key,
  data jsonb not null
)")

Now insert a row:

(pg/execute conn
            "insert into test_json (data) values ($1)"
            {:params [{:some {:nested {:json 42}}}]})

No need to encode a map manually nor wrap it into a sort of PGObject. Let’s fetch the new row by id:

(pg/execute conn
            "select * from test_json where id = $1"
            {:params [1]
             :first? true})

{:id 1 :data {:some {:nested {:json 42}}}}

Again, the JSON data returns as a Clojure map with no wrappers.

When using JSON with HoneySQL though, some circs are still needed. Namely, you have to wrap a value with [:lift ...] as follows:

(pgh/insert-one conn
                :test_json
                {:data [:lift {:another {:json {:value [1 2 3]}}}]})

{:id 2, :data {:another {:json {:value [1 2 3]}}}}

Without the [:lift ...] tag, HoneySQL will treat the value as a nested SQL map and try to render it as a string, which will fail of course or lead to a SQL injection.

Another way is to use HoneySQL parameters conception:

(pgh/insert-one conn
                :test_json
                {:data [:param :data]}
                {:honey {:params {:data {:some [:json {:map [1 2 3]}]}}}})

For details, see the “HoneySQL Integration” section.

PG2 supports not only Clojure maps but vectors, sets, and lists. Here is an example with with a vector:

(pg/execute conn
            "insert into test_json (data) values ($1)"
            {:params [[:some :vector [:nested :vector]]]})

{:id 3, :data ["some" "vector" ["nested" "vector"]]}

Json Wrapper

In rare cases you might store a string or a number in a JSON field. Say, 123 is a valid JSON value but it’s treated as a number. To tell Postgres it’s a JSON indeed, wrap the value with pg/json-wrap:

(pgh/insert-one conn
                :test_json
                {:data (pg/json-wrap 42)})

{:id 4, :data 42}

The wrapper is especially useful to store a “null” JSON value: not the standard NULL but "null" which, when parsed, becomes nil. For this, pass (pg/json-wrap nil) as follows:

(pgh/insert-one conn
                :test_json
                {:data (pg/json-wrap nil)})

{:id 5, :data nil} ;; "null" in the database

Custom Object Mapper

One great thing about Jsonista is a conception of mapper objects. A mapper is a set of rules how to encode and decode data. Jsonista provides a way to build a custom mapper. Once built, it can be passed to a connection config so the JSON data is written and read back in a special way.

Let’s assume you’re going to tag JSON sub-parts to track their types. For example, if encoding a keyword :foo, you’ll get a vector of ["!kw", "foo"]. When decoding that vector, by the "!kw" string, the mapper understands it a keyword and coerces "foo" to :foo.

Here is how you create a mapper with Jsonista:


(ns ...
  (:import
   clojure.lang.Keyword
   clojure.lang.PersistentHashSet)
  (:require
    [jsonista.core :as j]
    [jsonista.tagged :as jt]))

(def tagged-mapper
  (j/object-mapper
   {:encode-key-fn true
    :decode-key-fn true
    :modules
    [(jt/module
      {:handlers
       {Keyword {:tag "!kw"
                 :encode jt/encode-keyword
                 :decode keyword}
        PersistentHashSet {:tag "!set"
                           :encode jt/encode-collection
                           :decode set}}})]}))

The object-mapper function accepts even more options but we skip them for now.

Now that you have a mapper, pass it into a config:

(def config
  {:host "127.0.0.1"
   :port 10140
   :user "test"
   :password "test"
   :dbname "test"
   :object-mapper tagged-mapper})

(def conn
  (jdbc/get-connection config))

All the JSON operations made by this connection will use the passed object mapper. Let’s insert a set of keywords:

(pg/execute conn
            "insert into test_json (data) values ($1)"
            {:params [{:object #{:foo :bar :baz}}]})

When read back, the JSON value is not a vector of strings any longer but a set of keywords:

(pg/execute conn "select * from test_json")

[{:id 1, :data {:object #{:baz :bar :foo}}}]

To peek a raw JSON value, select it as a plain text and print (just to avoid escaping quotes):

(printl (pg/execute conn "select data::text json_raw from test_json where id = 10"))

;; [{:json_raw {"object": ["!set", [["!kw", "baz"], ["!kw", "bar"], ["!kw", "foo"]]]}}]

If you read that row using another connection with a default object mapper, the data is returned without expanding tags.

Utility pg.json namespace

PG2 provides an utility namespace for JSON encoding and decoding. You can use it for files, HTTP API, etc. If you already have PG2 in the project, there is no need to plug in Cheshire or another JSON library. The namespace is pg.json:

(ns ...
  (:require
   [pg.json :as json]))

Reading JSON

The read-string function reads a value from a JSON string:

(json/read-string "[1, 2, 3]")

[1 2 3]

The first argument might be an object mapper:

(json/read-string tagged-mapper "[\"!kw\", \"hello\"]")

:hello

The functions read-stream and read-reader act the same but accept either an InputStream or a Reader object:

(let [in (-> "[1, 2, 3]" .getBytes io/input-stream)]
  (json/read-stream tagged-mapper in))

(let [in (-> "[1, 2, 3]" .getBytes io/reader)]
  (json/read-reader tagged-mapper in))

Writing JSON

The write-string function dumps an value into a JSON string:

(json/write-string {:test [:hello 1 true]})

;; "{\"test\":[\"hello\",1,true]}"

The first argument might be a custom object mapper. Let’s reuse our tagger mapper:

(json/write-string tagged-mapper {:test [:hello 1 true]})

;; "{\"test\":[[\"!kw\",\"hello\"],1,true]}"

The functions write-stream and write-writer act the same. The only difference is, they accept either an OutputStream or Writer objects. The first argument might be a mapper as well:

(let [out (new ByteArrayOutputStream)]
  (json/write-stream tagged-mapper {:foo [:a :b :c]} out))

(let [out (new StringWriter)]
  (json/write-writer tagged-mapper {:foo [:a :b :c]} out))

Ring HTTP middleware

PG2 provides an HTTP Ring middleware for JSON. It acts like wrap-json-request and wrap-json-response middleware from the ring-json library. Comparing to it, the PG2 stuff has the following advantages:

  • it’s faster because of Jsonista, whereas Ring-json relies on Cheshire;
  • it wraps both request and response at once with a shortcut;
  • it supports custom object mappers.

Imagine you have a Ring handler that reads JSON body and returns a JSON map. Something like this:

(defn api-handler [request]
  (let [user-id (-> request :data :user_id)
        user (get-user-by-id user-id)]
    {:status 200
     :body {:user user}}))

Here is how you wrap it:

(ns ...
  (:require
   [pg.ring.json :refer [wrap-json
                         wrap-json-response
                         wrap-json-request]]))

(def app
  (-> api-handler
      (wrap-this-foo)
      (wrap-json <opt>)
      (wrap-that-bar)))

Above, the wrap-json wrapper is a combination of wrap-json-request and wrap-json-response. You can apply them both explicitly:

(def app
  (-> api-handler
      (wrap-this-foo)
      (wrap-json-request <opt>)
      (wrap-json-response <opt>)
      (wrap-that-bar)))

All the three wrap-json... middleware accept a handler to wrap and a map of options. Here is the options supported:

Name Direction Description
:object-mapper request, response An custom instance of ObjectMapper
:slot request A field to assoc the parsed JSON data (1)
:malformed-response request A ring response returned when payload cannot be parsed (2)

Notes:

  1. The default slot name is :json. Please avoid using :body or :params to prevent overriding existing request fields. This is especially important for :body! Often, you need the origin input stream to calculate an MD5 or SHA-256 hash-sum of the payload. If you overwrite the :body field, you cannot do that.

  2. The default malformed response is something like 400 “Malformed JSON” (plain text).

A full example:

(def json-opt
  {:slot :data
   :object-mapper tagged-mapper ;; see above
   :malformed-response {:status 404
                        :body "<h1>Bad JSON</h1>"
                        :headers {"content-type" "text/html"}}})

(def app
  (-> api-handler
      (wrap-this-foo)
      (wrap-json json-opt)
      (wrap-that-bar)))

Permalink

Clojure Deref (Mar 15, 2024)

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem (feed: RSS). Thanks to Anton Fonarev for link aggregation.

Blogs, articles, and projects

Libraries and Tools

New releases and tools this week:

  • pretty 2.3 - Library for helping print things prettily, in Clojure - ANSI fonts, formatted exceptions

  • ring 1.12.0 - Clojure HTTP server abstraction

  • hirundo 0.1.33 - Helidon 4.x RING adapter - using loom/java21+

  • clojure-lsp-intellij 2.1.0 - Intellij Plugin for Clojure & ClojureScript development via Language Server (LSP) made in Clojure

  • datalevin 0.9.3 - A simple, fast and versatile Datalog database

  • squint 0.7.96 - Light-weight ClojureScript dialect

  • Clojure-Sublimed 3.7.0 - Clojure support for Sublime Text 4

  • pg2 0.1.5 - A fast PostgreSQL driver for Clojure

  • calva 2.0.422 - Clojure & ClojureScript Interactive Programming for VS Code

  • timbre-json-appender 0.2.12 - JSON appender for Timbre

  • trace 1.1 - Better REPL based debugging output

  • vl-gallery-edn - vega-lite gallery in EDN format

  • deps.clj 1.11.2.1446 - A faithful port of the clojure CLI bash script to Clojure

  • guardrails 1.2.4 - Efficient, hassle-free function call validation with a concise inline syntax for clojure.spec and Malli

  • metamorph 0.2.4 - A Clojure library designed to providing pipelining operations

  • cli 0.8.58 - Turn Clojure functions into CLIs!

  • metamorph.ml 0.7.3 - Machine learning functions for metamorph based on machine learning pipelines. Part of scicloj.ml

  • clj-kondo 2024.03.13 - Static analyzer and linter for Clojure code that sparks joy

  • clojure-lsp 2024.03.13-13.11.00 - Clojure & ClojureScript Language Server (LSP) implementation

  • clay 2-alpha82 - A tiny Clojure tool for dynamic workflow of data visualization and literate programming

  • bubble-config - An aero powered config with environments aimed at Babashka tasks

  • yamlscript 0.1.42 - Programming in YAML

  • shadow-cljs-ext - Loading the shadow-cljs UI in browser devtools

  • easyreagent - Easy React Components for Reagent

Permalink

PG2 release 0.1.5: Migrations

PG2 version 0.1.5 ships its own migration engine through the pg2-migration package. Like Migratus or Ragtime, it allows to grow the database schema continuously, track changes and apply them with care.

Table of Content

Installation

;; lein
[com.github.igrishaev/pg2-migration "0.1.5]

;; deps
com.github.igrishaev/pg2-migration {:mvn/version "0.1.5"}

Concepts

Migrations are SQL files that are applied to the database in certain order. A migration has an id and a direction: next/up or prev/down. Usually it’s split on two files called <id>.up.sql and <id>.down.sql holding SQL commands. Say, the -up file creates a table with an index, and the -down one drops the index first, and then the table.

Migrations might have a slug: a short human-friendly text describing changes. For example, in a file called 002.create-users-table.up.sql, the slug is “Create users table”.

Naming

In PG2, the migration framework looks for files matching the following pattern:

<id>.<slug>.<direction>.sql

where:

  • id is a Long number, for example 12345 (a counter), or 20240311 (date precision), or 20240311235959 (date & time precision);

  • slug is an optional word or group of words joined with - or _, for example create-users-table-and-index or remove_some_view. When rendered, both - and _ are replaced with spaces, and the phrase is capitalized.

  • direction is either prev/down or next/up. Internally, down and up are transformed to prev and next because these two have the same amount of characters and files look better.

Examples:

  • 001-create-users.next.sql
  • 012-next-only-migration.up.sql
  • 153-add-some-table.next.sql

Above, the leading zeroes in ids are used for better alignment only. Infernally they are transferred into 1, 12 and 153 Long numbers. Thus, 001, 01 and 1 become the same id 1 after parsing.

Each id has at most two directions: prev/down and next/up. On bootstrap, the engine checks it to prevent weird behaviour. The table below shows there are two rows which, after parsing, have the same (id, direction) pair. The bootstrap step will end up with an exception saying which files duplicate each other.

Filename Parsed
001-some-trivial-slug.next.sql (1, next)
001-some-simple-slug.next.sql (1, next)

A migration might have only one direction, e.g. next/up or prev/down file only.

When parsing, the registry is ignored meaning that both 001-Create-Users.NEXT.sql and 001-CREATE-USERS.next.SQL files produce the same map.

SQL

The files hold SQL expressions to be evaluated by the engine. Here is the content of the 001-create-users.next.sql file:

create table IF NOT EXISTS test_users (
  id serial primary key,
  name text not null
);

BEGIN;

insert into test_users (name) values ('Ivan');
insert into test_users (name) values ('Huan');
insert into test_users (name) values ('Juan');

COMMIT;

Pay attention to the following points.

  • A single file might have as many SQL expressions as you want. There is no need to separate them with magic comments like --;; as Migratus requires. The whole file is executed in a single query. Use the standard semicolon at the end of each expression.

  • There is no a hidden transaction management. Transactions are up to you: they are explicit! Above, we wrap tree INSERT queries into a single transaction. You can use save-points, rollbacks, or whatever you want. Note that not all expressions can be in a transaction. Say, the CREATE TABLE one cannot and thus is out from the transaction scope.

For granular transaction control, split your complex changes on two or three files named like this:

# direct parts
001-huge-update-step-1.next.sql
002-huge-update-step-2.next.sql
003-huge-update-step-3.next.sql

# backward counterparts
003-huge-update-step-3.prev.sql
002-huge-update-step-2.prev.sql
001-huge-update-step-1.prev.sql

No Code-Driven Migrations

At the moment, neither .edn nor .clj migrations are supported. This is by design because personally I’m highly against mixing SQL and Clojure. Every time I see an EDN transaction, I get angry. Mixing these two for database management is the worst idea one can come up with. If you’re thinking about migrating a database with Clojure, please close you laptop and have a walk to the nearest park.

Migration Resources

Migration files are stored in project resources. The default search path is migrations. Thus, their physical location is resources/migrations. The engine scans the migrations resource for children files. Files from nested directories are also taken into account. The engine supports Jar resources when running the code from an uberjar.

The resource path can be overridden with settings.

Migration Table

All the applied migrations are tracked in a database table called migrations by default. The engine saves the id and the slug or a migration applied as well as the current timestamp of the event. The timestamp field has a time zone. Here is the structure of the table:

CREATE TABLE IF NOT EXISTS migrations (
  id BIGINT PRIMARY KEY,
  slug TEXT,
  created_at timestamp with time zone not null default current_timestamp
)

Every time you apply a migration, a new record is inserted into the table. On rollback, a corresponding migration is deleted.

You can override the name of the table in settings (see below).

CLI Interface

The migration engine is controlled with both API and CLI interface. Let’s review CLI first.

The pg.migration.cli namespaces acts like the main entry point. It accepts general options, a command, and command-specific options:

<global options> <command> <command options>

General options are:

-c, --config CONNFIG     migration.config.edn      Path to the .edn config file
-p, --port PORT          5432                      Port number
-h, --host HOST          localhost                 Host name
-u, --user USER          The current USER env var  User
-w, --password PASSWORD  <empty string>            Password
-d, --database DATABASE  The current USER env var  Database
    --table TABLE        :migrations               Migrations table
    --path PATH          migrations                Migrations path

Most of the options have default values. Both user and database names come from the USER environment variable. The password is an empty string by default. For local trusted connections, the password might not be required.

The list of the commands:

Name Meaning
create Create a pair of blank up & down migration files
help Print a help message
list Show all the migrations and their status (applied or not)
migrate Migrate forward (everything, next only, or up to a certain ID)
rollback Rollback (the current one, everything, or down to a certain ID)

Each command has its own sub-options which we will describe below.

Here is how you review the migrations:

<lein or deps preamble> \
    -h 127.0.0.1 \
    -p 10150 \
    -u test \
    -w test \
    -d test \
    --table migrations_test \
    --path migrations \
    list

|    ID | Applied? | Slug
| ----- | -------- | --------
|     1 | true     | create users
|     2 | false    | create profiles
|     3 | false    | next only migration
|     4 | false    | prev only migration
|     5 | false    | add some table

Every command has its own arguments and help message. For example, to review the create command, run:

lein with-profile +migrations run -m pg.migration.cli -c config.example.edn create --help

Syntax:
      --id ID             The id of the migration (auto-generated if not set)
      --slug SLUG         Optional slug (e.g. 'create-users-table')
      --help       false  Show help message

Config

Passing -u, -h, and other arguments all the time is inconvenient. The engine can read them at once from a config file. The default config location is migration.config.edn. Override the path to the config using the -c parameter:

<lein/deps> -c config.edn list

The config file has the following structure:

{:host "127.0.0.1"
 :port 10150
 :user "test"
 :password #env PG_PASSWORD
 :database "test"
 :migrations-table :migrations_test
 :migrations-path "migrations"}

The :migrations-table field must be a keyword because it takes place in a HoneySQL map.

The :migrations-path field is a string referencing a resource with migrations.

Pay attention to the #env tag. The engine uses custom readers when loading a config. The tag reads the actual value from an environment variable. Thus, the database password won’t be exposed to everyone. When the variable is not set, an exception is thrown.

Commands

Create

The create command makes a pair of two blank migration files. If not set, the id is generated automatically using the YYYYmmddHHMMSS pattern.

lein with-profile +migration run -m pg.migration.cli \
  -c config.example.edn \
  create

ls -l migrations

20240312074156.next.sql
20240312074156.prev.sql

You can also provide a custom id and a slug as well:

lein with-profile +migration run -m pg.migration.cli \
  -c config.example.edn \
  create \
  --id 100500 \
  --slug 'some huge changes in tables'

ll migrations

100500.some-huge-changes-in-tables.next.sql
100500.some-huge-changes-in-tables.prev.sql
20240312074156.next.sql
20240312074156.prev.sql

List

The list command renders all the migrations and their status: whether they are applied or not.

lein with-profile +migration run -m pg.migration.cli -c config.example.edn list

|    ID | Applied? | Slug
| ----- | -------- | --------
|     1 | true     | create users
|     2 | true     | create profiles
|     3 | true     | next only migration
|     4 | false    | prev only migration
|     5 | false    | add some table

Migrate

The migrate command applies migrations to the database. By default, all the pending migrations are processed. You can change this behaviour using these flags:

... migrate --help

Syntax:
      --all           Migrate all the pending migrations
      --one           Migrate next a single pending migration
      --to ID         Migrate next to certain migration
      --help   false  Show help message

With the --one flag set, only one next migration will be applied. If --to parameter is set, only migrations up to this given ID are processed. Examples:

... migrate           # all migrations
... migrate --all     # all migrations
... migrate --one     # next only
... migrate --to 123  # all that <= 123

Rollback

The rollback command reverses changes in the database and removes corresponding records from the migration table. By default, only the current migration is rolled back. Syntax:

... rollback --help

Syntax:
      --all           Rollback all the previous migrations
      --one           Rollback to the previous migration
      --to ID         Rollback to certain migration
      --help   false  Show help message

The --one argument is the default behaviour. When --all is passed, all the backward migrations are processed. To rollback to a certain migration, pass --to ID. Examples:

... rollback               # current only
... rollback --one         # current only
... rollback --to 20240515 # down to 20240515
... rollback --all         # down to the very beginning

Lein examples

Lein preamble looks usually something like this:

> lein run -m pg.migration.cli <ARGS>

The pg2-migration library must be in dependencies. Since migrations are managed aside from the main application, they’re put into a separate profile, for example:

:profiles
{:migrations
 {:main pg.migration.cli
  :resource-paths ["path/to/resources"]
  :dependencies
  [[com.github.igrishaev/pg2-core ...]]}}

Above, the migrations profile has the dependency and the :main attribute. Now run lein run with migration arguments:

> lein with-profile +migrations run -c migration.config.edn migrate --to 100500

Deps.edn examples

Here is an example of an alias in deps.edn that prints pending migrations:

{:aliases
 {:migrations-list
  {:extra-deps
   {com.github.igrishaev/pg2-migration {:mvn/version "..."}}
   :extra-paths
   ["test/resources"]
   :main-opts
   ["-m" "pg.migration.cli"
    "-h" "127.0.0.1"
    "-p" "10150"
    "-u" "test"
    "-w" "test"
    "-d" "test"
    "--table" "migrations_test"
    "--path" "migrations"
    "list"]}}}

Run it as follows:

> clj -M:migrations-list

You can shorten it by using the config file. Move all the parameters into the migration.config.edn file, and keep only a command with its sub-arguments in the :main-opts vector:

{:aliases
 {:migrations-migrate
  {:extra-deps
   {com.github.igrishaev/pg2-migration {:mvn/version "..."}}
   :extra-paths
   ["test/resources"]
   :main-opts ["migrate" "--all"]}}}

To migrate:

> clj -M:migrations-migrate

API Interface

There is a way to manage migrations through code. The pg.migration.core namespace provides basic functions to list, create, migrate, and rollback migrations.

To migrate, call one of the following functions: migrate-to, migrate-all, and migrate-one. All of them accept a config map:

(ns demo
  (:require
   [pg.migration.core :as mig]))

(def CONFIG
  {:host "127.0.0.1"
   :port 5432
   :user "test"
   :password "secret"
   :database "test"
   :migrations-table :test_migrations
   :migrations-path "migrations"})

;; migrate all pinding migrations
(mig/migrate-all CONFIG)

;; migrate only one next migration
(mig/migrate-one CONFIG)

;; migrate to a certain migration
(mig/migrate-to CONFIG 20240313)

The same applies to rollback:

;; rollback all previously applied migrations
(mig/rollback-all CONFIG)

;; rollback the current migration
(mig/migrate-one CONFIG)

;; rollback to the given migration
(mig/rollback-to CONFIG 20230228)

The read-disk-migrations function reads migrations from disk. It returns a sorted map without information about whether migrations have been applied:

(mig/read-disk-migrations "migrations")

{1
 {:id 1
  :slug "create users"
  :url-prev #object[java.net.URL "file:/.../migrations/001-create-users.prev.sql"]
  :url-next #object[java.net.URL "file:/.../migrations/001-create-users.next.sql"]}
 2
 {:id 2
  :slug "create profiles"
  :url-prev #object[java.net.URL "file:/.../migrations/foobar/002-create-profiles.prev.sql"]
  :url-next #object[java.net.URL "file:/.../migrations/foobar/002-create-profiles.next.sql"]}
 ...}

The make-scope function accepts a config map and returns a scope map. The scope map knows everything about the state of migrations, namely: which of them have been applied, what is the current migration, the table name, the resource path, and more.

The function create-migration-files creates and returns a pair of empty SQL files. By default, the id is generated from the current date & time, and the slug is missing:

(create-migration-files "migrations")

[#object[java.io.File "migrations/20240313120122.prev.sql"]
 #object[java.io.File "migrations/20240313120122.next.sql"]]

Pass id and slug in options if needed:

(create-migration-files "migrations" {:id 12345 :slug "Hello migration"})

[#object[java.io.File "migrations/12345.hello-migration.prev.sql"]
 #object[java.io.File "migrations/12345.hello-migration.next.sql"]]

Conflicts

On bootstrap, the engine checks migrations for conflicts. A conflict is a situation when a migration with less id has been applied before a migration with greater id. Usually it happens when two developers create migrations in parallel and merge them in a wrong order. For example:

  • the latest migration id is 20240312;
  • developer A makes a new branch and creates a migration 20240315;
  • the next day, developer B opens a new branch with a migration 20240316;
  • dev B merges the branch, now we have 20240312, then 20240316;
  • dev A merges the branch, and we have 20240312, 20240316, 20240315.

When you try to apply migration 20240315, the engine will check if 20240316 has already been applied. If yes, an exception pops up saying which migration cause the problem (in our case, these are 20240316 and 20240315). To recover from the conflict, rename 20240315 to 20240317.

In other words: this is a conflict:

id        applied?
20240312  true
20240315  false
20240316  true  ;; applied before 20240315

And this is a solution:

id        applied?
20240312  true
20240316  true
20240317  false ;; 20240315 renamed to 20240317

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.