OSS updates November 2023

In this post I'll give updates about open source I worked on during November 2023.

To see previous OSS updates, go here.

Sponsors

I'd like to thank all the sponsors and contributors that make this work possible! Without you, the below projects would not be as mature or wouldn't exist or be maintained at all.

Open the details section for more info.

Advent of Code

It is Advent of Code time of year again. You can solve puzzles in an online squint or cherry playground here.

Change the /squint/ part of the url to /cherry/ to switch ClojureScript dialect versions.

You can read more about the playground here.

Updates

Here are updates about the projects/libraries I've worked on last month.

• blog I've written two blog posts this month:
  • Writing a Cloudflare worker with squint and bun
  • Playing Advent of Code with Squint.
• squint: CLJS syntax to JS compiler
  Lots of stuff happened in November with squint! You could say that I've grown a little addicted to improving this project currently, driven by how users use it and also while developing the playground, a lot of potential improvements emerged..
  • Restore backward compatibility with code that is compiled with older versions of squint
  • Optimize various outputs for smaller size
  • Add js-in
  • Support into + xform
  • Support sort on strings
  • #386: allow expression in value position in map literal
  • Improvements with respect to laziness in mapcat and concat
  • Do not array mutate argument in reverse
  • Escape JSX attribute vector value (and more)
  • map + transduce support
  • Fix for in REPL mode
    • Throw when object is not iterable in for
  • Make next lazy when input is lazy
  • Fix playground shim (fixes issue in older versions of Safari)
  • Add js-mod and quot
  • #380: Don't emit space in between #jsx tags
  • Add re-find
  • Add condp macro
  • Use compare as default compare function in sort (which fixes numerical sorting)
  • Allow assoc! to be called on arbitrary classes (regression)
  • Improve get to call get method when present.
  • Allow keywords and collections to be used as functions in HOFs
  • Make filter, etc aware of truthiness
  • Reduce code size for truthiness checks
  • Add str/split-lines
  • Add partition-by
  • Add parse-long
  • Add sort-by
  • Fix top level await
  • Support multiple dimensions in aset
  • Add coercive-= as alias for ==
  • Add js-delete
  • Fix min-key and max-key and improve tests
  • Add min-key and max-key
  • Fix defonce in REPL-mode
  • Fix doseq and for when binding name clashes with core var
  • Several REPL improvements
  • Improve https://squint-cljs.github.io/squint/
  • Allow alias name to be used as object in REPL mode
  • Copy resources when using squint compile or squint watch
  • Return map when select-keys is called with nil
  • nREPL server: print values through cljs.pprint (@PEZ)
  • Initial (incomplete!) nREPL server on Node.js: npx squint nrepl-server :port 1888
  • Update/refactor threejs example
  • #360: assoc-in! should not mutate objects in the middle if they already exist
  • Evaluate lazy-seq body just once
  • Avoid stackoverflow with min and max
  • #360: fix assoc-in! with immutable objects in the middle
  • Add mod, object?
  • Optimize get
  • Add threejs example
  • #357: fix version in help text
  • Fix iterating over objects
  • Add clojure.string's triml, trimr, replace
  • Fix examples/vite-react by adding public/index.html
  • Add find, bounded-count, boolean?, merge-with, meta, with-meta, int?, ex-message, ex-cause, ex-info
  • Fix munging of reserved symbols in function arguments
• scittle-hoplon: a custom scittle distribution with Hoplon. I helped developing the SCI configuration for Hoplon.
• gespensterfelder: a demo that Jack Rusher wrote using Three.js ported to squint.
• neil: A CLI to add common aliases and features to deps.edn-based projects Version 0.2.63 released which adds mvn search and some bugfixes
• CLI: Turn Clojure functions into CLIs!
  • Small bugfix around priority of :exec-args and default
• aoc-proxy: a Cloudflare worker that can be used to fetch Advent of Code puzzle input from the browser (see Advent of Code playground)
• squint-macros: a couple of macros that stand-in for applied-science/js-interop and promesa to make CLJS projects compatible with squint and/or cherry.
• clojure-mode: Clojure/Script mode for CodeMirror 6.
  • Ported the eval-region extension to squint so you can use it straight from JS. This is used in the squint playground when you press Cmd-Enter after an expression.
• sci.configs: A collection of ready to be used SCI configs.
  • A helper macro was improved such that you can define macros that are usable in SCI
  • The re-frame configuration now has support for re-frame.alpha. See playground.
• babashka: native, fast starting Clojure interpreter for scripting. A new release: 1.3.186!
  • Support self-contained binaries as uberjars!
  • Add java.security.KeyFactory, java.security.spec.PKCS8EncodedKeySpec, java.net.URISyntaxException, javax.crypto.spec.IvParameterSpec
  • Fix babashka.process/exec wrt babashka.process/*defaults*
  • #1632: Partial fix for (.readPassword (System/console))
  • Enable producing self-contained binaries using uberjars
  • Bump httpkit to 2.8.0-beta3 (fixes GraalVM issue with virtual threads)
  • Bump deps.clj and fs
  • Expose taoensso.timbre.appenders.core
  • nREPL: implement ns-list op
  • SCI: optimize swap!, deref and reset! for normal atoms (rather than user-created IAtoms)
  • Add test for #1639
  • Upgrade to GraalVM 21.0.1
    Still unreleased:
  • Add java.util.ScheduledFuture
  • Support Runnable to be used without import
  • Allow catch to be used as var name
• SCI: Configurable Clojure/Script interpreter suitable for scripting and Clojure DSLs
  Released version 0.8.41
  • Bump edamame to 1.3.23
  • #889: allow (def foo/foo 1) when inside namespace foo
  • #891: reset file metadata on var when it's re-evaluated from other file
  • #893: expose sci.async/eval-form and sci.async/eval-form+
  • Improve sci.async/eval-string, respect top-level do forms
  • Add experimental new :static-methods option to override how static methods get evaluated.
  • Expose destructure
  • Macroexpand (.foo bar) form
  • Optimize deref, swap!, reset! for host values
  • Add time macro to core namespace
  • #896: allow catch to be used as var name
• cherry: Experimental ClojureScript to ES6 module compiler
  • Released version 0.1.10 which catches up with the latest compiler improvements in squint
• clj-kondo: static analyzer and linter for Clojure code that sparks joy.
  • New :condition-always-true and :underscore-in-namespace linters + couple of bugfixes. Release expected in December.

Other projects

These are (some of the) other projects I'm involved with but little to no activity happened in the past month.

Permalink

Advent of Code 2023 Day 01

It's that time again. Advent of Code 2023. A new two part programming problem each day from now through December 25th.

Since 2015 I've looked forward to the event, taken part, and posted at least a few write ups here on my blog. I figured this year would be the same.

Of course, it does end up putting a crunch in my routine. Usually each morning I wake up, work out, usually a run unless the weather's bad, have my coffee, do the puzzle, read blogs and other content over breakfast and then I'm ready for the rest of my day.

Advent of Code adds another morning activity. Actually, maybe not this year since I broke a toe last week and am not allowed to work out for the foreseeable future.

So, this morning, I woke up, made coffee, worked on the puzzle (not quite done yet) and then looked at today's problem.

To be honest, I was a little surprised. After reading it it seemed somewhat harder than past day 1 questions. It might have to do with the fact that I haven't programmed at all since last years event and really not much in the past couple of years but still.

Usually the first days problem involves some straight through processing of the data without a whole lot of thought. After I completed today's problem I looked back on past years adn actually, while this one is a bit more involved, it really isn't that different.

Part 1 of today's problem was a straightforward as past year. You've got a bunch of lines each with some digits interspersed:

1bcc2
a1b2c3fgh4
etc.

You have to find the first and last digits in each line and take them as a number. For the above you'd have 12 for the first line and 14 for the second. You then had to sum all of those numbers up.

I thought the most straightforward way to do this was to go through the data and keep only the digits.

While I wrote my solution in Clojure, in Python for each line you could use an expression like this:

[x for x in l if x>'0' and x<='9']

Then you can just take the first and last items in the list, put them together and convert to a number:

int(x[0]+x[-1])

Finally, add all the lines together and you're done.

Part 2 threw a wrench in the works.

Now, in addition to the digits counting as digits, spelled out digits (one, two, etc.) had to count as digits as well. Looking at the sample data, no problem:

two1nine
eightwothree
abcone2threexyz
xtwone3four
4nineeightseven2
zoneight234
7pqrstsixteen

One could just replace all occurences of "one" with "1," "two" with "2," and so on.

The catch was that you could also have overlapping numbers:

9oneighthree9

Should that be converted to "91839," "91igh39," "9on8hree9," or "91839."

This made things trickier - first, how do you interpret it and then how do you solve.

It turned out that you had to allow for overlapping words so the above example would translate to 91839.

My solution, in Clojure, which you can find here involves using overlapping regular expressions - I didn't know about them so it took a while to figure that out.

Another approach, at least I think would be to replace the strings one at a time (first do the "one" occurences, then "two" etc.) but replace them with the digit within the original text. So, "one" would be replaced by "o1ne" or something similar. This way, if the final "e" was next to an "ight" you'd get that "eight" when you replaced for "eight."

All of this isn't too bad but probably required more thought than your typical advent of code day 1.

Still, a fun morning activity.

Permalink

Senior Golang Developer

We are looking for Senior Golang Developer to join product development for our client from the USA. The product is an AWS hosted multi-module payment and analytical platform for the healthcare services, written in Clojure/Go language stack. The product encompasses a few applications for customer journeys (web, mobile) and a complex micro-service architecture. 

•  4+ years of development experience 
•  2+ years of working experience with Golang 
•  Experience with RDBMS (PostgreSQL) and NoSQL (MongoDB, DynamoDB) 
•  Experience with Messaging systems (SNS/SQS, Kafka, etc.) 
•  Experience with cloud services (preferably AWS) 
•  At least Upper-intermediate level of English 
•  Ability to write clean and easily maintained code 

Nice to have: 

• Experience with Kafka 
• Experience with Java 

Responsibilities: 

• Participate in solution design and development, deliver high-quality code 
• Regularly communicate with the team members at the client’s side, participate in status meetings, design sessions, and brainstorming 
• Provide estimation and reporting of assigned tasks 

We offer friendly working conditions with competitive compensation and benefits including: 

• Comfortable working environment 
• Friendly team and management 
• Competitive salary 
• Free English classes 
• Regular performance-based compensation review 
• Flexible working hours 
• 100% paid vacation, 4 weeks per year 
• 100% paid sick-leaves 
• Medical insurance (50% is paid) 
• Corporate and team building events 

Permalink

Clojure Deref (Dec 1, 2023)

Permalink

Elixir vs. Clojure for functional programming at scale

Permalink

Ep 101: Sportify!

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to feedback@clojuredesign.club, or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "introducing Sportify!". We tackle a new application, thinking it'll be an easy win—only to discover that our home run was a foul, and the real world is about to strike us out!

Our discussion includes:

• Introduce a new series!
• Sportsball! Sportsball!
• Going back in time.
• An overview of video production workflows.
• What is a media asset manager?
• How hard could it be?
• What could possibly go wrong?
• What are all the things we'll need to handle?
• What is a situated problem?
• Does immutability matter when most of the work is I/O?
• Code stability in Clojure.

Selected quotes:

• "Clojure has made our lives fun, so we want to make your lives fun."
• "What do people love when they're watching sporting events? They love their highlights."
• "This is not a business problem. This is a sports problem, and sports problems are different."
• "Now this is where we're reaching the edges of reality, but just hang on. Come with us."
• "How hard could it be?!"
• "What could possibly go wrong?!"
• "But this MAM...do you have to be polite? Can I have the video ma'am?"
• "We've got to do the right amount. That's the hard part: the right amount."
• "Is there a fraught problem that's not situated, or a situated problem that's not fraught?!"
• "Situated, in my mind, is useful. I don't just want to heat the room up with my computer. I want to actually get something done!"

Permalink

How to Get Started with TDD in Clojure

Testing is one of the aspects of software development that you can not hide yourself from. At least if you work in the industry, you shouldn't. Testing is vital to battle against regression in the code, meaning that when you add new features, you are not breaking the previous functionality. Of course, over-testing can be harmful and slow you down if you need to spend more time updating tests than the feature code itself.

This post is not about what, when, and if you should test but a tutorial on getting started when you've decided that you need them and how to integrate the testing into your workflow.

How Clojure is Different From the Mainstream

In other dynamic languages, you often run tests in watch mode so that your test runner triggers a new run whenever you save the file. This offers a fast feedback loop on your changes. Clojure differs from the others in that it has the REPL that can be used to evaluate the code while developing and testing it without running all of the tests every time. Let's get back to the topic of running the tests after we've gone through how to write tests.

Writing Tests

Clojure has its unit testing framework core.test ; therefore, extra dependencies are unnecessary when writing the tests. At the core of testing are assertions, and we use the macro clojure.test/is that takes any predicate with an optional message describing it.

(require '[clojure.test :as t])

(t/is (true? true) "true is true")
;; => true

(t/is (= true false) "true equals false")
;; => false

;; REPL output
;;
;; FAIL in () (NO_SOURCE_FILE:4)
;; true equals false
;; expected: (= true false)
;;  actual: (not (= true false))

You can think of this as a function call to evaluate whether a given predicate (boolean or a function that produces a boolean) is true.

Assertions are used within deftests macro that returns a function to evaluate the assertions inside.

(t/deftest test-assertion
  (t/is (= true false) "true equals false")
  (t/is (true? true) "true is true"))
;; => #'core-test/test-assertion

(test-assertion)
;; REPL output
;;
;; FAIL in (test-assertion) (NO_SOURCE_FILE:17)
;; expected: (= true false)
;;  actual: (not (= true false))

Read the Clojure style guide on naming conventions

If you read the REPL outputs, you might have noticed we lost the message attached to our failing assertion. With deftests we can use testing to define the context for the test.

(t/deftest test-assertion
  (t/testing  "true equals false"
    (t/is (= true false)))
  (t/testing "true is true"
      (t/is (true? true))))
;; => #'core-test/test-assertion

(test-assertion)

;; REPL output
;;
;; FAIL in (test-assertion) (NO_SOURCE_FILE:25)
;; true equals false
;; expected: (= true false)
;;   actual: (not (= true false))

You can see that we got our descriptions back. And that's pretty much the basics of testing in Clojure. There are more tools in clojure.test like are and use-fixture. Let me know if you want to learn more about them. But now, back to test runners.

There are several test runner options to choose from. You could use the functions provided by core.test, Cognitect's test-runner, or Kaocha, to name a few. We'll be using Kaocha this time. It is an extendable, build-tool agnostic test runner.

Setup the project

Let's set up a project with Kaocha testing in mind so that we have a deps.edn alias for running the tests once and in watch mode.

Here's the file structure

 deps.edn
 src
    core.clj
 test
     core_test.clj

Let's fill in the deps.edn first and add Kaocha as a test dependency.

{:paths ["src"]
 :deps {org.clojure/clojure {:mvn/version "1.11.1"}}

 :aliases
 {:test
  {:extra-paths ["test"]
   :extra-deps  {lambdaisland/kaocha {:mvn/version "1.77.1236"}}
   :exec-fn     kaocha.runner/exec-fn
   :exec-args   {:skip-meta :slow}}

  :watch
  {:exec-args   {:watch?     true
                 :skip-meta  :slow
                 :fail-fast? true}}}}

Under aliases, we have keys :test and :watch and these work together. With the Clojure CLI tool, we can execute the :exec-fn from the alias configuration by using the -X command line option

-X is configured with an arg map with :exec-fn and :exec-args keys, and stored under an alias in deps.edn:

Deps and CLI Reference: Execute a Function

so by running clj -X:test we can run the tests once, and with clj -X:test:watch we can continuously run the tests whenever we save the source code file.

Let's write something to test in core.clj namespace.

(ns core)

(defn multiply [x y]
  (* x y))

And write the tests we saw earlier, plus a test for multiplication in core-test.

(ns core-test
  (:require [clojure.test :as t]
            [core :as sut]))

(t/deftest test-assertion
  (t/testing  "true equals false"
    (t/is (= true false)))
  (t/testing "true is true"
      (t/is (true? true))))

(t/deftest multiply-test
  (t/testing "multiplication works as expected"
    (t/is (= 4 (sut/multiply 2 2)))))

SUT (Software Under Test) I picked up this from Practicalli years ago and it stuck with me.

And now we are ready to test.

By default, Kaocha uses the src and tests paths so we can run the tests without explicit configuration. There are various configuration options with plugins, so check out the docs to see what's possible.

Testing in the REPL

Clojure developers usually use their REPL to evaluate the tests, and in my case, it would be in Emacs with CIDER using one of the cider-test-* commands that I have behind keyboard shortcuts.

The same can be done with VS code and Calva. Read from the docs how to run tests with Calva.

Conclusion

Testing in Clojure is pretty effortless. We do not need much ceremony to start testing; we need a namespace for the tests and a test runner of our choice.

I hope you found this helpful, and thank you for reading.

Feel free to reach out and let me know what you thinksocial links in the menu.

Permalink

Clojure Bites - Mazeboard 1 - Dumdom event handler

Intro

In the previous post I've introduced the game I am working on, presenting the stack and the general code structure I'd like to follow, I've closed the post with a couple of screenshot of a static game board, now it is time to make it a bit more interactive.

The goal is to let the user to click on one of the arrows (which represent the available actions, more actions to come later) and move the player icon to correct new position.

The game UI state

The UI state is a map that holds the data that drives the rendering of the components, and includes information about:

• the boards with its own tiles, tiles can also include which action can be performed on them
• the players with name and position
• the current player
• the current round
• the end position

The map will be updated after player actions and changes will be reflected in the interface by re-rendering what have changed. To achieve this goal the UI state will be stored in an atom, attaching the rendering function to the atom watch list, something like:

(ns mazeboard.ui.core
  (:require [dumdom.core :as d]
            [mazeboard.test-data :as test-data]
            [mazeboard.ui.components.app :as app]))

(defonce state_ (atom {}))

(defn render [state]
  (d/render (app/App state)
            (js/document.getElementById "app")))

(defn init []
  (add-watch state_ :app (fn [_ _ _ new-state] (render new-state)))

  (swap! state_ assoc :game (test-data/create-game)))

create-game returns a map which is an implementation detail and can be ignored. With this setup, each time the state_ atom will be mutated, the rendering function will be called with the new state, causing a change in the UI. At this time the only way to make changes to the UI state is to manually swap! or reset! the atom, so not ideal for a real game, but it is enough get started; this approach can be useful to manually test ideas in the REPL until we are sure how something will be implemented, creating a nice interactive development environment.

Triggering actions on click events

After few iterations, playing with the game UI (man, I suck at CSS), I can shows where the players are in the board and what the player "One" can do i.e. move south or east, here is how it looks currently:

Clicking on one of the arrows should trigger an action that will update the player's position, change the current player, update the round number and update the UI to reflect what has changed. First step is to handle clicks on the arrows.

Dumdom provides two ways to attach event handlers to DOM elements, specifying the key :on-click in the element configuration map:

• if the value of the key is a function then it will be called directly
• if the value is anything else (for example lists, vects, maps) then a global handler will be called

The global event handler must be registered with dumdom.core/set-event-handler! whose only parameter is a function that will receive the target element and whatever data has been specified in the :on-click value of the element. The event handler function have a signature like (fn [target data]).

Both methods (fn or data) have use cases that can be more or less suitable depending on your specific needs, in this case I have opted for the data driven approach for two reasons:

• it feels more declarative, the element "tells" what it wants to do
• it makes it possible to attach actions to elements from the game logic that can be later used by the UI; especially useful when the game logic will be handled by a separate (remote) process

Lets re-write the init method to setup an event handler:

(defn init []
  (add-watch state_ :app (fn [_ _ _ new-state] (render new-state)))
  
  (d/set-event-handler!
   (fn [_e actions]
     (doseq [action actions]
       (prn action))))

  (swap! state_ assoc :game (test-data/create-game)))

And here is how the Tile component looks like, with data driven :on-click trigger:

(defcomponent Tile [{:keys [actions position end-position? players]} all-players]
  (cond-> [:div.tile {:class "tile-background"
                      :key position}

           (for [{:keys [on-click action-class]} actions]
             [:div.action {:on-click on-click :class action-class}])

           (for [player players]
             (let [{:keys [class name]} (nth all-players player)]
               [:div.player {:class class} name]))]

    end-position?
    (conj [:div.end-position])))

;; example of component's actions
#_[{:on-click [[:move-player {:direction :east :target 0}]]
                :action-class "action-move-east"}
   {:on-click [[:move-player {:direction :south :target 0}]]
                :action-class "action-move-south"}]

Dumdom does not impose any structure to the data that can be associated to an element's event handler so I have taken the examples in Parens of the dead and I will structure the actions as a vector of vectors, with the first element of the sub vectors describing the action as a keyword and the second element as the options/parameters map of the action itself.

After setting up the global event handler, if we click on one the arrows, for example to go east, in the browser console log we should see [:move-player {:direction :east :target 0}], meaning that the event handler is working properly. But still, nothing is happening in the game UI, lets fix that.

Dispatching actions and state updates

We are now able to trigger actions and we need something that can update the game state and reflect the changes to screen. In this case I think it is better jump straight to the code that will move a player in Mazeboard:

(ns mazeboard.actions
  (:require [clojure.core.match :refer [match]]))

(defn update-position
  [position direction]
  (case direction
    :north (update position :row dec)
    :east (update position :col inc)
    :south (update position :row inc)
    :west (update position :col dec)))

(defn action-move-player
  [{:keys [current-player players] :as game} {:keys [direction target]}]
  (let [old-pos (get-in players [target :position])
        new-pos (update-position (get-in game [:players target :position]) direction)]
    (-> game
        (assoc-in [:players target :position] new-pos)
        (assoc :current-player (mod (inc current-player) (count players)))
        (update-in [:board :tiles old-pos :players] disj target)
        (update-in [:board :tiles new-pos :players] conj target)
        (update :round-number inc))))

(defn dispatch
  [game action]
  (prn "Triggered action" action)
  (match action
         [:move-player opts] (action-move-player game opts)))

The code must be read bottom up, first thing to look at is the dispatch function that will match the requested action to a specific function calling it; the fact the here clojure.core.match is being used is an implementation detail, you may prefer a different approach (for example defmulti is another good candidate). This is quite simple, we try to match the requested action to something that we know how to handle, in this case moving the player. action-move-player will, of course, update the game (and UI) state, returning it.

There is still one bit missing, who is calling dispatch and how this will update the global state atom? We have to update the global handler to connect actions to state updates, here is how the init fn is looking now:

(defn init []
  (d/set-event-handler!
   (fn [_e actions]
     (doseq [action actions]
       (swap! state_ update :game actions/dispatch action))))

  (add-watch state_ :app (fn [_ _ _ state] (render state)))

  (swap! state_ assoc :game (test-data/create-game)))

After these changes, when clicking one of the arrows the player icon will be placed correctly in the next tile, in this screenshot we can see the effect of moving south

Together with the updated player "One" position we can also observe that the round number has changed and that now it is the turn of player "Two"; updating the tiles to reflect what the other user can do will be the content of another post.

Future improvements

At this point we have a working application that can react to user actions and render the updated state accordingly. What I don't like about the current approach is that it is coupling the game logic and the UI logic, for example knowing which actions are available and how to render them is a concern of the UI state and not of the game state, but now everything is mixed together and soon managing both the game and the UI will be a complete mess and hard to maintain. To make a concrete example here is how the tile is represented at the moment:

(def tile {:actions [{:on-click [[:move-player {:direction :east :target 0}]]
                      :action-class "action-move-east"}
                     {:on-click [[:move-player {:direction :south :target 0}]]
                      :action-class "action-move-south"}]
           :walls [:closed :open :open :closed]
           :end-position? true
           :players [0]})

The :actions, :end-position? and :players keys are closely related to the UI, instead the :walls key is more on the game logic side.

By separating game and UI I hope to get to a setup were:

• the UI will trigger actions that will be sent to the game layer
• the game layer will validate the requested action and, if valid, will update the game state
• after updating the game state, events will be triggered for all clients describing what has happened
• all clients will update their state using the received events and re-render the UI accordingly

A bonus benefit of this kind of separation of concerns is that unit testing the game logic and the event generation that will feed the UI will cover most of the application! (Thanks Parens of the dead for this idea).

Conclusion

In this post we have seen how to model the UI state (even if coupled with game state), how to dispatch actions from DOM elements and how to handle them to update the game.

The whole setup is quite primitive but good enough for a starting point. I am very excited to finally decouple game and UI concerns, it will make it easier to build game and UI logic separately.

Discuss

• Reddit
• Twitter
• Clojurians Slack

Sources

Mazeboard source code is available here.

Permalink

Holy Dev Newsletter Nov 2023

Welcome to the Holy Dev newsletter, which brings you gems I found on the web, updates from my blog, and a few scattered thoughts. You can get the next one into your mailbox if you subscribe.What is happeningExciting times! The gods of time have finally allowed me and Thomas Clark to get in sync and start reviving Wolframite, a bridge between Clojure and the awesome Wolfram Language and its built-in knowledge base. I am very excited about Wolfram because of the high level where it operates, its integration with scientific computing, and the integrated knowledge base. It truly makes everything "computable". With Wolframite, you can (soon) call to the free-for-dev Wolfram Engine or a full installation of Mathematica to do some work for you. If we are lucky then we will have Wolframite 1.0 by the end of the year. (Disclaimer: Wolframite is a refactoring of an older Clojuratica, and 99% of the work has been done by others. Me and Thomas just want to bring it to a conclusion and get it into your hands.)Aside of Wolfram, I have also spent too much 😅 time with improving my blog, namely updating cryogen-asciidoc to AsciidoctorJ v3, and consequently updating my builder to java > 8 (Netlify still only offers v8). I have also written a short note about The Four Heads of Complexity, based on a post by Kent Beck. (You should absolutely check out his newsletter Software Design. Click on the "No thanks >" link at the bottom of the page to get a preview and some free posts.)Finally, I have spent many hours these past two weeks on implementing pagination. You’d think it is a solved problem, but it turns out to be pretty complicated, when the entries come from 6 different tables. I hope to have time to write some property-based tests for this beast, and work on simplification in future iterations.

Permalink

Only true is true

Code

;; true.clj

(true? true)
(true? false)
(true? 0)
(true? 1)
(false? 1)
(false? false)
(false? 0)
(number? 1)
(= 1 1)
(empty? [])
(empty? nil)
;; (empty? 1) ; throws exception
(nil? nil)
(nil? 1)
(zero? 0)
;; (zero? nil) ; throws exception
;; (zero? "278") ; throws exception
;; (one? 1) ; throws exception
;; (one? "1") ; throws exception

Notes

• Clojureverse Discussion

Permalink

The beauty of using Graph Databases with TerminusDB and Clojure, Part I.

Most of the time, if not always, data is the key factor in the technology world. We transform data into information and use this information to create processes that can improve our lives. However, data tends to grow rapidly, so we need to make critical decisions about data design and study the different relationships.

As Flexianers, we understand the implications of good data design in the technology world. Therefore, we are always trying to stay informed about the different approaches we can use for our solutions. Today, we want to present our perspective on the foundations of Graph Databases, specifically focusing on TerminusDB in this instance.

Let’s start with the basics: what is a Graph Database?

A graph database is a type of database that is designed to store and manage data using graph theory principles. Instead of using tables, rows, and columns like traditional relational databases, graph databases use nodes, edges, and properties to represent and store data.

In the graph database world, nodes represent entities or objects, while the edges represent the relationships between these entities.

In this image, we have three types of nodes: customers, orders, and products. On the other hand, we have edges to represent when a customer creates an order and which orders contain some specific product.

Using this approach we can rapidly verify what are the products most ordered by the customer just traversing the graph and doing some “counting” right? If you noticed this, excellent! You already understood one of the many use cases of graph databases.

We are already using relational databases, why graphs now?

This question came to mind the first time I read about Graph Databases. The relevant factor here is use cases. While the relational approach is still extensively used and solves many different types of problems, there are difficulties that we have to confront when the complexity of relationships between different tables scales up.

As developers, if we remember how efficient it was to traverse a graph or a tree using the algorithms provided by graph theory, I think we can infer the answer.

Although we want to present some of the advantages of using Graph Databases:

• Graph Databases excel at representing and dealing with complex relationships between data entities. It is very intuitive to model various types of relationships, such as connections, associations, and networks using this structure. This makes this approach highly suitable for scenarios with rich and interconnected data.
• Making decisions based on relationship patterns is very efficient in the graph world. We will not need to evaluate results from complex joins between different tables; we only need to traverse the graph.

There are more advantages, but they are specific to the particular DBMS, so we prefer to mention the core ones. After this, we would like to talk a bit about our experience with Graph DBMS, using TerminusDB as our main guest today.

TerminusDB

TerminusDB is an open-source database that utilizes a graph-based approach and integrates a data version control system. To interact with the DBMS, you can use WOQL (Web Object Query Language), which is a datalog-like query language. For more detailed information about WOQL, I invite you to read “The Power of Web Object Query Language” at TerminusDB Blog.

In addition to the features commonly associated with graph databases, TerminusDB brings some unique capabilities:

• Version Control System: TerminusDB incorporates version control principles, allowing users to track and manage changes made to data over time and across different domains. It provides versioning capabilities to manage different versions of data and facilitates branching, merging, and conflict resolution.
• Easy Collaboration: TerminusDB simplifies collaboration by enabling users to share and clone databases. This allows multiple users to work together, make changes, and collaborate on the data with minimal risk of compromising data integrity.
• Succinct Data Structures: TerminusDB leverages succinct data structures, which utilize memory space close to the lower bounds of theoretical space complexity. This optimization ensures efficient query execution while minimizing resource consumption as well.

Now we know a bit more about our guest today, it’s time to start playing!

The easy part! The online dashboard

For this post, we used this way to play with TerminusDB and focus on showing you how to do the basic interaction using Clojure and how to model our basic use case in the database. Although, we will be showing you a bonus gist in the second part of this post to show other mechanisms to install and start with TerminusDB.

The first step to start using the dashboard is accessing this address. Where you will see something like this:

If you don’t have an account follow the sign-up process, else go and log in.

In this part, we invite you to plan a bit with the interface, but the most relevant parts for this post are:

• The dashboard: Here we we can create categories for our schemas. This will bring us the opportunity to create a classification layer and group schemas together.

• The profile section: This is a section where we can use our OpenAI key in order to associate indexes using AI, and we can collect our personal access token to be used with the Javascript/Python clients or via CURL.

The collection of our personal access token is vital here because we will be using it to access our Clojure application. After collecting the token be sure to create an environment variable named “TERMINUSDB_ACCESS_TOKEN” and associate to this token.

The funny part, Clojure(80%) code!

Sadly, while we were writing this post we had zero success finding a native Clojure client(Only Python and Javascript clients).

But, don’t worry, Clojure always provides amazing solutions for our problems! So we decided to use libpython-clj for this purpose and, guess what? It worked beautifully! To be honest, we finished using like 80% Clojure,15% Python, and 5% WOQL. Anyways, let’s describe the process.

First of all, we need to install the terminusdb python client, so the following command is enough(If you don’t have python3 installed, please go and install it):

python3 -m pip install terminusdb-client

After this, we proceed to add the libpython-clj dependency which can be found here in our project.clj file:

...
:dependencies [[org.clojure/clojure "1.11.1"]
                 [clj-http "3.12.3"]
                 [metosin/jsonista "0.3.8"                 [clj-python/libpython-clj "2.025"]]
...

With this addition, we can start the interaction with the python ecosystem. So the code to connect to the database and create it looks like this:

(ns terminusdb-post-sample.core
  (:require [libpython-clj2.require :refer [require-python]] ;; Require libpython-clj deps
            [libpython-clj2.python :refer [py.] :as py]))
            
(require-python '[terminusdb_client :as terminusdb-client]) ;; Require terminusdb_client module


(def terminusdb_server "https://cloud.terminusdb.com/")
(def team "")


(defn connect-to-terminus [server team]
  ;; First we need to create our WOQLClient object.
  (let [client (terminusdb-client/WOQLClient server)]
    ;; With the client value, we can use the function py. to call the method
    ;; client.connect
    (py. client connect :team team :use_token true)))
    
(defn create-db [client dbid team label description prefixes]
  ;; We receive the client as parameter. This is done on purpose because we don't want
  ;; to create a connection everytime we run a database operation.
  (py. client create_database :dbid dbid
                              :team team
                              :label label
                              :description description
                              :prefixes prefixes
                              :include_schema false))
...

This code is pretty self-explanatory and we added a few comments as a guide to our readers.

After reading and doing some research, we noticed that the Python and Javascript libraries are basically communicating with the TerminusDB API via HTTP. So we can create something similar with Clojure using a combination of clj-HTTP and jsonista for example, and create code like this:

(ns terminusdb-post-sample.terminusdb-connector
  (:require [clj-http.client :as client]
            [jsonista.core :as json]))
      
(def terminusdb-team "")
(def terminusdb-server "https://cloud.terminusdb.com/")
(def terminusdb-api-indicator "api/")
(def terminusdb-db-resource "db")
(def terminusdb-token (System/getenv "TERMINUSDB_ACCESS_TOKEN"))

(defn send-request-to-list-dbs
  []
  (let [headers {"API_TOKEN" terminusdb-token}
        response (client/get (str terminusdb-server 
                                  terminusdb-api-indicator
                                  terminusdb-db-resource)
                             {:headers headers})
        body (:body response)]
    (json/read-value body)))

(defn create-database
  [db-data]
  (let [headers {"API_TOKEN" terminusdb-token}
        response (client/post (str terminusdb-server
                                   terminusdb-api-indicator
                                   terminusdb-db-resource
                                   "/"
                                   terminusdb-team
                                   "/"
                                   (:db-id db-data))
                              {:headers headers
                               :content-type :json
                               :body (json/write-value-as-string (apply dissoc  db-data [:db-id :team-id]))})]
    (:body response)))

This code block is more complicated but not rocket science either. To summarize, we are doing HTTP calls to the API exposed by TerminusDB using clj-http and decoding a map to a JSON string with jsonista. We could make some adjustments to the code to make it look more elegant, but we preferred to show this in the simplest way.

Conclusion

In this post, we showed you a little introduction to graph databases and we played a bit with a few different possibilities offered by Clojure to connect with a TerminusDB instance by using an already Python client or using native Clojure libraries.

In the next part, we will expose an interesting use case of graph databases, the rest of the code to interact with TerminusDB, and the complete implementation of our basic terminusdb-connector.

Thank you for reading this post! Be safe!

The post The beauty of using Graph Databases with TerminusDB and Clojure, Part I. appeared first on Flexiana.

Permalink

Clojure(Script) Engineer at Vara

Permalink

Coding in the Shadows: Hidden Gems of Lisp, Clojure, and friends

In the world of programming, rankings are more than just numbers; they're a source of endless debate and pride. Much like how people are fascinated by top 10 lists in music or movies, there's a similar ranking system for programming languages. But unlike box office hits or Billboard charts, the criteria for ranking programming languages aren't as clear-cut.

There's a constant back-and-forth among programmers about which language is superior. You'll often hear them boast, "My language is better than yours," followed by demonstrations of elegant one-liners or efficient algorithms. But the truth is, we haven't settled on a universally accepted measure for ranking these languages. So, when it comes to these rankings, they're based primarily on popularity.

Now, this isn't about social media popularity – it's not a matter of likes or shares. It's about how many people are actively using these languages. Measuring this is tricky, as you can't simply count heads or survey every programmer out there. However, with a combination of job market analysis, community activity, and usage statistics in various projects, a somewhat accurate picture can be formed.

This approach to ranking may not satisfy everyone, especially since it doesn't necessarily reflect the technical merits or innovativeness of a language. Yet, it does offer a snapshot of the programming landscape, showing us which languages are currently in demand and shaping the world of technology.

The top spots in the programming language rankings are often occupied by names that even those outside the IT world might recognize. Languages like Java, Python, C, and JavaScript are household names in the tech community and beyond. When you delve into the top 20, the "least popular" among these is still used by approximately 1% of programmers worldwide – a small percentage, perhaps, but still significant when you consider the sheer number of people coding across the globe.

However, the list of programming languages doesn't stop at the top 20. Venture further down to the top 50, and you'll start to encounter some truly niche languages. Among them is Logo, a language developed in 1967 with the aim of teaching programming concepts. It's famously associated with the 'turtle graphics,' where you can control a turtle's movement on-screen through coding commands. It’s intriguing to ponder what modern applications it could have, given that it’s still used, albeit by a mere 0.1% of programmers.

Also nestled in this segment of the list is COBOL – an even older language that first appeared 64 years ago. Despite its age, COBOL remains a vital cog in the machinery of large insurance companies and banks. These institutions rely on massive mainframe computers running on millions of lines of COBOL code. It’s a language that, while seemingly antiquated, underpins systems too critical and complex to risk rewriting. The mere thought of transitioning away from COBOL raises concerns about potentially catastrophic system failures.

Lisp

Venturing even further down the list, you encounter two languages that are my personal heroes. One such language is Lisp, a language that has its roots in academia and theoretical computation. Fascinatingly, Lisp was initially conceived on paper, without any practical implementation on a computer. It's a testament to the pure, almost philosophical approach to programming language design.

Lisp is built on a handful of core concepts and a mere eight operations. Yet, from this simplicity springs an immense power. Its creator, John McCarthy, mathematically demonstrated that Lisp was a fully capable programming language, earning it the classification of being "Turing-complete." This term refers to a system of computation that, given enough resources, can solve any problem that a Turing machine (a basic model of a computer) can. In this sense, Lisp is akin to geometry as formulated by Euclid – based on just a few axioms, it unfolds into a vast and intricate world of possibilities.

The elegance of Lisp lies in its minimalism and its flexibility. It's a language that encourages a different way of thinking, one that is more about the essence of computation and less about the specifics of syntax. This philosophical underpinning makes it not just a tool for writing software, but also a medium for exploring the very nature of programming itself. For those who delve into its depths, Lisp offers a unique perspective, revealing the elegant complexity that can arise from the simplest of foundations.

Lisp's role in the evolution of computer science extends beyond its elegant design. It was the first programming language widely used for research in Artificial Intelligence (AI) during the 1960s and 1970s. At that time, Lisp was at the forefront of this groundbreaking field. This deep association with AI led to the creation of specialized hardware known as Lisp machines. These were computers designed specifically to run Lisp efficiently, representing a unique convergence of software and hardware dedicated entirely to one programming language. Unlike modern computers, which are built to run a wide range of software, Lisp machines were optimized for the nuances and particularities of Lisp.

The phrase "it was Lisp all the way down" aptly captures this era. In these machines, Lisp wasn't just a programming language running on general-purpose hardware; it was an integral part of the entire system. This period in computing history highlights Lisp's significant influence and the fervent belief in its potential to unlock the secrets of artificial intelligence and advanced computation.

As the years progressed, however, Lisp's prominence in the programming world began to wane for reasons that are as complex as they are varied. This shift in the landscape saw the rise and fall of various programming languages, each vying for a spot in the rapidly evolving tech industry.

This was the time when I came across an article by Paul Graham called Beating the Averages. Graham described Lisp as a secret weapon that wasn’t for everyone - it was special, meant for a select group of programmers. Reading this, I felt a strong connection; I knew I was part of that group. Lisp just clicked with me.

This led me to dive deeper into Graham's other writings about Lisp. The more I read, the more natural it felt to use Lisp. It was as if my mind was perfectly suited for it. This idea reminded me of a quote from Eric Raymond:

"Lisp is worth learning for the profound enlightenment experience you will have when you finally get it; that experience will make you a better programmer for the rest of your days, even if you never actually use Lisp itself a lot."

During this time, I was playing around with Lisp, immersing myself in its unique style. That's when Clojure appeared, like a fresh wave for those of us who loved Lisp. It felt like a modern version of Lisp, mixing old Lisp ideas with new ways of programming. Clojure made it easier to use Lisp with today's technology, first with the Java Virtual Machine (JVM), then with the browser. For a while, it looked like Clojure would make Lisp popular again. It even became one of the top 20 programming languages for a bit.

Yet, despite this promising start, Clojure didn't maintain its momentum. It eventually settled into a niche, maintaining a presence in a few actively developed applications but largely receding from the mainstream programming scene. For me, personally, Clojure has been more than just a programming language. It has profoundly shaped my thinking, offering a unique perspective on problem-solving, system design, and the elegance of code.

Doesn't Clojure deserve its own chapter? It certainly does. You can read more about why it lies so close to my heart: Why My Brain Is Wired for Clojure

APL

I promised to mention two programming languages that hold a special place for me; Lisp is the first, and APL is the other. APL, which stands for "A Programming Language," began its life not even as a language for programming, but as a notation system used by people at IBM. It was developed as a way to clearly communicate complex programming and mathematical ideas, extending beyond traditional mathematical notation.

When you first see APL code, two things will likely catch your eye. Firstly, it looks almost like a secret code from a pirate's treasure map, filled with unique symbols like backward-crossed circles, triangles with extra lines, and an array of other special characters. At the height of APL's popularity, these symbols were so integral to the language that you could even find special keyboards made just for typing in APL.

This mysterious and almost romantic quality of APL's appearance was one of the things that drew me to it. But there's another aspect of APL that's immediately noticeable: its conciseness. APL is renowned for its ability to condense what would be ten lines of code in other languages into a single line of APL.

What exactly makes APL so concise? At first glance, it might seem like a simple trick. Other programming languages are filled with long keywords and functions like continue and length, while APL appears to just replace these with a single character. But there's much more to it than that.

Yes, it's true that APL often uses single characters, like the Greek letter rho (ρ), to perform tasks that would require more elaborate expressions in other languages. Rho, for example, is used to determine the length of a string or a vector. But the real power of APL lies in its fundamental approach to data. APL treats nearly everything as a vector, or a list of items, and it's designed to operate on these vectors efficiently.

Take a simple operation like adding 2 to a vector. In strict mathematical terms, this might not make sense, but APL interprets it as adding 2 to each element of the vector. This characteristic allows you to perform complex operations without the need for writing loops or conditions, which are common and space-consuming in other languages. By thinking in terms of operations on whole vectors or arrays at once, APL enables a level of conciseness and elegance that is hard to match in more verbose programming languages.

Recognizing the limitations imposed by the need for a specialized keyboard, the creators of APL took an innovative step forward. They developed a new language, J, which was designed to be used with the standard ASCII characters available on every keyboard. This move addressed one of APL's biggest accessibility issues, making the language more approachable for a broader audience.

J's creation marked the beginning of an interesting progression in programming language development. It inspired the creation of K, a language that further refined the ideas of APL and J, focusing on efficiency and speed. K, in turn, led to the development of Q, a language built on the foundations of K but geared more towards database and query operations.

Q might not be a widely known programming language, but it has a very important role in one specific area: handling financial transactions. This is because Q is really good at processing large amounts of data quickly, which is crucial in finance where every second counts. The demand for Q programmers may not be widespread across the tech industry, but within the finance sector, their skills are highly sought after. If you know how to program in Q, you can actually earn a lot of money.

K

This part is going to be more technical. For programmers familiar with the concepts of functional programming, K offers a powerful yet concise way to perform operations like summation. For example, the expression +/!8 in K efficiently sums up the numbers up to 8. Here, !8 generates a list of numbers from 0 to 7. Then, +/ acts as what's known in functional programming as a "reduce" operation, applying the addition function across the list. Another example is calculating factorial: */1+!8. We generate a list of the number from 0 to 7; then add 1 to add number: 1+!8 -- we already know that adding 1 to a list adds 1 to each element. Finally, we reduce the list applying multiplication.

One more example. Here is how to find the largest number in a list: |/'4 2 7 1. Let's break it down how it works. The phrase |/' is a combination of two symbols:

• |: This is the 'maximum' function in K. When used on its own, it gives you the maximum of two numbers.
• /': This is known as an 'over' operator. It is used to apply a function repeatedly between the elements of a list.

Together, |/' applies the 'maximum' function across the list. It repeatedly compares pairs of numbers and carries forward the larger number each time. This process continues across the entire list until the largest number is determined.

If programmers have heard of K, it's usually in connection with code golf. Code golf is a unique and playful programming challenge where the objective is to solve a specific problem or complete a task using the least amount of code possible. Unlike typical programming practices where code is often measured in lines, code golf counts every single character, making each one count. This form of programming competition emphasizes brevity and resourcefulness, encouraging programmers, often referred to as 'code golfers,' to think outside the box. They must employ ingenious methods and deep knowledge of their chosen language's syntax and features to craft a solution that is as short as possible.

K has gained a reputation as one of the most successful languages for code golf. Its concise syntax and powerful built-in functions allow programmers to express complex operations in just a few characters, making it ideal for these challenges. However, in the world of code golf, some languages have been created solely to excel in this arena. These languages, often referred to as 'esoteric' or 'golfing' languages, might include built-in functions for common code golf tasks, like a one-character function to generate prime numbers - a frequent requirement in code golf challenges.

While the one-liners showcase K's ability to succinctly express complex ideas, it has a serious consequence beyond winning at code golf. Not only K applications are compact in size, but the K interpreter itself is extremely small, especially when compared to similar programs in other languages. This minimal footprint allows K applications, along with the K interpreter, to fit entirely within the L2 cache of a CPU. The L2 cache is a smaller, faster memory compared to a computer's main RAM, and because it's closer to the CPU, data stored in the L2 cache can be accessed much more quickly. This proximity significantly enhances the execution speed of K applications.

This compactness and efficient use of the L2 cache set K apart from many other programming languages. In most languages, especially those that produce larger, more memory-intensive applications, critical parts of the program and the interpreter or runtime environment are less likely to fit entirely in this fast-access cache. As a result, K applications, despite being interpreted, can often achieve execution speeds comparable to compiled C code, which is renowned for its performance. The efficiency of K doesn’t just stem from its concise syntax but also from how it harmonizes with the underlying hardware, making the most of the CPU's capabilities to deliver high performance. The speed and efficiency of K make it perfect for working with huge amounts of data in real-time, like in stock trading.

If you want to try out K, there are some open source implementations, like John Earnest's oK which has a REPL and a calculator-like interface for mobile phones with a charting feature.

Permalink

Full Stack Engineer (mid- to senior-level) at OpenMarkets Health

Permalink

Adding Dependencies on Clojure Projects the Node Way: A Small Intro to neil CLI

One of the things that I found really hard when starting with Clojure is handling dependencies. I come from a PHP and JS background so composer and npm were the standard to me. I also worked with some Ruby projects, and gem is also cool.

They all have something in common: a way to remove/add dependencies by simply using a command followed by the name of the package. Like in npm:

# latest version
npm install package-name

# specific version
npm install package-name@version

The Clojure Way

If you find yourself trying to add a dependency on a Clojure project you will find yourself having to copy and paste the dependency "token" to a deps.edn or project.clj or the maven way.

If you take a look a this library which is an "official" library you will find three snippets of code for adding the dependency to your project and some with a very strange syntax.

After doing that, the next time you open your project or REPL clojure will download the dependencies for you.

The neil Way

Recently, doing my Clojure repository searching for studying purposes (or CRSSP for short) I stumbled on this very underrated and not much used in tutorial CLI helper for deps.edn Clojure projects called neil.

neil is:

A CLI to add common aliases and features to deps.edn-based projects.

Created by the same guy who created babashka which is a way to write bash scripts, node scripts, and even apple scripts using Clojure. A very proficient and influential developer in the Clojure community. This is how borkduke's neil helps us:

neil add dep com.stuartsierra/component 

And done. The dependency was added to deps.edn. Automatization wins again!

Neil is the Way!

So neil has a bunch of other features like project scaffolding, building, testing, adding license, etc. I really recommend you take a deep look at the repository and learn all the automatized possibilities that neil adds to your project.

Thanks!

In closing, I hope this introduction to the neil CLI was helpful in simplifying your Clojure dependency workflow. A big thank you to the developers behind neil for creating a tool that streamlines common tasks. Please feel free to reach out if you have any other questions about using neil in your own projects.

Permalink