New Library: clj-reload

The problem

Do you love interactive development? Although Clojure is set up perfectly for that, evaluating buffers one at a time can only get you so far.

Once you start dealing with the state, you get data dependencies, and with them, evaluation order starts to matter, and now you change one line but have to re-eval half of your application to see the change.

But how do you know which half?

The solution

Clj-reload to the rescue!

Clj-reload scans your source dir, figures out the dependencies, tracks file modification times, and when you are finally ready to reload, it carefully unloads and loads back only the namespaces that you touched and the ones that depend on those. In the correct dependency order, too.

Let’s do a simple example.

a.clj:

(ns a
  (:require b))

b.clj:

(ns b
  (:require c))

c.clj:

(ns c)

Imagine you change something in b.clj and want to see these changes in your current REPL. What do you do?

If you call

(clj-reload.core/reload)

it will notice that

  • b.clj was changed,
  • a.clj depends on b.clj,
  • there’s c.clj but it doesn’t depend on a.clj or b.clj and wasn’t changed.

Then the following will happen:

Unloading a
Unloading b
Loading b
Loading a

So:

  • c wasn’t touched — no reason to,
  • b was reloaded because it was changed,
  • a was loaded after the new version of b was in place. Any dependencies a had will now point to the new versions of b.

That’s the core proposition of clj-reload.

Usage

Here, I recorded a short video:

But if you prefer text, then start with:

(require '[clj-reload.core :as reload])

(reload/init
  {:dirs ["src" "dev" "test"]})

:dirs are relative to the working directory.

Use:

(reload/reload)
; => {:unloaded [a b c], :loaded [c b a]}

reload can be called multiple times. If reload fails, fix the error and call reload again.

Works best if assigned to a shortcut in your editor.

Usage: Return value

reload returns a map of namespaces that were reloaded:

{:unloaded [<symbol> ...]
 :loaded   [<symbol> ...]}

By default, reload throws if it can’t load a namespace. You can change it to return exception instead:

(reload/reload {:throw false})

; => {:unloaded  [a b c]
;     :loaded    [c b]
;     :failed    b
;     :exception <Throwable>}

Usage: Choose what to reload

By default, clj-reload will only reload namespaces that were both:

  • Already loaded
  • Changed on disk

If you pass :only :loaded option to reload, it will reload all currently loaded namespaces, no matter if they were changed or not.

If you pass :only :all option to reload, it will reload all namespaces it can find in the specified :dirs, no matter whether loaded or changed.

Usage: Skipping reload

Some namespaces contain state you always want to persist between reloads. E.g. running web-server, UI window, etc. To prevent these namespaces from reloading, add them to :no-reload during init:

(reload/init
  {:dirs ...
   :no-reload '#{user myapp.state ...}})

Usage: Unload hooks

Sometimes your namespace contains stateful resource that requires proper shutdown before unloading. For example, if you have a running web server defined in a namespace and you unload that namespace, it will just keep running in the background.

To work around that, define an unload hook:

(def my-server
  (server/start app {:port 8080}))

(defn before-ns-unload []
  (server/stop my-server))

before-ns-unload is the default name for the unload hook. If a function with that name exists in a namespace, it will be called before unloading.

You can change the name (or set it to nil) during init:

(reload/init
  {:dirs [...]
   :unload-hook 'my-unload})

This is a huge improvement over tools.namespace. tools.namespace doesn’t report which namespaces it’s going to reload, so your only option is to stop everything before reload and start everything after, no matter what actually changed.

Usage: Keeping vars between reloads

One of the main innovations of clj-reload is that it can keep selected variables between reloads.

To do so, just add ^:clj-reload/keep to the form:

(ns test)

(defonce x
  (rand-int 1000))

^:clj-reload/keep
(def y
  (rand-int 1000))

^:clj-reload/keep
(defrecord Z [])

and then reload:

(let [x test/x
      y test/y
      z (test/->Z)]
  
  (reload/reload)
  
  (let [x' test/x
        y' test/y
        z' (test/->Z)]
    (is (= x x'))
    (is (= y y'))
    (is (identical? (class z) (class z')))))

Here’s how it works:

  • defonce works out of the box. No need to do anything.
  • def/defn/deftype/defrecord/defprotocol can be annotated with ^:clj-reload/keep and can be persistet too.
  • Project-specific forms can be added by extending clj-reload.core/keep-methods multimethod.

Why is this important? With tools.namespace you will structure your code in a way that will work with its reload implementation. For example, you’d probably move persistent state and protocols into separate namespaces, not because logic dictates it, but because reload library will not work otherwise.

clj-reload allows you to structure the code the way business logic dictates it, without the need to adapt to developer workflow.

Simply put: the fact that you use clj-reload during development does not spill into your production code.

Comparison: Evaluating buffer

The simplest way to reload Clojure code is just re-evaluating an entire buffer.

It works for simple cases but fails to account for dependencies. If something depends on your buffer, it won’t see these changes.

The second pitfall is removing/renaming vars or functions. If you had:

(def a 1)

(def b (+ a 1))

and then change it to just

(def b (+ a 1))

it will still compile! New code is evaluated “on top” of the old one, without unloading the old one first. The definition of a will persist in the namespace and let b compile.

It might be really hard to spot these errors during long development sessions.

Comparison: (require ... :reload-all)

Clojure has :reload and :reload-all options for require. They do track upstream dependencies, but that’s about it.

In our original example, if we do

(require 'a :reload-all)

it will load both b and c. This is excessive (b or c might not have changed), doesn’t keep track of downstream dependencies (if we reload b, it will not trigger a, only c) and it also “evals on top”, same as with buffer eval.

Comparison: tools.namespace

tools.namespace is a tool originally written by Stuart Sierra to work around the same problems. It’s a fantastic tool and the main inspiration for clj-reload. I’ve been using it for years and loving it, until I realized I wanted more.

So the main proposition of both tools.namespace and clj-reload is the same: they will track file modification times and reload namespaces in the correct topological order.

This is how clj-reload is different:

  • tools.namespace reloads every namespace it can find. clj-reload only reloads the ones that were already loaded. This allows you to have broken/experimental/auxiliary files lie around without breaking your workflow TNS-65
  • First reload in tools.namespace always reloads everything. In clj-reload, even the very first reload only reloads files that were actually changed TNS-62
  • clj-reload supports namespaces split across multiple files (like core_deftype.clj, core_defprint.clj in Clojure) TNS-64
  • clj-reload can see dependencies in top-level standalone require and use forms TNS-64
  • clj-reload supports load and unload hooks per namespace TNS-63
  • clj-reload can specify exclusions during configuration, without polluting the source code of those namespaces.
  • clj-reload can keep individual vars around and restore previous values after reload. E.g. defonce doesn’t really work with tools.namespace, but it does with clj-reload.
  • clj-reload has 2× smaller codebase and 0 runtime dependencies.
  • clj-reload doesn’t support ClojureScript. Patches welcome.

That’s it!

Clj-reload grew from my personal needs on Humble UI project. But I hope other people will find it useful, too.

Let me know what works for you and what doesn’t! I’ll try to at least be on par with tools.namespace.

And of course, here’s the link:

Permalink

cljcastr, or a young man's Zencastr clonejure

A man with the Babashka logo for a face sits in front of a laptop and a mic

Who amongst us hasn't wanted to make a podcast? And of those who, who amongst them hasn't drooled over Zencastr, which is a web-based podcast recording studio thingy?

I don't even know how to answer that question, as convoluted as it got, but what I'm trying to say is that I got to see Zencastr's cool interface, and have been meaning to play around with the browser's audio / video API anyway, so why not see if I can whip up a quick Zencastr clone in Clojure? I mean, how hard can it be?

Popping in a Scittle

You may recall from my adventures cloning Flickr that I love ClojureScript but feel sad at my own lack of knowledge when trying to use shadow-cljs. You may also recall that the sweet sweet antidote to this was Scittle, which allows you to "execute Clojure(Script) directly from browser script tags via SCI". Since I'm now an expert Scittler, I figured that's the obvious place to start a Zencastr clone. So let's start a project!

$ mkdir cljcastr && cd cljcastr

Then we need a bb.edn, which we can just steal from Scittle's nrepl demo and modify ever so slightly to serve resources out of the public/ directory:

{:deps {io.github.babashka/sci.nrepl
        {:git/sha "2f8a9ed2d39a1b09d2b4d34d95494b56468f4a23"}
        io.github.babashka/http-server
        {:git/sha "b38c1f16ad2c618adae2c3b102a5520c261a7dd3"}}
 :tasks {http-server {:doc "Starts http server for serving static files"
                      :requires ([babashka.http-server :as http])
                      :task (do (http/serve {:port 1341 :dir "public"})
                                (println "Serving static assets at http://localhost:1341"))}

         browser-nrepl {:doc "Start browser nREPL"
                        :requires ([sci.nrepl.browser-server :as bp])
                        :task (bp/start! {})}

         -dev {:depends [http-server browser-nrepl]}

         dev {:task (do (run '-dev {:parallel true})
                        (deref (promise)))}}}

Given this, let's create a public/index.html to bootstrap our ClojureScript:

<!doctype html>
<html class="no-js" lang="">

<head>
    <meta charset="utf-8">
    <meta http-equiv="x-ua-compatible" content="ie=edge">
    <title>cljcastr</title>
    <meta name="description" content="">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="stylesheet" href="style.css">

    <link rel="apple-touch-icon" href="/apple-touch-icon.png">
    <!-- Place favicon.ico in the root directory -->

    <script src="https://cdn.jsdelivr.net/npm/scittle@0.6.15/dist/scittle.js" type="application/javascript"></script>
    <script>var SCITTLE_NREPL_WEBSOCKET_PORT = 1340;</script>
    <script src="https://cdn.jsdelivr.net/npm/scittle@0.6.15/dist/scittle.nrepl.js"
        type="application/javascript"></script>
    <script type="application/x-scittle" src="cljcastr.cljs"></script>
</head>

<body>
    <!--[if lt IE 8]>
      <p class="browserupgrade">
      You are using an <strong>outdated</strong> browser. Please
      <a href="http://browsehappy.com/">upgrade your browser</a> to improve
      your experience.
      </p>
    <![endif]-->
</body>

</html>

And of course we need to get stylish with a public/style.css:

body {
  font-family: Proxima Nova,helvetica neue,helvetica,arial,sans-serif;
}

And finally, we need public/cljcastr.clj to script some Clojure:

;; To start a REPL:
;;
;; bb dev
;;
;; Then connect to it in Emacs:
;;
;; C-c l C (cider-connect-cljs), host: localhost; port: 1339; REPL type: nbb

(ns cljcastr)

I always forget how to start the REPL and connect to it, so I left myself some nice explicit instructions, which we shall now follow. In the terminal:

$ bb dev
Serving static assets at http://localhost:1341
nREPL server started on port 1339...
Websocket server started on 1340...

We'll then visit http://localhost:1341 in the browser and open up the JavaScript console, which should say:

   :ws #object[WebSocket [object WebSocket]]
> 

Finally, back in Emacs, hitting C-c l C (cider-connect-cljs), selecting localhost for the host, 1339 for the port, and nbb for the REPL type, then C-c C-k (cider-load-buffer) shows us this in the terminal:

:msg "{:versions {\"scittle-nrepl\" {\"major\" \"0\", \"minor\" \"0\", \"incremental\" \"1\"}}, :ops {\"complete\" {}, \"info\" {}, \"lookup\" {}, \"eval\" {}, \"load-file\" {}, \"describe\" {}, \"close\" {}, \"clone\" {}, \"eldoc\" {}}, :status [\"done\"], :id \"3\", :session \"5e3f1fb0-1f13-4db0-a25a-b63a9e7d7d72\", :ns \"cljcastr\"}"
:msg "{:value \"nil\", :id \"5\", :session \"5e3f1fb0-1f13-4db0-a25a-b63a9e7d7d72\", :ns \"cljcastr\"}"
:msg "{:status [\"done\"], :id \"5\", :session \"5e3f1fb0-1f13-4db0-a25a-b63a9e7d7d72\", :ns \"cljcastr\"}"

Exciting! Let's prove we're connected with a Rich comment:

(comment

  (println "Now we're cooking with Scittle!")  ; <- C-c C-v f c e (cider-pprint-eval-last-sexp-to-comment)
  ;; => nil

  )

If all went well, we should see glorious things in the JavaScript console:

Screenshot of a browser window with the JavaScript console displaying: Now we're cooking with Scittle!

Left to our own devices

Now that we have a solid platform to stand on (namely: the REPL), let's get on with the cljcasting! We'll start by asking ourselves what audio and video devices we have at our disposal.

Modern browsers implement the Media Capture and Streams API, which provides support for streaming audio and video data. You can read a bit about the backstory of the API in a nice little article by Eric Bidelman and Sam Dutton: Capture audio and video in HTML5. This article points to a great demo of A/V capture that Sam Dutton did.

I am relating all this because Sam Dutton's demo comes with source code that shows not tells how to use this API, and like any great artist, I stole that code and used it for my own nefarious purposes. Well, "nefarious" might be a bit of a stretch, but c'mon, I've got a reputation to uphold over here. 😅

Our entrypoint into the wonderful world of browser-based A/V is the MediaDevices interface, which is exposed as navigator.mediaDevices. MediaDevices has an instance method enumerateDevices(), which we can use to, well, enumerate the audio and video devices availabile to our browser:

(comment

  (.enumerateDevices js/navigator.mediaDevices)
  ;; => #object[Promise [object Promise]]

  )

Blergh, looks like it returns a promise instead of an actual value (OK, OK, a promise is a value, but you know what I mean). That means that we need to feed a function to the promise that actually does the thing. We do this by using Promise.then(), which calls a function when the promise is fulfilled and returns a promise which wraps the return value of the function, allowing us to chain calls in a very similar way to Clojure's threading operators, -> and ->>.

Now, before we do this, I discovered during the writing of this post that my browser hides all devices from me until I give it permission to use my audio and video devices. We can trigger that permission request with this incantation:

(comment

  (.getUserMedia js/navigator.mediaDevices #js {:video true, :audio true})
  ;; => #object[Promise [object Promise]]

  )

What should happen is that we're presented with a dialog asking for permission to use our video camera and microphone. Assuming we trust ourselves this far, we can accept and get back to seeing what mediaDevices.enumerateDevices() returns:

(comment

  (-> (.enumerateDevices js/navigator.mediaDevices)
      (.then println))
  ;; => #object[Promise [object Promise]]

  )

This results in some awesome stuff being printed to the JS console:

#js [#object[InputDeviceInfo [object InputDeviceInfo]]
     #object[InputDeviceInfo [object InputDeviceInfo]]
     #object[InputDeviceInfo [object InputDeviceInfo]]
     #object[InputDeviceInfo [object InputDeviceInfo]]
     #object[InputDeviceInfo [object InputDeviceInfo]]
     #object[InputDeviceInfo [object InputDeviceInfo]]
     #object[MediaDeviceInfo [object MediaDeviceInfo]]
     #object[MediaDeviceInfo [object MediaDeviceInfo]]
     #object[MediaDeviceInfo [object MediaDeviceInfo]]
     #object[MediaDeviceInfo [object MediaDeviceInfo]]
     #object[MediaDeviceInfo [object MediaDeviceInfo]]
     #object[MediaDeviceInfo [object MediaDeviceInfo]]]

OK, so we have an array of MediaDeviceInfo and InputDeviceInfo objects, which have convenient .label and .kind properties that we can avail ourselves of:

(comment

  (-> (.enumerateDevices js/navigator.mediaDevices)
      (.then (fn [devices]
               (->> devices
                    (group-by #(.-kind %))
                    (sort-by key)
                    (map (fn [[kind ds]]
                           (str kind ":\n  "
                                (str/join "\n  " (map #(.-label %) ds)))))
                    (str/join "\n")
                    println))))
  ;; => #object[Promise [object Promise]]

  )

This gives us something much more reasonable in the console:

audioinput:
  Default
  Tiger Lake-LP Smart Sound Technology Audio Controller Digital Microphone
  HD Webcam B910 Analog Stereo
  Yeti X Analog Stereo
audiooutput:
  Default
  Tiger Lake-LP Smart Sound Technology Audio Controller HDMI / DisplayPort 3 Output
  Tiger Lake-LP Smart Sound Technology Audio Controller HDMI / DisplayPort 2 Output
  Tiger Lake-LP Smart Sound Technology Audio Controller HDMI / DisplayPort 1 Output
  Tiger Lake-LP Smart Sound Technology Audio Controller Speaker + Headphones
  Yeti X Analog Stereo
videoinput:
  Integrated Camera (04f2:b6ea)
  UVC Camera (046d:0823) (046d:0823)

This looks like useful information indeed! Let's extract some functions out of the mess we made in our REPL:

(ns cljcastr
  (:require [clojure.string :as str]))

(defn log-devices [devices]
  (->> devices
       (group-by #(.-kind %))
       (sort-by key)
       (map (fn [[kind ds]]
              (str kind ":\n  "
                   (str/join "\n  " (map #(.-label %) ds)))))
       (str/join "\n")
       println)
  devices)

(defn get-devices []
  (.enumerateDevices js/navigator.mediaDevices))

Now, taking inspiration from Sam Dutton's demo, let's make a UI that lets you choose your video source and stream from it into a window:

Screenshot of a browser window showing Sam Dutton's mediaDevices demo

We'll start by opening up our public/index.html and sprinkling in some UI elements:

<body>
    <!--[if lt IE 8]>
      ...
    <![endif]-->

  <div id="container">
    <h1>cljcastr</h1>
    <div class="select">
      <label for="videoSource">Video source:</label>
      <select id="videoSource"></select>
    </div>
    <video autoplay muted playsinline></video>
  </div>
</body>

Since the labels of my devices were quite lengthy, let's make the select quite widthy by dropping the following in public/style.css:

select {
  width: 300px;
}

After doing this, we'll sadly have to refresh the browser to get it to pick up the changes to index.html and style.css. We could of course add in some awesome watching and live reloading like quickblog does, but that smacks of effort and we don't have any useful state in the REPL to mourn anyway, so we'll bite our tongue and hope we don't have too many HTML or CSS changes left to make.

Now that we have the bones of a UI, let's actually populate the select element with the video devices that we've detected. We can try grabbing all of the video input devices:

(comment

  (-> (get-devices)
      (.then (fn [devices]
               (->> devices
                    (filter #(= "videoinput" (.-kind %)))
                    log-devices))))
  ;; => #object[Promise [object Promise]]

  )

The JS console now reads:

videoinput:
  Integrated Camera (04f2:b6ea)
  UVC Camera (046d:0823) (046d:0823)

Stuffing these in the select should be fairly straightforward:

(def video-select (.querySelector js/document "select#videoSource"))

(comment

  (-> (get-devices)
      (.then (fn [devices]
               (doseq [device
                       (->> devices
                            (filter #(= "videoinput" (.-kind %)))
                            log-devices)]
                 (let [option (.createElement js/document "option")]
                   (set! (.-value option) (.-deviceId device))
                   (set! (.-text option)
                         (or (.-label device)
                             (str "Camera " (inc (.-length video-select)))))
                   (.appendChild video-select option))))))
  ;; => #object[Promise [object Promise]]

  )

Et voilà! The browser now shows our cameras:

Screenshot of a browser window showing a selection box containing two video input sources

Having proven this works, let's make a function out of it:

(defn populate-device-selects! [devices]
  (doseq [device
          (->> devices
               (filter #(= "videoinput" (.-kind %)))
               log-devices)]
    (let [option (.createElement js/document "option")]
      (set! (.-value option) (.-deviceID device))
      (set! (.-text option)
            (or (.-label device)
                (str "Camera " (inc (.-length video-select)))))
      (.appendChild video-select option))))

In case you haven't come across this convention before, adding a ! to the end of a function name indicates that the function is mutating something, in this case, adding options to the select element.

<video> killed the Adobe Flash star

Having given ourselves a way to select a video input device, we just need to actually display the video being input into said device. For this, we'll need to avail ourselves of the MediaDevices.getUserMedia() method. Given a device ID, it will "prompt the user for permission to use a media input which produces a MediaStream".

Let's check which video input device is selected:

(comment

  (.-value video-select)
  ;; => "d9862f4684c6b3f21bf95436a09b58dfa1b7a442e79aff225314e5e9bab45217"

  )

If we feed this ID to getUserMedia(), we should get a stream back:

(comment

  (-> js/navigator.mediaDevices
      (.getUserMedia (clj->js {:video {:deviceId {:exact (.-value video-select)}}}))
      (.then #(println (.getVideoTracks %))))
  ;; => #object[Promise [object Promise]]

  )

This clj->js business is taking a ClojureScript hashmap and turning it into a JavaScript object with nested objects. You have to remember to use it whenever you're calling JavaScript functions that take "maps" as arguments, lest those functions basically ignore your arguments. Don't ask me how I know! 😅

As an interesting aside, ClojureScript also has a #js reader tag, which says "turn the following ClojureScript literal into the JavaScript equivalent". As an interesting aside to the aside, this is not recursive. Don't ask me how I know! 😅

(comment

  #js {:video "killed the radio star"}
  ;; => #js {:video "killed the radio star"}

  #js {:video {:deviceId {:exact (.-value video-select)}}}
  ;; => #js {:video
  ;;         {:deviceId
  ;;          {:exact
  ;;           "24705d21befb46ac4b2596716eee02c2fecb819447ef3edb91562aad41d2db50"}}}

  (clj->js {:video {:deviceId {:exact (.-value video-select)}}})
  ;; => #js {:video
  ;;         #js {:deviceId
  ;;              #js {:exact
  ;;                   "24705d21befb46ac4b2596716eee02c2fecb819447ef3edb91562aad41d2db50"}}}

  )

OK, getting back to our code:

(comment

  (-> js/navigator.mediaDevices
      (.getUserMedia (clj->js {:video {:deviceId {:exact (.-value video-select)}}}))
      (.then #(println (.getVideoTracks %))))
  ;; => #object[Promise [object Promise]]

  )

When we evaluated this, two interesting things should have happened. First, the JS console should say something like this:

#js [#object[MediaStreamTrack [object MediaStreamTrack]]]

And second, the recording light on your webcam should light up. OMG we're getting somewhere! 🎉

Of course, our goal isn't simply to turn on the webcam, but rather to turn it on and then start streaming video to our webpage. This is actually pretty straightforward, compared to what we've done to get to this point.

(def video-element (.querySelector js/document "video"))

(comment

  (-> js/navigator.mediaDevices
      (.getUserMedia (clj->js {:video {:deviceId {:exact (.-value video-select)}}}))
      (.then #(set! (.-srcObject video-element) %)))
  ;; => #object[Promise [object Promise]]

  )

The results are stunning...ly bad. Unless of course you're more photogenic than I am, in which case, congrats!

Screenshot of a browser window showing a video of me

Now that we're streaming video, it looks pretty ugly to have the video pressed right up against the bottom of the select element, so let's add some margin in our style.css:

select {
  width: 300px;
  margin-bottom: 10px;
}

With all of this plumbing, we can hook it up to the actual select box so it automatically starts playing video when we make a camera selection, rather than requiring us to go all 1337 h4ckZ0r in the REPL.

(def active-stream (atom nil))

(def video-element (.querySelector js/document "video"))

(defn log-error [e]
  (.error js/console e))

(defn log-devices [devices]
  ;; ...
  )

(defn get-devices []
  ;; ...
  )

(defn populate-device-selects! [devices]
  ;; ...
  )

(defn select-device! [select-element tracks]
  (let [label (-> tracks first (.-label))
        index (->> (.-options select-element)
                   (zipmap (range))
                   (some (fn [[i option]]
                           (and (= label (.-text option)) i))))]
    (when index
      (println "Setting selected video source to index" index)
      (set! (.-selectedIndex select-element) index))))

(defn start-video! [stream]
  (reset! active-stream stream)
  (select-device! video-select (.getVideoTracks stream))
  (set! (.-srcObject video-element) stream))

(defn stop-video! []
  (when @active-stream
    (println "Stopping currently playing video")
    (doseq [track (.getTracks @active-stream)]
      (.stop track))))

(defn set-video-stream! []
  (stop-video!)
  (let [video-source (.-value video-select)
        constraints {:video {:deviceId (when (not-empty video-source)
                                         {:exact video-source})}}]
    (println "Getting media with constraints:" constraints)
    (-> js/navigator.mediaDevices
        (.getUserMedia (clj->js constraints))
        (.then start-video!)
        (.catch log-error))))

(defn load-ui! []
  (set! (.-onchange video-select) set-video-stream!)
  (-> (set-video-stream!)
      (.then get-devices)
      (.then log-devices)
      (.then populate-device-selects!)))

(comment

  (load-ui!)
  ;; => #object[Promise [object Promise]]

  )

Let's break these functions down to see what's going on here:

(defn load-ui! []
  (set! (.-onchange video-select) set-video-stream!)
  (-> (set-video-stream!)
      ;; ...
      ))

First, we set the change handler for the video select element to the set-video-stream! function, then we call set-video-stream!.

(defn set-video-stream! []
  (stop-video!)
  ;; ...
  )

set-video-stream! calls stop-video!:

(defn stop-video! []
  (when @active-stream
    (println "Stopping currently playing video")
    (doseq [track (.getTracks @active-stream)]
      (.stop track))))

stop-video! checks to see if we have a truthy value in our active-stream atom, which we won't at this point, since we initiatise the atom with a nil value:

(def active-stream (atom nil))

Back to set-video-stream!:

(defn set-video-stream! []
  ;; ...
  (let [video-source (.-value video-select)
        constraints {:video {:deviceId (when (not-empty video-source)
                                         {:exact video-source})}}]
    ;; ...
    ))

Since we haven't yet populated the video select element with video sources, video-select.value will be "", which is not not empty (in other words, it's empty), so our constraints map will look like this:

(comment

  (let [video-source (.-value video-select)
        constraints {:video {:deviceId (when (not-empty video-source)
                                         {:exact video-source})}}]
    constraints)
  ;; => {:video {:deviceId nil}}

  )

Feeding this to navigator.mediaDevices.getUserMedia() will result in prompting the user for permission to access whichever of their cameras the browser considers the default, then turning on that camera and providing a MediaStream containing a video track with the input, which we then feed to start-video!.

(defn set-video-stream! []
  ;; ...
  (let [ ;; ...
       ]
    (println "Getting media with constraints:" constraints)
    (-> js/navigator.mediaDevices
        (.getUserMedia (clj->js constraints))
        (.then start-video!)
        (.catch log-error))))

start-video! is fairly simple:

(defn start-video! [stream]
  (reset! active-stream stream)
  (select-device! video-select (.getVideoTracks stream))
  (set! (.-srcObject video-element) stream))

The first thing it does is reset the value of the active-stream atom to the stream returned by getUserMedia(), then calls select-device! with the video select DOM element and the video tracks of the stream, then finally sets the srcObject property of the <video> element to the stream, which results in us seeing ourselves (or whatever our default camera is aimed at).

select-device! is responsible for setting the value of a select element to the device corresponding to the first of the MediaStreamTrack objects we passed it:

(defn select-device! [select-element tracks]
  (let [label (-> tracks first (.-label))
        index (->> (.-options select-element)
                   (zipmap (range))
                   (some (fn [[i option]]
                           (and (= label (.-text option)) i))))]
    (when index
      (println "Setting selected video source to index" index)
      (set! (.-selectedIndex select-element) index))))

In this case, that will be the video select element and the video tracks from the default camera.

The tracks are labelled with the name of the device they correspond to:

(comment

  (->> (.getVideoTracks @active-stream)
       (map #(.-label %)))
  ;; => ("Integrated Camera (04f2:b6ea)")

  )

Which are the same names we used to populate our video select options:

(comment

  (->> (.-options video-select)
       (map #(.-text %)))
  ;; => ("Integrated Camera (04f2:b6ea)"
  ;;     "UVC Camera (046d:0823) (046d:0823)")

  )

To select an option, we need to set the selectedIndex property of the select element to the index corresponding to the option we want. We can turn the list of options into a map of index to option using zipmap, which takes a list of keys and a list of values and returns a map with the keys mapped to the corresponding values:

(comment

  (->> (.-options video-select)
       (zipmap (range)))
  ;; => {0 #object[HTMLOptionElement [object HTMLOptionElement]]
  ;;     1 #object[HTMLOptionElement [object HTMLOptionElement]]}

  )

Finally, we need to return the index of first option where the value of the text property matches the label we're looking for:

(comment

  (->> (.-options video-select)
       (zipmap (range))
       (some (fn [[i option]]
               (and (= label (.-text option)) i)))))
  ;; => 0

  )

Note that the some function returns the first truthy value return by the predicate function, so we can use a neat little trick to return the index:

(and (= label (.-text option)) i)

When the text matches the label, the first clause of the and will be truthy (a literal true), and the second clause, the index, will also be truthy because only false and nil are not truthy in Clojure, and and returns the last truthy value, which is the index, so the return value of some is the index corresponding to the label. Without this trick, we'd have to resort to something like this:

(comment

  (let [label "Integrated Camera (04f2:b6ea)"]
    (->> (.-options video-select)
         (zipmap (range))
         (filter (fn [[i option]]
                   (= label (.-text option))))
         ffirst))
  ;; => 0

  )

I hope we can all agree that this is gross! 🤮

So that's what happens on the initial load of the page. If we have more than one camera, we can select it, which results in set-video-stream! being called again:

(defn set-video-stream! []
  (stop-video!)
  (let [video-source (.-value video-select)
        constraints {:video {:deviceId (when (not-empty video-source)
                                         {:exact video-source})}}]
    (println "Getting media with constraints:" constraints)
    (-> js/navigator.mediaDevices
        (.getUserMedia (clj->js constraints))
        (.then start-video!)
        (.catch log-error))))

This time, the video select element will have a value:

(comment

  (let [video-source (.-value video-select)
        constraints {:video {:deviceId (when (not-empty video-source)
                                         {:exact video-source})}}]
    constraints)
  ;; => {:video
  ;;     {:deviceId
  ;;      {:exact
  ;;       "24705d21befb46ac4b2596716eee02c2fecb819447ef3edb91562aad41d2db50"}}}

  )

Hence .getUserMedia() will return a MediaStream for that specific camera, and then start-video! and the rest of it work as before.

OK, that was a lot! 😅

Audioimmolation

Now that we have video on lockdown, let's see if we can add some sweet sweet audio. We'll start with the HTML:

<!doctype html>
<html class="no-js" lang="">
<!-- ... -->
<body>
  <!-- ... -->
  <div id="container">
    <h1>cljcastr</h1>
    <div id="sources">
      <div class="select">
        <label for="videoSource">Video source:</label>
        <select id="videoSource"></select>
      </div>
      <div class="select">
        <label for="audioSource">Audio source:</label>
        <select id="audioSource"></select>
      </div>
    </div>
    <video autoplay muted playsinline></video>
  </div>
</body>

</html>

Note that we're wrapping the two .select divs in another div. In our style.css, we can move the margin down to this div instead of directly on the <select> elements:

div#sources {
  margin-bottom: 10px;
}

If we refresh the page, we'll now see a select element labelled "Audio source" pop up.

Now back to cljcastr.cljs! First we add a binding for the audio select to the top of the file alongside the video select:

(ns cljcastr
  (:require [clojure.string :as str]))

(def video-element (.querySelector js/document "video"))
(def video-select (.querySelector js/document "select#videoSource"))
(def audio-select (.querySelector js/document "select#audioSource"))

Now, let's walk through the UI flow, starting with load-ui!, and see where to sprinkle in audio stuff:

(defn load-ui! []
  (set! (.-onchange video-select) set-video-stream!)
  (-> (set-video-stream!)
      (.then get-devices)
      (.then log-devices)
      (.then populate-device-selects!)))

Digging into set-video-stream!, it looks like we can grab the audio source in exactly the same way as we do the video one, so let's add that in:

(defn set-video-stream! []
  (stop-video!)
  (let [audio-source (.-value audio-select)
        video-source (.-value video-select)
        constraints {:audio {:deviceId (when (not-empty audio-source)
                                         {:exact audio-source})}
                     :video {:deviceId (when (not-empty video-source)
                                         {:exact video-source})}}]
    (println "Getting media with constraints:" constraints)
    (-> js/navigator.mediaDevices
        (.getUserMedia (clj->js constraints))
        (.then start-video!)
        (.catch log-error))))

We should also rename the function, now that it's responsible for audio as well. set-media-stream! seems like a pretty decent name, so let's go for that! Whilst we're at the renaming, we can rename stop-video! to stop-media! as well. The contents of the function itself look pretty good, except the log statement, so we can fix that:

(defn stop-media! []
  (when @active-stream
    (println "Stopping currently playing media")
    (doseq [track (.getTracks @active-stream)]
      (.stop track))))

If we keep going in set-media-stream!, the .getUserMedia() call is fine, since we've added an audio constraint. The next thing that happens is the call to start-video!, which we can rename to start-media! and then have a look at:

(defn start-media! [stream]
  (reset! active-stream stream)
  (select-device! video-select (.getVideoTracks stream))
  (set! (.-srcObject video-element) stream))

It looks like we can use select-device! to handle the audio as well, so let's try that out:

(defn start-media! [stream]
  (reset! active-stream stream)
  (select-device! audio-select (.getAudioTracks stream))
  (select-device! video-select (.getVideoTracks stream))
  (set! (.-srcObject video-element) stream))

OK, it looks like we're in pretty good shape in set-media-stream! now, so let's keep walking through load-ui!:

(defn load-ui! []
  (set! (.-onchange video-select) set-media-stream!)
  (-> (set-media-stream!)
      (.then get-devices)
      (.then log-devices)
      (.then populate-device-selects!)))

Next up after the call to set-media-stream! is the call to get-devices, so let's dig in there:

(defn get-devices []
  (.enumerateDevices js/navigator.mediaDevices))

That looks pretty reasonable, so let's look at the final function called from load-ui!, which is populate-device-selects!.

(defn populate-device-selects! [devices]
  (doseq [device
          (->> devices
               (filter #(= "videoinput" (.-kind %)))
               (log-devices "Populating video inputs with devices"))]
    (let [option (.createElement js/document "option")]
      (set! (.-value option) (.-deviceId device))
      (set! (.-text option)
            (or (.-label device)
                (str "Camera " (inc (.-length video-select)))))
      (.appendChild video-select option))))

Yikes! 😱 Looks like we have a little refactoring to do here. After diving into the closest phonebooth (they still have those, right?) to replace our nerdy glasses with our LISP superhero cape, we can do a top-down design move and rewrite the function the way we wish it worked:

(defn populate-device-selects! [devices]
  (populate-device-select! audio-select (audio-devices devices))
  (populate-device-select! video-select (video-devices devices)))

Looks quite nice, doesn't it? Given this, let's write populate-device-select!:

(defn populate-device-select! [select-element devices]
  (let [select-label (->> (.-labels select-element) first .-textContent)]
    (doseq [device (log-devices (str "Populating options for " select-label) devices)]
      (let [option (.createElement js/document "option")]
        (set! (.-value option) (.-deviceId device))
        (set! (.-text option) (.-label device))
        (.appendChild select-element option)))))

It's nice to specify in the log output which select we're populating, and since we specified a label in our HTML:

<label for="videoSource">Video source:</label>
<select id="videoSource"></select>

we can access the label through the labels property on the select element. Since we know that we only have one label, we can take the first one and grab the value of its textContent property:

(let [select-label (->> (.-labels select-element) first .-textContent)]
  ;; ...
  )

Pretty neat!

OK, now that we have populate-device-select!, the last two functions we need to write are audio-devices and video-devices. Well, it turns out that we've more or less already written video-devices in the original populate-device-selects! code:

(->> devices
     (filter #(= "videoinput" (.-kind %)))
     (log-devices "Populating video inputs with devices"))

Let's transform this into a function:

(defn video-devices [devices]
  (filter #(= "videoinput" (.-kind %)) devices))

Given this, writing audio-devices is just some copy / paste / query-replace:

(defn audio-devices [devices]
  (filter #(= "audioinput" (.-kind %)) devices))

We can test this out:

(comment

  (doseq [f [audio-devices video-devices]]
    (-> (get-devices)
        (.then (comp log-devices f))))
  ;; => #object[Promise [object Promise]]

  )

We should now see something like this in the JavaScript console:

audioinput:
  Default (default)
  Tiger Lake-LP Smart Sound Technology Audio Controller Digital Microphone (94edc85f1f91926d1e9f9da6995188d6263dee15e8a45a6d1add28f64f74c13b)
  HD Webcam B910 Analog Stereo (f78a32bacbfbe6ffe238b0d3b046f11bf4ed8e5ad8ce6cf25f18d431be3cd9af)
  Yeti X Analog Stereo (5af7607e641d0c8061291e648a5bec4958a588147bf0ffcc61a1ef5f2afb6cb6)
videoinput:
  Integrated Camera (04f2:b6ea) (d9862f4684c6b3f21bf95436a09b58dfa1b7a442e79aff225314e5e9bab45217)
  UVC Camera (046d:0823) (046d:0823) (24705d21befb46ac4b2596716eee02c2fecb819447ef3edb91562aad41d2db50)

Amazing!

At this point, we should be able to call load-ui! and see both the audio and video sources in their respective select dropdowns:

(comment

  (load-ui!)
  ;; => #object[Promise [object Promise]]

  )

Screenshot of a browser window showing a selection box containing four audio input sources

As a brief aside, I'm annoyed by that 404 (Not Found) when trying to get favicon.ico, so let's fix that, using lessons learned in Hacking the blog: favicon!

An iconic favicon

We'll just pop over to RealFaviconGenerator and supply a logo such as this one:

A microphone with the Clojure logo for the top part

Then download the favicon package and unzip it into our webserver root:

$ cd public
$ unzip ~/Downloads/cljcastr-favicon_package_v0.16.zip
Archive:  /home/jmglov/Downloads/cljcastr-favicon_package_v0.16.zip
  inflating: android-chrome-192x192.png
  inflating: mstile-150x150.png
  inflating: favicon-16x16.png
  inflating: safari-pinned-tab.svg
  inflating: favicon.ico
  inflating: site.webmanifest
  inflating: android-chrome-512x512.png
  inflating: apple-touch-icon.png
  inflating: browserconfig.xml
  inflating: favicon-32x32.png

And finally, drop this goodness into the <head> section of public/index.html:

<head>
    <!-- ... -->
    <link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
    <link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png">
    <link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png">
    <link rel="manifest" href="/site.webmanifest">
    <link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5">
    <meta name="msapplication-TileColor" content="#da532c">
    <meta name="theme-color" content="#ffffff">
    <!-- ... -->
</head>

Reloading the cljcastr page should now show a delightful little icon in the tab.

Hey Mr. Selector

The only thing lacking at this point is adding an onchange event handler to the audio select. We can do that in load-ui!, and then we might as well call load-ui! on page load whilst we're at it:

(defn load-ui! []
  (set! (.-onchange audio-select) set-media-stream!)
  (set! (.-onchange video-select) set-media-stream!)
  (-> (set-media-stream!)
      (.then get-devices)
      (.then log-devices)
      (.then populate-device-selects!)))

(load-ui!)

Evaluating the buffer results in seeing ourselves, and we seem to be able to switch video and audio sources happily, but there's one annoying thing happening when switching audio source. Quoth the JS console:

Getting media with constraints:
  {:audio
   {:deviceId
    {:exact
     5af7607e641d0c8061291e648a5bec4958a588147bf0ffcc61a1ef5f2afb6cb6}},
   :video
   {:deviceId
    {:exact
     24705d21befb46ac4b2596716eee02c2fecb819447ef3edb91562aad41d2db50}}}
Setting selected video source to index 3
Setting selected video source to index 1

It looks like that first "video source" is actually an audio source. 😬

Looking at select-device!, it's obvious why this is:

(defn select-device! [select-element tracks]
  (let [label (-> tracks first (.-label))
        index (->> (.-options select-element)
                   (zipmap (range))
                   (some (fn [[i option]]
                           (and (= label (.-text option)) i))))]
    (when index
      (println "Setting selected video source to index" index)
      (set! (.-selectedIndex select-element) index))))

Since we have the select DOM element here, we can use the same trick as in populate-device-select! to get its label. Let's extract that stuff to a function of its very own, then update populate-device-select! and select-device! to use it:

(defn label-for [element]
  (->> (.-labels element) first .-textContent))

(defn populate-device-select! [select-element devices]
  (doseq [device (log-devices (str "Populating options for "
                                   (label-for select-element))
                              devices)]
    ;; ...
    ))

(defn select-device! [select-element tracks]
  (let [
        ;; ...
       ]
    (when index
      (println "Setting index for" (label-for select-element) index)
      (set! (.-selectedIndex select-element) index))))

This looks much better now!

Getting media with constraints:
  {:audio
   {:deviceId
    {:exact
     5af7607e641d0c8061291e648a5bec4958a588147bf0ffcc61a1ef5f2afb6cb6}},
   :video
   {:deviceId
    {:exact
     24705d21befb46ac4b2596716eee02c2fecb819447ef3edb91562aad41d2db50}}}
Setting index for Audio source: 3
Setting index for Video source: 1

Stop, in the name of privacy, before you break my heart

This is all wonderful, but if you're anything like me, you probably don't like your webcam and mic surveilling you when you're not actively using them. Let's add a stop button that shuts down this whole dog and pony show. First, we can add the button in index.html:

  <div id="container">
    <h1>cljcastr</h1>
    <div id="sources">
      <div id="selects">
        <div class="select"> <!-- ... --> </div>
        <div class="select"> <!-- ... --> </div>
      </div>
      <div id="stop">
        <input id="stop" type="button" value="Stop" />
      </div>
    </div>
    <video autoplay muted playsinline></video>
  </div>

Yes, yes, I added another <div>. Listen, I never claimed to actually know what I was doing with this whole new-fangled HTML thing, OK? Back in my day, we had Gopher and counted ourselves lucky! Also, we FTP'd files uphill both ways in a blizzard and so on.

Speaking of not knowing what I'm doing, lemme sprinkle some CSS on the top of this lovely cake:

div#sources {
  display: flex;
  margin-bottom: 10px;
}

div#stop {
  padding-left: 10px;
}

Now that we have a button, let's make it do stuff and things:

(ns cljcastr
  (:require [clojure.string :as str]))

(def video-element (.querySelector js/document "video"))
(def video-select (.querySelector js/document "select#videoSource"))
(def audio-select (.querySelector js/document "select#audioSource"))
(def stop-button (.querySelector js/document "input#stop"))

;; ...

(defn load-ui! []
  (set! (.-onclick stop-button) stop-media!)
  ;; ...
  )

Reload the page, click the button, and watch your face disappear!

Screenshot of a browser window showing a black video screen

And there you have it! A fully functional Zencastr clone!

*Ahem*

Perhaps we're missing recording and connecting to other people and transcription and so on, but those are just bonus features that people don't really need for podcasting, right? Anyway, I'm quite proud of what we accomplished in 106 lines of ClojureScript! 🏆

Permalink

Ep 112: Purify!

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to feedback@clojuredesign.club, or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "pure data, pure simplicity". We loop back to our new approach and find more, and less, than we expected!

Our discussion includes:

  • Sportify reaches the end zone!
  • How can you make software extremely reliable?
  • How can you keep logic pure when it depends on the outcome of I/O?
  • What is the "think, do, assimilate" pattern?
  • What is a "context"? What information should it contain?
  • What is an "operation"? What information should it contain?
  • Why does "structural sharing" matter?
  • When to use multimethods vs case statements.
  • The benefits of reducing functions.
  • Testing with mocks vs data.
  • Why test with real-world data?
  • How does minimizing I/O make your code testable, inspectable, and repeatable?
  • How to support any process conditional.
  • A general way to model any process.

Selected quotes

  • Let's get some Hammock Time. We're big fans of Hammock Time!

  • We make a function for each of those: think, do, assimilate. A function to figure out the next thing, a function to do it, and a function to integrate the results back into the context. ("determine-operation", "execute-operation", and "update-context".)

  • It's not a single point of failure! It's a single point of context.

  • Where you have a binding in a let block, you have a key in a context map. There's a symmetry there.

  • You can make the operation map as big, fat, and thick as you want, so "execute-operation" has 100% of everything it needs for that operation.

  • The "determine-operation" function can decide anything because it has the full context—the full world at its disposal!

  • Clojure has structural sharing, so all the information is cheap. We can keep a bunch of references to it in memory. We're not going to run out of memory if we keep information about every step.

  • The "update-context" is a reducer, so we can make a series of fake results in our test and run through different scenarios using "determine-operation" and "update-context". We're able to test all of our logic in our test cases because we can just pass in different types of data.

  • Your tests are grounded in reality. They're grounded in what has happened.

  • We've aggressively minimized the side effects down to the tiniest thing possible!

  • Data is inert. Mocks are not. Mocks are behavior.

  • You can just literally copy from the exception and put it in your test. There's no need transform it. It is already useful.

  • It's very testable. It's very inspectable. It's very repeatable. It creates a really simple overall loop.

  • You want those I/O implementations so small and dumb that the only way to mess them up is if you're calling the wrong function or you're passing the wrong args. Once it works, it will always work, and you no longer have to test it.

  • We need to build into context every little bit of information we need to know to make a decision.

  • Context takes anything that is implicit and makes it 100% explicit, because you can't get data across the boundaries without putting it in the context. You have no option but to put everything in the context, so you know everything that's going on.

  • We're in this machine, and there's no exit. We're on the freeway, and there's no off-ramp. We're in the infinite loop! How do we know we're done?

  • How do we know we're done?

Permalink

Kotlin Discovered: Variance (again) #Kotlin #TypeDriven

You never touched Groovy, nor did you jump on the Scala train. Clojure never attracted you; and you heard about Ceylon long after the language had already died. You are one of those old-fashioned Java folks! But now, after all those years, you want to join the cool Kotlin kids. So, where to start? Let’s discover the language together by decompiling it to Java code. Today: The things we tend to forget!

Permalink

Clojure 1.12.0-alpha8

Clojure 1.12.0-alpha8 is now available!

New:

  • CLJ-2568 - clojure.walk/walk - preserve metadata on lists and seqs

  • CLJ-2783 - replace calls to deprecated URL constructor

Reverted:

  • CLJ-1162 - deref - improve error message when called on non IDRef

Permalink

ClassNotFoundException: java.util.SequencedCollection

Recently I’ve had users of my libraries start reporting mysterious errors due to a missing reference to SequencedCollection, a Java interface added in JDK 21:

Execution error (ClassNotFoundException) at
jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:641).
java.util.SequencedCollection

Specifically, projects using Jepsen 0.3.5 started throwing this error due to Clojure’s built-in rrb_vector.clj, which is particularly vexing given that the class doesn’t reference SequencedCollection at all.

It turns out that the Clojure compiler, when run on JDK 21 or later, will automatically insert references to this class when compiling certain expressions–likely because it now appears in the supertypes of other classes. Jepsen had :javac-options ["-source" "11" "-target" "11"] in Jepsen’s project.clj already, but it still emitted references to SequencedCollection because the reference is inserted by the Clojure compiler, not javac. Similarly, adding ["--release" "11"] didn’t work.

Long story short: as far as I can tell the only workaround is to downgrade to Java 17 (or anything prior to 21) when building Jepsen as a library. That’s not super hard with update-alternatives, but I still imagine I’ll be messing this up until Clojure’s compiler can get a patch.

Permalink

nREPL 1.1.1: Improved Completion with compliment-lite

Today I’ve released nREPL 1.1.1 with a couple of small bug-fixes and one more notable, if mostly invisible change.

Historically nREPL’s completions op (introduced in nREPL 0.8) used internally a modified version of clojure-complete, that I eventually released as a library named incomplete. clojure-complete was pretty much abadonware at this point and I wanted a simple drop-in replacement that we could use in tools like REPLy (and by association - Leiningen).1

Using the much better compliment library wasn’t an option, as we needed something that was quite literally a single file. (so it’d be easy to inline in nREPL and load in REPLy) Recently, however, compliment’s author Oleksandr Yakushev released exactly what we needed - a single-file, stripped-down version of compliment called compliment-lite. Here’s how it differs from the full-fledged compliment:

  • Context. Completion features that require context (e.g., filtering methods by the class of the receiver object) don’t work.
  • Local bindings and resources. Those two sources of completions completely rely on context, so they are disabled in Compliment-lite.
  • Documentation. The documentation function is absent from Compliment-lite.

None of those were things supported by incomplete, so nothing lost. Naturally, it made sense for nREPL to adopt it and offer improved out-of-the-box completion experience to Clojure programmers. As this doesn’t affect the public API of nREPL2 I’ve opted to ship this change in a “bug-fix” release. A version like 1.1.1 is just too cool to pass up!

In light of the creation of compliment-lite I now consider incomplete to be obsolete and I’d encourage everyone looking for a simple, but capable code-completion library to go for compliment-lite instead.

That’s all I have for you today. Keep hacking!

Permalink

PG2 release 0.1.3: Next.JDBC-compatible API

Table of Content

PG2 version 0.1.3 is out. One of its new features is a module which mimics Next.JDBC API. Of course, it doesn’t cover 100% of Next.JDBC features yet most of the functions and macros are there. It will help you to introduce PG2 into the project without rewriting all the database-related code from scratch.

Obtaining a Connection

In Next.JDBC, all the functions and macros accept something that implements the Connectable protocol. It might be a plain Clojure map, an existing connection, or a connection pool. The PG2 wrapper follows this design. It works with either a map, a connection, or a pool.

Import the namespace and declare a config:

(require '[pg.jdbc :as jdbc])

(def config
  {:host "127.0.0.1"
   :port 10140
   :user "test"
   :password "test"
   :dbname "test"})

Having a config map, obtain a connection by passing it into the get-connection function:

(def conn
  (jdbc/get-connection config))

This approach, although is a part of the Next.JDBC design, is not recommended to use. Once you’ve established a connection, you must either close it or, if it was borrowed from a pool, return it to the pool. There is a special macro on-connection that covers this logic:

(jdbc/on-connection [bind source]
  ...)

If the source was a map, a new connection is spawned and gets closed afterwards. If the source is a pool, the connection gets returned to the pool. When the source is a connection, nothing happens when exiting the macro.

(jdbc/on-connection [conn config]
  (println conn))

A brief example with a connection pool and a couple of futures. Each future borrows a connection from a pool, and returns it afterwards.

(pool/with-pool [pool config]
  (let [f1
        (future
          (jdbc/on-connection [conn1 pool]
            (println
             (jdbc/execute-one! conn1 ["select 'hoho' as message"]))))
        f2
        (future
          (jdbc/on-connection [conn2 pool]
            (println
             (jdbc/execute-one! conn2 ["select 'haha' as message"]))))]
    @f1
    @f2))

;; {:message hoho}
;; {:message haha}

Executing Queries

Two functions execute! and execute-one! send queries to the database. Each of them takes a source, a SQL vector, and a map of options. The SQL vector is a sequence where the first item is either a string or a prepared statement, and the rest values are parameters.

(jdbc/on-connection [conn config]
  (jdbc/execute! conn ["select $1 as num" 42]))
;; [{:num 42}]

Pay attention that parameters use a dollar sign with a number but not a question mark.

The execute-one! function acts like execute! but returns the first row only. Internaly, this is done by passing the {:first? true} parameter that enables the First reducer.

(jdbc/on-connection [conn config]
  (jdbc/execute-one! conn ["select $1 as num" 42]))
;; {:num 42}

To prepare a statement, pass a SQL-vector into the prepare function. The result will be an instance of the PreparedStatement class. To execute a statement, put it into a SQL-vector followed by the parameters:

(jdbc/on-connection [conn config]
  (let [stmt
        (jdbc/prepare conn
                      ["select $1::int4 + 1 as num"])
        res1
        (jdbc/execute-one! conn [stmt 1])

        res2
        (jdbc/execute-one! conn [stmt 2])]

    [res1 res2]))

;; [{:num 2} {:num 3}]

Above, the same stmt statement is executed twice with different parameters.

More realistic example with inserting data into a table. Let’s prepare the table first:

(jdbc/execute! config ["create table test2 (id serial primary key, name text not null)"])

Insert a couple of rows returning the result:

(jdbc/on-connection [conn config]
  (let [stmt
        (jdbc/prepare conn
                      ["insert into test2 (name) values ($1) returning *"])

        res1
        (jdbc/execute-one! conn [stmt "Ivan"])

        res2
        (jdbc/execute-one! conn [stmt "Huan"])]

    [res1 res2]))

;; [{:name "Ivan", :id 1} {:name "Huan", :id 2}]

As it was mentioned above, in Postgres, a prepared statement is always bound to a certain connection. Thus, use the prepare function only inside the on-connection macro to ensure that all the underlying database interaction is made within the same connection.

Transactions

The with-transaction macro wraps a block of code into a transaction. Before entering the block, the macro emits the BEGIN expression, and COMMIT afterwards, if there was no an exception. Should an exception pop up, the transaction gets rolled back with ROLLBACK, and the exception is re-thrown.

The macro takes a binding symbol which a connection is bound to, a source, an a map of options. The standard Next.JDBC transaction options are supported, namely:

  • :isolation
  • :read-only
  • :rollback-only

Here is an example of inserting a couple of rows in a transaction:

(jdbc/on-connection [conn config]

  (let [stmt
        (jdbc/prepare conn
                      ["insert into test2 (name) values ($1) returning *"])]

    (jdbc/with-transaction [TX conn {:isolation :serializable
                                     :read-only false
                                     :rollback-only false}]

      (let [res1
            (jdbc/execute-one! conn [stmt "Snip"])

            res2
            (jdbc/execute-one! conn [stmt "Snap"])]

        [res1 res2]))))

;; [{:name "Snip", :id 3} {:name "Snap", :id 4}]

The Postgres log:

BEGIN
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
insert into test2 (name) values ($1) returning *
  $1 = 'Snip'
insert into test2 (name) values ($1) returning *
  $1 = 'Snap'
COMMIT

The :isolation parameter might be one of the following:

  • :read-uncommitted
  • :read-committed
  • :repeatable-read
  • :serializable

To know more about transaction isolation, refer to the official [Postgres documentation][transaction-iso].

When read-only is true, any mutable query will trigger an error response from Postgres:

(jdbc/with-transaction [TX config {:read-only true}]
  (jdbc/execute! TX ["delete from test2"]))

;; Execution error (PGErrorResponse) at org.pg.Accum/maybeThrowError (Accum.java:207).
;; Server error response: {severity=ERROR, message=cannot execute DELETE in a read-only transaction, verbosity=ERROR}

When :rollback-only is true, the transaction gets rolled back even there was no an exception. This is useful for tests and experiments:

(jdbc/with-transaction [TX config {:rollback-only true}]
  (jdbc/execute! TX ["delete from test2"]))

The logs:

statement: BEGIN
execute s1/p2: delete from test2
statement: ROLLBACK

The table still has its data:

(jdbc/execute! config ["select * from test2"])

;; [{:name "Ivan", :id 1} ...]

The function active-tx? helps to determine if you’re in the middle of a transaction:

(jdbc/on-connection [conn config]
  (let [res1 (jdbc/active-tx? conn)]
    (jdbc/with-transaction [TX conn]
      (let [res2 (jdbc/active-tx? TX)]
        [res1 res2]))))

;; [false true]

It returns true for transactions tha are in the error state as well.

Keys and Namespaces

The pg.jdbc wrapper tries to mimic Next.JDBC and thus uses kebab-case-keys when building maps:

(jdbc/on-connection [conn config]
  (jdbc/execute-one! conn ["select 42 as the_answer"]))

;; {:the-answer 42}

To change that behaviour and use snake_case_keys, pass the {:kebab-keys? false} option map:

(jdbc/on-connection [conn config]
  (jdbc/execute-one! conn
                     ["select 42 as the_answer"]
                     {:kebab-keys? false}))

;; {:the_answer 42}

By default, Next.JDBC returns full-qualified keys where namespaces are table names, for example :user/profile-id or :order/created-at. At the moment, namespaces are not supported by the wrapper.

For more information, please refer to the official README file.

Permalink

Configuring fixed/tonsky indentation in clojure-mode

A few years ago Nikita Tonsky made popular a certain style of formating Clojure code, that became known as “fixed” or “tonsky” indentation in the community.

clojure-mode has long had some support for this via clojure-indent-style:

(setq clojure-indent-style 'always-indent)

However, it was kind of hard to get exactly the same indentation from Nikita’s article, as there was no way to suppress the usage of indent specs and forms starting with a keyword were indented by separate rules from what was set by clojure-indent-style. A recent upstream change made the indentation configuration more granular and now you can get fixed indentation with the following snippet:

(setq clojure-indent-style 'always-indent
      clojure-indent-keyword-style 'always-indent
      clojure-enable-indent-specs nil)

clojure-indent-keyword-style and clojure-enable-indent-specs are the new additions, that made this possible. Here’s how clojure-indent-keyword-style can be used:

  • always-align (default) - All args are vertically aligned with the first arg in case (A), and vertically aligned with the function name in case (B).

     (:require [foo.bar]
               [bar.baz])
     (:require
      [foo.bar]
      [bar.baz])
    
  • always-indent - All args are indented like a macro body.

      (:require [foo.bar]
         [bar.baz])
      (:x
         location
         0)
    
  • align-arguments - Case (A) is indented like always-align, and case (B) is indented like a macro body.

      (:require [foo.bar]
                [bar.baz])
      (:x
         location
         0)
    

By the way, clojure-ts-mode also supports the fixed indentation style:

(setq clojure-ts-indent-style 'fixed)

The configuration here is a lot simpler, as this functionality existed since day 1.

For the record, I still don’t endorse/like fixed indentation1 (and funny enough - neither does Arne, who implemented those changes). Still, I acknowledge that this style is somewhat popular in the Clojure community and I’m all for making it easier for people who like/need it to be able to use it.

That’s all I have for you today. Keep hacking!

Permalink

Names, Values, Identities, States and Time

“No man can cross the same river twice, because neither the man nor the river are the same.”

Heraclitus

The following post is extracted & paraphrased from Rich Hickey's excellent Are We There Yet? - specifically the section of the talk that focuses on the model for discussing and thinking about the titular concepts. These concepts are in turn taken from the philosophical writings of Alfred North Whitehead (a co-author of Principia Mathematica).

I often find myself wanting to explain this core concept to people who are new to Clojure, and particularly people who I am trying to make into people who are new to Clojure. While I think I have a good handle on this concept in my head - I sometimes struggle to explain it succinctly, hopefully this post achieves that goal.

Definitions

These definitions are not really globally applicable, but they represent the precise meaning I try to capture when discussing values changing over time in the context of software development and programming.

Value

A value is some measurable quantity, magnitude or amount for which its equivalence to some other value can be determined. A value can also be some immutable composite of other values. The number 42 is a value, the string "Hello World" is a value, the list (1, 2, 3) is also a value.

Identity

An identity is some composite psychological concept we apply to a succession of values over time where they are in some way causally related. You are an identity, I am an identity, a river is an identity (see below).

Name

A name is simply a label that we apply to an identity or a value to make it easier to refer to. The same identity can have many names. "Glen", "Glen Mailer", "Mr Mailer", "Glenjamin" and "@glenathan" are all names which could legitimately be considered to refer to the identity that is myself in the appropriate context. Likewise the "Answer to the Ultimate Question of Life, The Universe, and Everything" is a name for the value 42.

State

A state is simply the value of an identity at a particular time. A snapshot, if you will. Under this definition state does not change, and thus there is no such thing as "mutable state".

Time

Purely relative, it has no dimensions - it can only tell us whether something happened before or after some other thing (or at the same time).

The River

Let us consider the title quote in the context of these definitions. To help us examine the proverbial river under this light, we shall give ourselves the same powers as when running a computer program but in the real world - which requires us to sidestep some fairly fundamental physics - hopefully this will not cause any lasting damage.

The third-longest river in Asia runs through China. Depending on context it is known as the "Yellow River", "Huang He", "རྨ་ཆུ།", "the cradle of Chinese civilization" and "China's Sorrow". All of these are names for the same river, which itself is an identity.

If we were to freeze an instant in time into a snapshot of our proverbial river crossing, this state would contain a value composed of a large number of atomic (in the irreducible sense) smaller values. For simplicity, lets assume that water molecules are immutable. In this case then the state of the river we are crossing can be said to be the current arrangement of all these immutable water molecule values.

At some point in the future when returning for our second crossing, we take another snapshot of the river as our new state. The river's value is again the arrangement of all the immutable water molecules - but this time they are all different molecules with different values.

The state of the identity which is the river named "Huang He" at this later point in time is measurably different from the value we took during the first crossing.

In Clojure

Since immutability is at it's core, we'll start here for some code examples.

The following code should work correctly when pasted into a running Clojure REPL.

In JavaScript

JavaScript doesn't have the same set of immutable primitives, but we can achieve a similar effect with a little sleight of hand.

The following code should work correctly when pasted into the browser console or a Node.js REPL line-by-line.

Summary

The ancient Greeks knew about the perils of mutable state a long time ago - we're only now rediscovering this for ourselves.

In a language like Clojure, that was designed from the ground up with this in mind, it's easy to take back control and tease apart the separate concepts I've described. Even in a language like JavaScript, designed in a week at a time when mutability was commonplace, we can achieve a similar effect with a measure of self-control. There are also libraries like mori and immutable-js which provide much fuller implementations of the data-types required to avoid mutability.

If you remain unconvinced, I recommend watching Are We There Yet?. If you're still not sure after that, you're either a far better programmer than me, or you're yet to experience the pain of trying to reason about a large codebase riddled with unchecked mutability.

Addendum

As well as the above definitions, Are We There Yet? contains this gem, which is Rich visualising the idea of obvious complexity while saying "Grrrrr".









Permalink

Ep 111: Loopify!

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to feedback@clojuredesign.club, or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "trying again". We throw our code in a loop, and it throws us for a loop.

Our discussion includes:

  • Sportify continues!
  • When is it time to stop developing?
  • How do we handle retries?
  • What if you need to recur from catch?
  • How do we recover mid-process?
  • Where should the recovery logic go?
  • Is there a way to get all the critical context at the same level?
  • What should you preserve across a recur?
  • What does it mean to be "loop native"?
  • What is the basic structure for any automation?
  • What is a "single application state"?

Selected quotes

  • It's a lot like having a project on a workbench. You have all of the tools and all the information laid out before you on that workbench. Nothing is tucked in a drawer or inside a cabinet.

  • That's a very important lesson for any developer: you can always stop—at least after it's working.

  • Nothing in the world is solved except by adding another level of abstraction.

  • I was not expecting that level of mutation! I was expecting a Kafka log written in stone!

  • The positive is it has everything. The negative is it has everything.

  • We would like more loop-native code inside of our cloud-native application.

  • Are you suggesting that just because we can, it doesn't mean we should? We're programmers! If the language lets us do it, it must be a good idea!

  • One of the reasons why I like Clojure is because it specifically tells me that I can't do some things that are bad to do.

  • All of the context is in one map. It has everything in it. One map to rule them all!

  • Might this be the fabled "single application state"?

  • We have the thinking function, the doing function, and the assimilate function.

Permalink

Clojure Deref (Feb 15, 2024)

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem (feed: RSS). Thanks to Anton Fonarev for link aggregation.

Blogs, articles, and projects

Libraries and Tools

New releases and tools this week:

  • asdf-cljstyle - cljstyle plugin for the asdf version manager

  • clj-kondo 2024.02.12 - Static analyzer and linter for Clojure code that sparks joy

  • biff 1.8.0 - A Clojure web framework for solo developers

  • edamame 1.4.25 - Configurable EDN/Clojure parser with location metadata

  • calva 2.0.411 - Clojure & ClojureScript Interactive Programming for VS Code

  • datalevin 0.8.26 - A simple, fast and versatile Datalog database

  • splint 1.13 - A Clojure linter focused on style and code shape

  • http-client 0.4.16 - HTTP client for Clojure and Babashka built on java.net.http

  • nbb 1.2.182 - Scripting in Clojure on Node.js using SCI

  • joyride 0.0.43 - Making VS Code Hackable like Emacs since 2022

  • guardrails 1.2.1 - Efficient, hassle-free function call validation with a concise inline syntax for clojure.spec and Malli

  • cli 0.8.56 - Turn Clojure functions into CLIs!

  • overarch 0.9.0 - A data driven description of software architecture based on UML and the C4 model

  • squint 0.6.92 - Light-weight ClojureScript dialect

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.