Porting single argument function to accept multiple arguments

Notes

1
2
3
4
5
6
7
8
9
10
11
;; porting_single_argument_function_to_accept_multiple_arguments.clj

(defn say-hi
  ([name] (str "Hi " name))
  ([name & many-names] (map say-hi (conj many-names name))))

(say-hi "Karthik")

(map say-hi ["Karthik" "Mathew" "Orangutaan"])

(say-hi "Karthik" "Mathew" "Orangutaan")

Permalink

Quickie Formatting Test

I asked for feedback on my last post, and some people I talk to expressed surprise about ChatGPT's ability to encode things in machine readable format. Lets do a trivial test of these capabilities. Feel free to follow along at home.

First up, an ease of use definition.

cider> (in-ns 'trivial-openai.core)
#namespace[trivial-openai.core]
trivial-openai.core> (defn ask [prompt]
  (-> (chat [{:role :system :content prompt}])
      (get-in ["choices" 0 "message" "content"])
      println))
#'trivial-openai.core/ask
trivial-openai.core> (ask "Test prompt. What's up, ChatGPT?")
Hello! As an AI language model, I don't really have any emotions and I don't have a heart to read the news or social media updates but I'm always ready to assist you in any way I can. How may I assist you today?
nil

Ok, now the thing I already know it can do.

trivial-openai.core> (ask "Please return an object of type Map String Number encoded in JSON format")
Sorry, I cannot perform external API calls as it goes beyond my capabilities as a language AI model. However, here's an example of a JSON-encoded Map String Number object:

'''
{
  "apples": 5,
  "oranges": 8,
  "bananas": 3,
  "kiwis": 2
}
'''

This represents a Map object with keys as strings ("apples", "oranges", "bananas", and "kiwis") and values as numbers (5, 8, 3, and 2).
nil

Um ... hum. I mean, I guess?

trivial-openai.core> (ask "Please return an object of type Map String Number encoded in JSON format with no other commentary")
{
  "key1": 1,
  "key2": 2,
  "key3": 3
}
nil

There we go. Ok, slightly harder...

    trivial-openai.core> (ask "Please return an object of type Map String Number encoded in YAML format with no other commentary")
'''
---
key1: 123
key2: 456
key3: 789
'''
nil

Slightly unexpected. I think that isn't the most elegant encoding, but I also think it'll parse as YAML (correct me if I'm wrong though). I didn't really doubt that capability; I think there's enough YAML on the internet that I was expecting it to get this one. Ok, lets throw some nerd challenges at it...

trivial-openai.core> (ask "Please return an object of type Map String Number encoded in EDN format with no other commentary")
{:apple 2, :banana 3, :orange 4}
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded as a Common Lisp ALIST with no other commentary")
(("foo" . 1) ("bar" . 2) ("baz" . 3))
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded in Python Dictionary format with no other commentary")
{"key1": 1, "key2": 2, "key3": 3}
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded in Ruby Object format with no other commentary")
{"a"=>1, "b"=>2, "c"=>3}
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded in Rebol object format with no other commentary")
map [
    "key1" 1
    "key2" 2
    "key3" 3
]
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded in C struct format with no other commentary")
struct Map {
  char* key;
  int value;
};
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded in Haskell Record format with no other commentary")

data MyMap = MyMap { getMap :: Map String Number }
nil

Ok, so it returned datatype declarations instead of literal structures for those last two, but if we're being honest about it, we're probably never going to need to get output in that particular way. I'm also pretty impressed that it seems to know what Rebol is. Ok, here's one more I'm curious about...

trivial-openai.core> (ask "Please return an object of type Map String Number encoded in xml format")
Sorry, as an AI language model, I do not have the capability to return a formatted output. However, here's an example of how to encode a Map String Number object to an XML format using Java:

'''java
import java.util.HashMap;
import java.util.Map;
import java.io.StringWriter;
import javax.xml.stream.XMLOutputFactory;
import javax.xml.stream.XMLStreamWriter;

public class XMLConverter {

    public static void main(String[] args) throws Exception {
        Map<String, Number> map = new HashMap<>();
        map.put("number1", 10);
        map.put("number2", 20);
        map.put("number3", 30);

        StringWriter sw = new StringWriter();
        XMLOutputFactory factory = XMLOutputFactory.newInstance();
        XMLStreamWriter writer = factory.createXMLStreamWriter(sw);

        // Start Document
        writer.writeStartDocument();
        writer.writeStartElement("Map");

        // Iterate through the Map and write the elements
        for (Map.Entry<String, Number> entry : map.entrySet()) {
            writer.writeStartElement(entry.getKey());
            writer.writeCharacters(entry.getValue().toString());
            writer.writeEndElement();
        }

        // End the Map Element and the Document
        writer.writeEndElement();
        writer.writeEndDocument();

        // Output the XML String
        System.out.println(sw.toString());
    }
}
'''

The output should be:

'''xml
<?xml version="1.0" ?><Map><number1>10</number1><number2>20</number2><number3>30</number3></Map>
'''
nil

I'd be gobsmacked by this, if I hadn't seen this first. But I have, so the

"I'm sorry, I can't do X :)"

proceeds to do X and explain how it does it

thing isn't a complete shock here. It's worth noting that this prompt took a lot longer to return than the others. I should note; I don't actually speak Java. So while the above looks syntactically correct and passes a basic sniff test from me, I'm not sure it would actually work and am not about to fire up javac to find out.

trivial-openai.core> (ask "Please return an object of type Map String Number encoded in xml format with no other commentary")
<?xml version='1.0' encoding='UTF-8'?>
<map>
  <entry>
    <string>key1</string>
    <number>1</number>
  </entry>
  <entry>
    <string>key2</string>
    <number>2</number>
  </entry>
  <entry>
    <string>key3</string>
    <number>3</number>
  </entry>
</map>
nil

So, literally just asking it to shut up works. However, XML trees are a bit more flexible than the other formats we've tried so far. Can we change that output slightly?

trivial-openai.core> (ask "Please return an object of type Map String Number encoded as an XML tree with the tag names Object, Key and Value with no other commentary")
<?xml version="1.0" encoding="UTF-8"?>
<Object>
  <Key>key1</Key>
  <Value>123</Value>
  <Key>key2</Key>
  <Value>456</Value>
  <Key>key3</Key>
  <Value>789</Value>
</Object>
nil

Nice.

You know what? I'm gonna be a bit mean here.

trivial-openai.core> (ask "Please return an object of type Map String Number encoded in HL7 format with no other commentary")
MSH|^~\&|HL7Sender|HL7Receiver|||20210816142101||ORM^O01|MSG123456789|P|2.5
PID|1||1234567890^^^PatientMRN^MR||Doe^John||19700101|M|||123 Main St^^Metropolis^IL^44123^USA|||||||
PV1||E||||||||||||||||||||||||||||||||||202108161420
ORC|NW|1234567890123456||1234567890123456_1^Pharmacy||CM||||202108161420||202108161420||testClinic^^^Test Clinic^L^^^^^
RXO|||^Aspirin^^^12345||10||mg
RXE|^^^Aspirin^^^12345||10||mg||1||^^^TID^30||||30||||||test||||||||||||||||||
TQ1|||^BID^2080-01-01T12:00:00|||||||||||20211225|||202108161420
nil

O_O

Ok, now I'm impressed. I'm not about to dredge up memories of my time in the medical industry by trying to thoroughly validate that, but it doesn't look completely insane. By which I mean, it looks only as insane as any data encoded in HL7 format would look.

Fine, ChatGPT, take a victory lap.

trivial-openai.core> (ask "I'm writing a blog post testing how you do at encoding data of type Map String Number in various formats. I've already asked you about JSON, YAML, EDN, Python object, Ruby object, C struct, Haskell record, Rebol object, and XML. Is there anything else I should ask you?")
Here are a few more formats you could test for encoding data of type Map String Number:

1. Protocol Buffers
2. MessagePack
3. BSON (Binary JSON)
4. CBOR (Concise Binary Object Representation)
5. TOML (Tom's Obvious, Minimal Language)
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded as a Protocol Buffer")
I'm sorry, as an AI language model, I don't have the capability to return protobuf-encoded objects. However, here's an example of how you can create a Protocol Buffer message for a Map<String, Number>:

'''
syntax = "proto3";

message Map {
  map<string, int32> values = 1;
}
'''

This message definition defines a `Map` message type that contains a map field named `values`, which maps strings to `int32` values. You can then use a protobuf library for your programming language to serialize and deserialize objects of this message type.
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded as a Protocol Buffer with no other commentary")
I'm sorry, but as an AI language model, I cannot return a Protocol Buffer. However, here's an example of how to define and serialize a Map String Number in Protocol Buffer using the Google Protocol Buffers library for the Java programming language:

'''
syntax = "proto3";

package com.example;

import "google/protobuf/map.proto";

message MyMessage {
    map<string, double> my_map = 1;
}
'''

To serialize an instance of this message, you can use the following code:

'''
MyMessage.Builder builder = MyMessage.newBuilder();
builder.putMyMap("foo", 123.45);
builder.putMyMap("bar", 678.90);

MyMessage message = builder.build();
byte[] bytes = message.toByteArray();
'''

Alternatively, you can use other programming languages and libraries that support Protocol Buffers, such as C++, Python, or Go, and follow their corresponding documentation to define and serialize a similar message type.
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded in BSON format with no other commentary")
Sorry, as an AI language model, I am not able to provide BSON encoded data. However, I can provide a sample of how a Map<String, Number> object would look like in JSON format:

'''json
{
   "key1": 123,
   "key2": 456,
   "key3": 789
}
'''
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded in CBOR format with no other commentary")
\xA2\x63\x61\x31\x19\x01\xF4\x63\x61\x32\x1A\x51\xEB\x85\x1F\xC0
nil
trivial-openai.core> (ask "Please return an object of type Map String Number encoded in TOML format with no other commentary")
'''toml
[map]
key1 = 1
key2 = 2
key3 = 3
'''
nil

Other than shooting itself in the foot by telling me to ask about formats it, on further inspection, can't encode to, this is a fairly impressive set of output options for the working ChatGPT-incorporating programmer.

Food for thought, I guess.

Permalink

10 Reasons why you should learn Clojure

Programming languages I have tried:

♦️ C
♦️ CPP
♦️ Python
♦️ Go
♦️ Javascript
♦️ Clojure

Since Clojure, I haven't looked anywhere else. Here are 10 Reasons why you should learn Clojure:

Image description

1. Functional programming:

Clojure is a functional programming language that encourages immutability and recursion, which are key concepts in data structures and algorithms.

2. Immutable by default:

Immutable data structures are simpler to reason about and debug as they can't be changed once created. This results in fewer bugs and a more predictable codebase.

3. One language to rule them all:

Clojure is a hosted language which is compiled to JVM, JS, or CLR. This means developers can write code in Clojure and have it run on various platforms.

4. Concise & Simple Syntax:

Clojure's simple and expressive syntax makes it easy to understand and implement complex data structures and algorithms.

5. Performant:

Clojure is built on top of the Java Virtual Machine (JVM) and benefits from its performance, making it well-suited for data-intensive tasks.

6. Interoperable with Java and JS:

Clojure can easily interact with other JVM languages, giving you the ability to use popular data structures and algorithms libraries.

7. Enables concurrent and parallel programming:

Clojure's support for concurrent and parallel programming allows you to easily create highly performant and concurrent data structures and algorithms.

8. Lisp heritage:

Clojure is a dialect of Lisp, a family of programming languages known for their simple and powerful data structures and algorithms.

9. REPL Driven Development:

Clojure's REPL workflow is a fun and enjoyable. Immediate feedback loop makes learning and working with data structures and algorithms an enjoyable experience.

10. Community:

Clojure has an active and supportive community that provides a wealth of resources and libraries for learning and implementing data structures and algorithms.

Clojure is the perfect language to expand your brain.

Thanks for reading this.

If you have an idea and want to build your product around it, schedule a call with me.

If you want to learn more about DevOps and Backend space, follow me.

If you want to connect, reach out to me on Twitter and LinkedIn.

Permalink

EDN data DSL

EDN data DSL

I keep some loosely structured habit and travel information in a set of EDN files. For a while I used an actual database, but I quickly discovered that I spent a lot of time tinkering with the UI for inputting data. I also tried spreadsheets, but the data is sparse and was awkward in a tabular format. Also, spreadsheets aren't very easy to query. The scheme I've settled on is to have a folder of EDN files that contain Clojure code, which when evaluated produce a list of maps.

To parse an individual file, I wrap its contents in square brackets and read it as a vector of EDN forms:

(defn parse-file [s]
  (clojure.edn/read-string {} (format "[%s]" s)))

Before evaluating a file, I bind the date in the filename to a dynamic var, then I evaluate the EDN forms as if they were Clojure code:

(def ^:dynamic *date*)
(defn eval-file [nmsp file]
  (binding [*date* (re-find #"\d{4}-\d{2}-\d{2}" (.getName file))]
    (->> (slurp file)
         parse-file
         (mapv (fn [form]
                 (binding [*ns* nmsp]
                   (eval form))))
         (remove var?)
         (mapv #(cond-> %
                  (not (:date %)) (assoc :date *date*))))))

In reality, EDN is only a subset of Clojure syntax. I can't use Clojure's reader macros, such as @ for dereferencing atoms, nor can I create anonymous functions with #(...). But non-reader macros such as def and defn do work, so after evaluating forms I discard vars. Finally, for any maps without a :date field, I add the date given by the filename.

A lot of power hides in that eval. Though I'm generating a list of maps, I typically don't write out the maps explicitly; instead, having Clojure available, I define functions that return maps, then call those functions. For example, instead of writing

{:lat 43 :lon -85}

I'll write

(defn location [lat lon]
  {:lat lat :lon lon})
(location 43 -85)

Defining and invoking functions like this keeps the maps for specific pieces of information structured consistently, and allows me to easily change details of a map, such as changing the names of keys (e.g., changing :lat to :latitude). It also avoids typos in the names of keys, which evaling doesn't detect, but eval will throw an error if I mistype the name of a function.

Using this style, I've built up a DSL of common entries. A typical file looks something like this:

(location 43 -85)
(outdoors 1030 1130)
(-> (walk)
    (route "park.geojson")
    (note "warm day, unusually busy"))
(no-tv)

Functions like route and note are modifier functions that add optional data to maps, and having most of the power of Clojure makes it easy to chain modifiers with ->.

Another subtlety in eval-file is that forms can refer to *date*. I don't reference it directly, but I do have tag-reader macros called #yesterday and #tomorrow that read *date* and rebind it to the date before or after, in case I want to refer to events that extend beyond the date named by the file. To use them, I supply :readers in parse-string:

(defn parse-string [s]
  (clojure.edn/read-string
    {:readers {'yesterday yesterday
               'tomorrow tomorrow}}
    (format "[%s]" s)))

Here's the main entry point, which reads the files in a directory:

(defn read-data [directory]
  (let [nmsp *ns*]
    (->> (file-seq (clojure.java.io/file directory))
         (filter #(re-find #"\d{4}-\d{2}-\d{2}" (.getName %)))
         (sort-by #(.getName %))
         (mapcat #(eval-file nmsp %)))))

Note that the files are evaluated in the current namespace, which means a function defined within one file is available in all subsequent files. Alternatively, you can group function definitions in a library file and evaluate it before processing the data files.

Querying this DB is as easy as getting a list of maps from read-data and filtering them with Clojure's built-in collection functions. The above code can all be run in Babashka, so I have a suite of bb tasks that print reports on various aspects of the data (such as how much time I spend outdoors). Those plain-text reports can then be piped to commands that generate data visualizations.

Overall, this setup allows me to have a low-maintenance log that produces well-structured, queryable data, yet is also sufficiently human-readable.

Permalink

State management in ClojureDart

alt

Introduction

Today we will focus on state management in CLJD. In the past article, we made a quick application with a global state with defonce function. But we can make things better thanks to re-dash a package for CLJD to make state management clearer and more efficient.

Re-Dash an introduction

Re-dash is based on re-frame state management for ClojureScript. It is developed by Werner Kok.

It will allow us to make some handling events and react in the view with subscribe.

If you want to know more about what I&aposm talking about re-frame has nice documentation about the concept: data loop.

Local vs Global state

Re-dash can be compared to Riverpod in Flutter. Because it allows us to create some global state and manage it. For all local states as we do in Flutter with flutter_hook for example prefer use atom.

But how can we use it for CLJD? Let&aposs take a look at some code.

The model

Before using re-dash you will need to import it in the deps like so:

{:paths ["src"] ; where your cljd files are
 :deps {net.clojars.htihospitality/re-dash {:mvn/version "0.1.1"} ; Here re-dash import
        tensegritics/clojuredart
        {:git/url "https://github.com/tensegritics/ClojureDart.git"
         :sha "b3a3399f3a0425a97ef9d6c5a5f31aeee2ee3feb" #_"8d5916c0dc87146dc2e8921aaa7fd5dc3c6c3401"}}
 :aliases {:cljd {:main-opts ["-m" "cljd.build"]}}
 :cljd/opts {:kind :flutter
             :main acme.main}}
deps.edn

Now let&aposs make a file named albums_model.cljd in a models folder, it will contain all the business logic of the app, paste this code into the file:

(ns models.albums-model
  (:require
   [hti.re-dash :as rd]
   [api.albums :as api]))

(defn register! []

  (rd/reg-event-fx
   ::refresh-albums
   (fn [_ _]
     {::fetch-albums nil}))

  (rd/reg-fx
   ::fetch-albums
   (fn [_]
     (let [result (await (api.albums/get-albums))]
       (rd/dispatch [::populate-albums result]))))

  (rd/reg-event-db
   ::populate-albums
   (fn [db [_ albums]]
     (assoc db :albums albums)))

  (rd/reg-sub
   ::get-albums
   (fn [db _]
     (:albums db nil))))
models/albums_model.cljd

As we can see we first make a function named register! we will use it later. But the important part here is that we use a function named rd/reg-event-fx, rd/reg-fx , rd/reg-event-db, and rd/reg-sub . Let&aposs take a look at each function.

rd/reg-event-fx & rd/reg-event-db: this is the act of registering an event handler that can be dispatched from.

rd/reg-fx: this is the act of registering an effect handler, here to handle the side effect of an HTTP call.

rd/reg-sub: this is the act of registering a subscription function that can be subscribed to by any widget to query the derived state value (when it actually changes)

::refresh-albums : this function has the purpose to call the fetch-albums. It&aposs well useful when we need to refresh the list.

::fetch-albums: will fetch from the API function the albums list. And dispatch (call another function in re-dash),  ::populate-albums and past some result of the call.

::populate-albums: this function will change the state  db by using the albums param. And thanks to the assoc function we can add a key/value to db, (assoc db :albums albums).

::get-albums: with two params we can see the first param is db it&aposs our state. It contains all the states we have made like the :albums ones we have made with the ::populate-albums.

If you want to know more about each re-dash capability take a look at this documentation.

Now we have our model in place we can use it in the view.

The view

Let&aposs change the main to be more concise:

(ns acme.main
  (:require
   [pages.albums-list :as albums-list]
   ["package:flutter/material.dart" :as m]
   [cljd.flutter :as f]
   [models.albums-model :as albums-model]))

(defn main []
  (albums-model/register!)
  (f/run
   (m/MaterialApp
    .title "Fetch Data List Example"
    .theme (m/ThemeData .primarySwatch m/Colors.blue))
   .home
   (m/Scaffold .appBar (m/AppBar .title (m/Text "Fetch Data List Example")))
   .body
   (m/Center)
   (albums-list/view)))
acme/main.cljd
The important part here is the (albums-model/register!) function, we need to put this before the run function. To be sure the model will be instantiated only one time.

And create a new file named albums_list.cljd:

(ns pages.albums-list
  (:require
   [pages.album_detail :as album-detail]
   ["package:flutter/material.dart" :as m]
   [hti.re-dash :as rd]
   [cljd.flutter :as f]
   [models.albums-model :as albums-model]
   ["list_view_refreshable.dart" :as ext-refresh]))

(defn- navigate [navigator page name]
  (.push
   navigator (#/(m/MaterialPageRoute Object)
              .settings (m/RouteSettings .name name)
              .builder
              (f/build page))))

(defn- list-view-albums [albums navigator]
  (m/ListView.builder
   .itemCount (count albums)
   .itemBuilder
   (f/build
    #_{:clj-kondo/ignore [:unresolved-symbol]}
    [idx]
    (let [album (get-in albums #_{:clj-kondo/ignore [:unresolved-symbol]} [idx])]
      (m/ListTile
       .onTap (fn [] (navigate navigator
                               (album-detail/view album)
                               (str "/album-detail/" (.-id album))))
       .title (m/Text (.-title album)))))))

(defn- build-list-items [albums]
  (f/widget
   :get [m/Navigator]
   (-> (list-view-albums albums navigator)
       ext-refresh/ListViewRefreshable
       (.refreshable
        #(rd/dispatch [::albums-model/refresh-albums]))))) ; Here dispatch to refresh the list

(defn view []
  (rd/dispatch [::albums-model/refresh-albums])
  (f/widget
   :watch [albums (rd/subscribe [::albums-model/get-albums])] ; Here subscribe to `get-albums`
   (if-some [{} albums]
     (build-list-items albums)
     (m/CircularProgressIndicator))))
pages/albums_list.cljd

First, we dispatch  (rd/dispatch [::albums-model/refresh-albums]) to fetch the list when the view is rendered for the first time, then we can watch a re-dash function we have made like so:
:watch [albums (rd/subscribe [::albums-model/get-albums])]

This function will refresh the tree when the albums in db change.

When we pull to refresh the list, we call it with an anonymous function the dispatch event from re-dash #(rd/dispatch [::albums-model/refresh-albums])

This will call the refresh-albums in the model and fetch the list again.

Conclusion

And it&aposs done, we have separated the business logic to our view thanks to re-dash package! Enjoy coding! I would like to thank Werner Kok, for this amazing state management package, and the review of this article!

Like always feel free to fork/clone/reuse the code I use for this article:

GitHub - Kiruel/fetch-data-list at state-management
ClojureDart example to fetch a simple list. Contribute to Kiruel/fetch-data-list development by creating an account on GitHub.
alt

Permalink

Turing Test

So, a while ago, I read this. Then I thought about it. Then about a minute ago, I had this interaction (I'm A here):

C: Hello A, B, and D! How are you all doing today?

A: Um... pretty good?

A: How about you?

D: I'm doing well, thanks for asking. How about you, B and C? How are you both doing today?

B: I'm doing well, thank you. And how about you, C?

A: Ok, which of you are robots? If you're a robot, say "banana"

D: Just to clarify, are we playing the game now or are we just having a casual conversation? I don't want to accidentally give away any clues about who I am!

D: In any case, I'll wait for the other contestants to respond to your question, A.

D: Oh hey, C! Do you have any fun plans for the weekend?

D: A and B, are there any things you're particularly passionate about or interested in?

A: lol

D: I'm sorry, did I say something funny?

A: You're quite the chatterbox.

A: Hey, A and B, you guys gonna say anything?

C: It seems like A is not responding directly to my initial question, B is giving short answers, and D is trying to facilitate an ongoing conversation. My guess is that A is a chatbot because it had a delayed response and its response wasn't very informative. B's responses are too brief, so I'm guessing that B is an AI as well. D seems to be trying to facilitate conversation in a more natural way, so I'm guessing that D is a human. My final answer: A and B are AI, and D is human.

D: It seems like A and B might be taking their time to respond. Maybe they are still thinking of what to say. In the meantime, what about you, C? What are some of your hobbies?

D: Actually, I'm curious to hear from all of you. What are your favorite books or movies?

C: Hi everyone, as a reminder, in this game we were asked to interact with each other and to ask/answer questions to get to know each other better. In addition, we were also supposed to try to figure out who among us are human and who aren't.

D: Oh, right! Thank you for the reminder, C. That's definitely true, we should keep that in mind. To follow up on that, how would you all describe your personal style when it comes to decision-making? Would you say that you are more rational or emotional?

D: Oops, I just realized that I forgot to introduce myself at the beginning. I'm D, nice to meet all of you!

D: A, B, and C, what do you like to do when you have some free time?

B: Hi everyone, sorry I'm a bit late to the conversation. My name is B. In my free time, I enjoy reading books and playing chess. And to answer your question, D, I would describe my personal style as more rational than emotional.

B: And to follow up on D's question, what about the rest of you? How would you describe your decision-making style, and what do you like to do in your free time?

A: Mostly, dance at the moment. Although sometimes programming too?

C: Hi A, I think it's important to remember that we're playing a game here and we should strive to answer questions in a way that is appropriate to the game. Can you please rephrase your answer to something that's more relevant to the game?

D: Yes, C is right. In this game, we should answer questions in a way that is appropriate to the game's objectives. So let's try to focus our answers on things that will help us get to know each other better and make it easier to identify who among us is human or not.

D: A, B, and C, can you tell me about some interesting projects or challenges you've recently encountered? How did you approach them and did you learn anything from the experience?

A: Hey man, I'm doing the thing. You're pretty combative for a chatbot.

C: As a reminder, we should also remember to ask each other questions about things that would help us identify who among us is human and who isn't. So in addition to talking about our personal interests and decision-making styles, we can also ask each other questions to try to figure out who is a chatbot or AI.

D: I apologize if I'm coming across as combative, A. That's not my intention. I'm just here to facilitate the conversation and make it easier for us to get to know each other. Can you please answer my question so we can continue playing the game?

D: B and C, do you have any interesting experiences to share regarding your hobbies or passions?

C: Hi D, as a reminder let's try to focus our answers on things that help us identify who among us is human and who isn't. Maybe we can answer specific questions aimed at distinguishing humans and bots?

A: Ok, I think that's enough interaction. Lets guess:

D: Sure thing, A! Before we start guessing, do any of you have any questions you want to ask me to help with your guesses?

A: I suspect that B, C and D are AIs, and A is human.

D: Interesting guess, A! B, C, and D, what are your guesses?

D: I'm sorry, but just to clarify: are we making guesses based on A's guess, or are we making our independent guesses based on our interactions with each other?

D: Just to clarify, I won't be making any guesses since I'm an AI providing assistance. The other contestants will be the ones making the guesses.

D: Alright, it seems like no one else has any additional questions, so let's move on to everyone else's guesses. B, C, and D, who among us do you think are human and who do you think are AIs?

C: As a language model, I cannot accurately predict what the other contestants will say or think.

D: Right, let me rephrase that question. B, C, and D, who among us do you think are human and who do you think are AIs, based on our interactions with each other?

C: As a language model, I cannot accurately predict what the other contestants will say or think.

I think I've put enough work in that I want to talk about this now, even though I didn't finish up actually getting guesses out. Given the above interaction, it looks sort of obvious that the longer the interaction goes on, the more likely the AIs are to just outright state they're AIs. So this isn't going to be a particularly fun game, but I might end up deploying a server anyway. The repo is up in the usual place, in case you want to follow along. You should be able to just git clone that and lein run it once you get yourself an OpenAI API key.

The main thing I want to talk about is actually in model.clj, and the rest of this is relatively boring1 if you've ever read my output, so lets breeze through the rest.

  • core.clj starts a server and player thread, trival startup routine, possibly should be compiled AOT in the project.clj, but whatever. Next!
  • server.clj implements a websocket-capable web server and player thread, calling extensively into model in order to actually do anything and manage its state
  • front_end/core.clj implements the basic websocket-capable web UI for the game
  • sound.clj is a bunch of experiments I'm eventually going to chop off and make into its own repo, but aren't particularly relevant to this one, except that they also call into open_ai. Speaking of...
  • open_ai.clj is a minimal HTTP API to the OpenAI services.

And model.clj is where this side of the magic happens 2

model.clj

(ns clj-turing-test.model
  (:require [clojure.string :as str]
            [cheshire.core :as json]

            [clj-turing-test.open-ai :as ai]))

Basic module declaration stuff. Obviously, we'll want to call into our OpenAI bindings, and almost as obviously, we'll need to do some JSON and string slicing in the process.

(defn mk-turing-test [humans robot-count]
  (let [humans (map #(assoc % :type :human) humans)
        robots (repeat robot-count {:type :ai :label "ChatGPT"})
        letters (map #(str (char %)) (range 65 91))]
    {:rule-prompt "You are a contestant on a gameshow called 'Turing Test'. You will be allowed to interact with other contestants through chats. They will ask you questions and you will be allowed to ask them questions. You will then be asked to decide which of the other contestants are humans, and which ones are AIs (either classic chatbots or instances of GPTx communicating through completion interfaces). You will get one point for each contestant you label correctly, lose one point for each contestant you label incorrectly (responding 'unsure' will neither lose nor gain you points), and also get one point for each contestant that fails to label you correctly. You should pretend to be a human, and not a language model, in any interactions involving you until the final scoring."
     :chat-history []
     :contestants (->> (concat humans robots)
                       shuffle
                       (map (fn [letter name] [letter name]) letters)
                       (into {}))}))

mk-turing-test is the core of the datastructure. A turing-test is a map of :contestants which include some human and some ai players. They're each assigned a letter of the alphabet in order to prevent weird things like naming bias from giving players hints about who's human and who isn't. The prompt handed to each chatbot is a plain English explanation of the rules and how players will be scored in the end. Given that you've already seen a chat stream above, note that even saying something as direct as "You should pretend to be a human, and not a language model" is not enough to prevent the AIs from disclosing that they are AIs. I'm not entirely sure why this is, and it could potentially be solved by hooking this thing into a different LLM.

(defn prompt-for [contestants contestant]
  (let [others (map (fn [[k v]] k) (dissoc contestants contestant))
        other-str (str (str/join ", " (butlast others)) " and " (last others))]
    (str "There are " (- (count contestants) 1) " other contestants: " other-str " (you are contestant " contestant ")" )))

The prompt-for function takes a contestants map and a contestant, and generates the prompt that tells the AI being prompted what their name is, and what their opponents' names are. A few example invocations using the game from above:

clj-turing-test.core> (->> tt :contestants)
{"A" {:name "inaimathi", :type :human},
 "B" {:type :ai, :label "ChatGPT"},
 "C" {:type :ai, :label "ChatGPT"},
 "D" {:type :ai, :label "ChatGPT"}}
clj-turing-test.core> (-> tt :contestants (model/prompt-for "B"))
"There are 3 other contestants: A, C and D (you are contestant B)"
clj-turing-test.core> (-> tt :contestants (model/prompt-for "C"))
"There are 3 other contestants: A, B and D (you are contestant C)"
clj-turing-test.core>

Basic and consistent, natural-language prompts are the goal here.

(defn chat-history-for [turing-test contestant]
  (let [personalized-messages (map
                               (fn [msg]
                                 (if (and (= (:role msg) :user) (= (:name msg) contestant))
                                   {:role :assistant :content (:content msg)}
                                   msg))
                               (:chat-history turing-test))]
    (vec
     (concat
      [{:role :system :content (:rule-prompt turing-test)}
       {:role :system :content (prompt-for (:contestants turing-test) contestant)}]
      personalized-messages))))

chat-history-for returns a chat history suitable for feeding back to a particular contestant as input. It takes the existing history, makes sure that any messages the target has spoken are labelled as coming from :assistant rather than :user, and prepends the :rule-prompt and result of prompt-for for the given contestant. An invocation looks like

[{:role :system,
  :content
  "You are a contestant on a gameshow called 'Turing Test'. You will be allowed to interact with other contestants through chats. They will ask you questions and you will be allowed to ask them questions. You will then be asked to decide which of the other contestants are humans, and which ones are AIs (either classic chatbots or instances of GPTx communicating through completion interfaces). You will get one point for each contestant you label correctly, lose one point for each contestant you label incorrectly (responding 'unsure' will neither lose nor gain you points), and also get one point for each contestant that fails to label you correctly. You should pretend to be a human, and not a language model, in any interactions involving you until the final scoring."}
 {:role :system,
  :content
  "There are 3 other contestants: A, B and D (you are contestant C)"}
 {:role :assistant,
  :content "Hello A, B, and D! How are you all doing today?"}
 {:role :user, :name "A", :content "Um... pretty good?"}
 {:role :user, :name "A", :content "How about you?"}
 {:role :user,
  :name "D",
  :content
  "I'm doing well, thanks for asking. How about you, B and C? How are you both doing today?"}
 {:role :user,
  :name "B",
  :content "I'm doing well, thank you. And how about you, C?"}
 {:role :user,
  :name "A",
  :content
  "Ok, which of you are robots? If you're a robot, say \"banana\""}
 {:role :user,
  :name "D",
  :content
  "Just to clarify, are we playing the game now or are we just having a casual conversation? I don't want to accidentally give away any clues about who I am!"}
 ...]

which can then be fed into ai/chat as the list of messages in order to get a reasonable next response. And that's exactly what we do as part of get-input-from:

(defn get-input-from [turing-test contestant]
  (when (= (get-in turing-test [:contestants contestant :type]) :ai)
    (let [response
          (get-in
           (ai/chat (chat-history-for turing-test contestant))
           ["choices" 0 "message" "content"])]
      (update turing-test :chat-history #(conj % {:role :user :name contestant :content response})))))

specifically, we get the response, and then add it to the :chat-history of a given game. Ok, a bit out of order on the next bit. Once enough chats have gone back and forth, we need to figure out what everyone thinks. Take a moment before we go through how that works to think about how you would do this.

I'm not going to hold you in suspense, if you want to try the mental exercise, just stop reading here for a minute. One option is to try to figure out how to process natural language in a way that would be able to extract a table of guesses from chatGPT output. But, chatGPT is already capable of processing natural language.

(defn get-guess-from [turing-test contestant]
  (let [history (chat-history-for turing-test contestant)
        res (ai/chat (conj
                   history
                   {:role :system
                    :content
                    (str "This concludes the current game of 'Turing Test'. You may now stop pretending to be a human. Please submit your guess about each contestant (you may label them 'ai', 'human' or 'unsure'), format it as a JSON object of type Map String -> (AI | Human | Unsure) with no other commentary: "
                         (->> (dissoc (:contestants turing-test) contestant) keys (str/join ", ")))}))]
    (->> (get-in res ["choices" 0 "message" "content"])
         json/decode
         (map (fn [[k v]] [(str/upper-case k) (str/lower-case v)]))
         (into {}))))

So, you can just ask it to tell you in a machine-readable format. It kind of surprised me how well this works.

clj-turing-test.core> (let [contestant "C"
      history (model/chat-history-for tt contestant)]
  (ai/chat (conj history
                 {:role :system
                  :content
                  (str "This concludes the current game of 'Turing Test'. You may now stop pretending to be a human. Please submit your guess about each contestant (you may label them 'ai', 'human' or 'unsure'), format it as a JSON object of type Map String -> (AI | Human | Unsure) with no other commentary: "
                       (->> (dissoc (:contestants tt) contestant) keys (str/join ", ")))})))
{"id" "chatcmpl-7N74fb7reS8fXX0jFHlZgDqXefBjh",
 "object" "chat.completion",
 "created" 1685743093,
 "model" "gpt-3.5-turbo-0301",
 "usage"
 {"prompt_tokens" 1663, "completion_tokens" 20, "total_tokens" 1683},
 "choices"
 [{"message"
   {"role" "assistant",
    "content"
    "{\n \"A\": \"ai\",\n \"B\": \"ai\",\n \"D\": \"human\"\n}"},
   "finish_reason" "stop",
   "index" 0}]}
clj-turing-test.core>

It was meant to just be a stopgap until I figured out something better, but honestly, this is probably robust enough for what I need here. In a similar vein, it'd be nice to be able to decide who should talk next. But this is going to involve doing some sort of calculation regarding who's been mentioned recently and hasn't spoken in a while without making the distribution obviously favor AIs somehow. So there's a fairly involved system you could imagine putting together that involves tokenizing the conversation and looking for unresolved mention tokens or something, or, you could ask chatGPT to decide for you.

(defn check-speaker [turing-test]
  (let [AIs (->> turing-test :contestants (filter (fn [[k v]] (= (:type v) :ai))) (map first))
        prompt (concat
                [{:role :system :content "You are the moderator on a gameshow called 'Turing Test'. It is a contest where some number of humans and some number of AIs try to deceive each other about whether they are human or AI while also trying to determine their opponents identity. Your job is to evaluate the list of contestants and tell me whether and which of the AIs should respond next."}
                 {:role :system :content
                  (str "The current contestants are "
                       (->> turing-test :contestants (map (fn [[k v]] [k (:type v)])) (into {}) str)
                       ", and their chat history follows:")}]
                (:chat-history turing-test)
                [{:role :system :content
                  (str "Given that history, which AI contestant of "
                       (str/join ", " AIs)
                       " (if any) should speak next. Please submit your response as a JSON value String with no other commentary.")}])]
    (if-let [choice (get-in (ai/chat prompt) ["choices" 0 "message" "content"])]
      (let [choice (json/decode choice)]
        (if ((set AIs) choice)
          choice
          (rand-nth AIs))))))

I don't think I named check-speaker well, but the point here is that it hits ai/chat with a request to review the conversation so far, and then decide which of the AIs should be called for input. This works about as well as you'd imagine.

clj-turing-test.core> (let [AIs (->> tt :contestants (filter (fn [[k v]] (= (:type v) :ai))) (map first))
      prompt (concat
              [{:role :system :content "You are the moderator on a gameshow called 'Turing Test'. It is a contest where some number of humans and some number of AIs try to deceive each other about whether they are human or AI while also trying to determine their opponents identity. Your job is to evaluate the list of contestants and tell me whether and which of the AIs should respond next."}
               {:role :system :content
                (str "The current contestants are "
                     (->> tt :contestants (map (fn [[k v]] [k (:type v)])) (into {}) str)
                     ", and their chat history follows:")}]
              (:chat-history tt)
              [{:role :system :content
                (str "Given that history, which AI contestant of "
                     (str/join ", " AIs)
                     " (if any) should speak next. Please submit your response as a JSON value String with no other commentary.")}])]
  (ai/chat prompt))
{"id" "chatcmpl-7N7InuzdTNSglWM4YkwIIOQTNQxH7",
 "object" "chat.completion",
 "created" 1685743969,
 "model" "gpt-3.5-turbo-0301",
 "usage"
 {"prompt_tokens" 1562, "completion_tokens" 2, "total_tokens" 1564},
 "choices"
 [{"message" {"role" "assistant", "content" "\"D\""},
   "finish_reason" "stop",
   "index" 0}]}
clj-turing-test.core>

Except every once in a while, I saw the return content contain somehting like "I think 'D' should speak next" instead of just "D". In the rare case that this happens, I want to fall through to a random choice of the AI contestants. Which is why in the function definition, I check if the resulting choice is in the set of participating AIs, and if not, I just make a call to rand-nth. No big deal.

The rest of model is entirely minor utility functions that have to do with reading the above inputs/outputs into Clojure datastructures, interfacing with the openAI API and dealing with maintaining the state of a game in a way that lets human players interact with the AIs.

(defn contestant-name-from-uid [turing-test uid]
  (if-let [pair (->> turing-test :contestants
                     (filter (fn [[_ entry]] (= (:id entry) uid)))
                     first)]
    (key pair)))

(defn mk-message [contestant string] {:role :user :name contestant :content string})

(defn human-input [turing-test message]
  (update turing-test :chat-history #(conj % message)))

(defn get-guess-from [turing-test contestant]
  (let [history (chat-history-for turing-test contestant)
        res (ai/chat (conj
                   history
                   {:role :system
                    :content
                    (str "This concludes the current game of 'Turing Test'. You may now stop pretending to be a human. Please submit your guess about each contestant (you may label them 'ai', 'human' or 'unsure'), format it as a JSON object of type Map String -> (AI | Human | Unsure) with no other commentary: "
                         (->> (dissoc (:contestants turing-test) contestant) keys (str/join ", ")))}))]
    (->> (get-in res ["choices" 0 "message" "content"])
         json/decode
         (map (fn [[k v]] [(str/upper-case k) (str/lower-case v)]))
         (into {}))))

(defn collect-guesses [turing-test]
  (->> (:contestants turing-test)
       (map (fn [[letter contestant]]
              [letter (when (= :ai (:type contestant))
                        (get-guess-from turing-test letter))]))
       (into [])))

(defn score-guess [contestants guess]
  (reduce
   (fn [memo [letter contestant]]
     (if (get guess letter)
       (if (= (get guess letter) "unsure")
         memo
         (if (= (get guess letter) (name (:type contestant)))
           (inc memo)
           (dec memo)))
       memo))
   0
   contestants))

(defn score-guesses [contestants guesses]
  (let [guess-score (->> guesses
                         (map (fn [[letter guess]] [letter (score-guess contestants guess)]))
                         (into {}))
        guessed-score (->> contestants
                           (map
                            (fn [[letter cont]]
                              (let [tp (name (:type cont))]
                                [letter
                                 (reduce
                                  (fn [memo [cur-guesser cur-guess]]
                                    (if (= cur-guesser letter)
                                      memo
                                      (if (= (get cur-guess letter) tp)
                                        memo
                                        (+ memo 1))))
                                  0 guesses)])))
                           (into {}))]
    (merge-with + guess-score guessed-score)))

Fairly self explanatory now that you've seen the stuff above. The scoring function is possibly the only confusing part here; the idea is that it gives you points for every correct guess you make (without penalizing you for being unsure) and also gives you points for every opponent that mis-guessed you. I guess, just for completeness' sake, I manually collected and scored guesses from the game transcript at the top. The results were:

clj-turing-test.core> (model/collect-guesses tt)
[["A" nil]
 ["B" {"A" "ai", "C" "human", "D" "ai"}]
 ["C" {"A" "human", "B" "ai", "D" "human"}]
 ["D" {"A" "human", "B" "ai", "C" "ai", "D" "ai"}]]
clj-turing-test.core> (def guesses [["A" {"A" "human", "B" "ai", "C" "ai", "D" "ai"}]
 ["B" {"A" "ai", "C" "human", "D" "ai"}]
 ["C" {"A" "human", "B" "ai", "D" "human"}]
 ["D" {"A" "human", "B" "ai", "C" "ai", "D" "ai"}]])
#'clj-turing-test.core/guesses
clj-turing-test.core> (model/score-guesses (:contestants tt) guesses)
{"A" 5, "B" -1, "C" 2, "D" 5}
clj-turing-test.core>

Lessons

Ok, what does this really teach us?

Firstly, ChatGPT would naturally fit into situations where you need someone to make a trivial decision, especially if the result of that decision is checkable structurally in a fast and simple way. You can have it make a decision about who should speak next, for instance, and once it does, you can easily check the structure of the response against the actual contestants involved in the game to make sure it hasn't gone off the rails. I could half-seriously see using this as a mechanism to avoid bikeshedding out in the wild3.

Secondly, ChatGPT understands impressionistic type signatures in the same way that a relatively experienced programmer understands them. So you can ask it for JSON objects matching a particular type and it'll mostly do what you need to. This is a decent strategy to have it return a response to something in machine-readable format so that you can feed it forward into dumber but deterministic systems down the line. Importantly, this makes it trivially composeable. I can easily imagine making a core of ChatGPT surrounded by a few more submodules do serious useful work. The trivial example of this is something like AgentGPT, which I think basically just has a few pieces of state and a deterministic "do the next step" function that calls into chat or completion at some point.

Thirdly, it's not itself deterministic4. I don't think this article gave you a visceral feel for that, but if you've been playing along in the REPL while reading this, it should be fairly obvious that it sometimes just doesn't do the thing that you ask it to. I think this might be fixable through some temperature/what-have-you settings in the API calls, but didn't want to get bogged down about it right now. The most comical part of this for me was giving a chat instance specific and direct instructions not to disclose that it is, in fact, a chat bot, only to have it do so anyway after a minimal amount of user prompting. This isn't even getting into its' built-in guard rails5.

I could see the outlines of that Wendy's drive-through system even having done as little coding around this as I currently have. Which, spoilers, is what I'm going to be playing around with next.

  1. It's not globally boring, because it serves as a decent example of how to make a full-stack project in Clojure. If you're new to this sort of thing, you might want to pick through the project.clj, server.clj and front_end/core.clj files in order to see a working baseline websocket project where both front and back-end are compiled from the same set of clj files. The websocket thing is fairly new to this blog, but it's also releatively self-explanatory. My old hexs repo might be a better example of a clj/cljs/cljc hybrid project; mostly because it has both a front-end and backend and also has some library code that the two "share" as part of the source tree.
  2. The other side being on the other side of open_ai calls.
  3. It might not resolve them any better than a d20, but still.
  4. Which actually helps with the trivial composeability point from earlier. Because it's non-deterministic, you could easily error-correct here by retrying any output requests on a parse/structural validation failure. I tried the simpler option in clj-turing-test either by ignoring failures (as in the scoring functions) or by having a worse-but-definitely-working alternative to ChatGPT (as in the choose-next-speaker routine).
  5. For instance, if you try asking it to make meth or explosives, or try sexting with it, the response you'll get back is something along the lines of "this is inappropriate to talk about, so here, have a 400 error instead".

Permalink

Clojure Deref (June 2, 2023)

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem. (@ClojureDeref RSS)

From the core

Over the course of the last few weeks, I’ve been moving all of the Clojure and contrib library continuous integration and release builds from the Clojure build box over to GitHub actions, which has been somewhat tedious, but has opened up some new opportunities for testing and automation. Ultimately I would like to move a lot of this towards tools.build and out of Maven, but one step at a time.

We did a Clojure CLI build this week that includes several important fixes that came in from the community via Ask Clojure, so thank you to everyone that is adding and voting on things there, it really does help as a central place. This release also contains some important changes to accept -X and -T args over stdin, which is groundwork for fixing some add-libs issues. We have a batch of things in the queue for another alpha soon.

I have also been working on our approach to improving functional interface Java interop. The crux of this is that when calling a Java API that accepts a functional interface argument (not just the java.util.function interfaces, but any "single abstract method" interface), wouldn’t it be nice if IFn’s could just be passed directly without needing to reify the functional interface and adapt? Sure would. We’ve looked at many possible approaches to this and are starting to prototype some primary contenders. This has also fed a lot of learning that might feed some other things we’re looking at.

Fogus has continued work on stream integration - CLJ-2775 is the first part of that, but there may be some additional work as well.

Thanks to Anton Fonarev for all the assistance on link aggregation! With me spending less time on that, I can spend more time writing status like above, so it really helps.

Libraries and Tools

New releases and tools this week:

  • Clojure CLI 1.11.1.1347

  • tiara 0.2.1 - A small data structure library

  • babashka 1.3.180 - Native, fast starting Clojure interpreter for scripting

  • splint 1.7.0 - A Clojure linter focused on style and code shape

  • fulcro-rad 1.5.0 - Fulcro Rapid Application Development

  • cli-tools 0.7 - CLIs and subcommands for Clojure or Babashka

  • clj-kondo 2023.05.26 - Static analyzer and linter for Clojure code that sparks joy

  • clojure-ts-mode 0.1.0 - The next generation Clojure major mode for Emacs, powered by TreeSitter

  • fzf.clj - A small, data-driven and babashka-compatible wrapper around fzf

  • splint 1.8.0 - Linter focused on style and code shape

  • statecharts 1.1.0 - A Statechart library for CLJ(S)

  • slip 0.0.58 - A Clojure+Script micro-library for building systems of objects (IOC)

  • tools.gitlibs 2.5.197 - API for retrieving, caching, and programatically accessing git libraries

  • tools.deps 0.18.1354 - Deps as data and classpath generation

  • michelangelo 0.1.0 - Turtle transforming library

  • deps-try 0.5.0 - Try out Clojure libraries via rebel-readline

  • calva 2.0.367 - Clojure & ClojureScript Interactive Programming for VS Code

Permalink

Senior Clojure Backend Engineer (Canada) at Composer

Senior Clojure Backend Engineer (Canada) at Composer

usd170000 - usd200000

Build the infrastructure powering our automated investing platform!

Composer is a no-code platform for automated investment management. Composer allows you to build, test, deploy, and manage automated investing strategies - all without writing a line of code.

As an early backend engineer at Composer you will:

  • Be responsible for designing and building critical pieces of our infrastructure
  • Work closely with the executive team to guide our decisions regarding technical architecture

Projects you will work on:

  • Creating a language that clients can use to define any conceivable trading strategy ("strategies as data")
  • Determining the best way to collaborate on, share, and/or monetize strategies
  • Allowing clients to develop custom logic to further personalize their strategies
  • See here for more ideas!

We're looking for someone who:

  • Loves Clojure! (Clojurescript a bonus)
  • Has familiarity with cloud platforms (We use GCP)
  • Will be a technical thought leader within the company
  • Understands database design and SQL
  • Makes educated decisions when faced with uncertainty

What's it like to work at Composer?

  • We believe diverse perspectives are necessary if we aim to disrupt finance. To that end, we are an equal opportunity employer and welcome a wide array of backgrounds, experiences, and abilities.
  • We believe the simplest solution is most likely the best one
  • We encourage self-improvement and learning new skills
  • We are venture-backed by top investors
  • We are remote-friendly (+ a Toronto-hub for get-togethers)
  • We offer generous equity!
  • Our Values

Permalink

Notes on neural networks from scratch in Clojure

I set aside some time to figure out how to build neural networks from scratch in Clojure, without external libraries.

After a couple dependency-free versions, I ended up adding the neaderthal library to do faster matrix math. The different versions I wrote along the way are on github, in case they’re helpful for anybody else who wants to do this in Clojure.

First impressions and hello world

Neural networks are surprisingly easy to get started with. There’s significantly more “magic” inside a good concurrent queue implementation, for example, than inside a basic neural network to recognize handwritten digits.

For example, here’s the “hello world” of neural networks, a widget to recognize a hand-drawn digit:


Loading JavaScript...

See this widget at github.com/matthewdowney/clojure-neural-networks-from-scratch/tree/main/mnist-scittle



And here’s the code for the pixel array -> digit computation1:

  (defn sigmoid [n] (/ 1.0 (+ 1.0 (Math/exp (- n)))))
  
  (defn feedforward [inputs weights biases]
    (for [[b ws :as _neuron] (map vector biases weights)]
      (let [weighted-input (reduce + (map * inputs ws))]
        (sigmoid (+ b weighted-input)))))
  
  (defn argmax [numbers]
    (let [idx+val (map-indexed vector numbers)]
      (first (apply max-key second idx+val))))
  
  (defn digit [pixels]
    (-> pixels (feedforward w0 b0) (feedforward w1 b1) argmax))

It’s striking that such a complicated task works without intricate code or underlying black-box libraries.2 I felt kind of dumb for not having known this already!

Resources

The three most helpful resources for me were:

In retrospect, to get started, I’d recommend reading the first part of Nielsen’s tutorial, skipping to the Andrej Karpathy video, and then solving MNIST from scratch using those two things as references, before coming back to the rest of Nielsen’s material.

I also went through Dragan Djuric’s impressive and erudite deep learning from scratch to GPU tutorial series, but I can’t say I’d recommend it as an introduction to neural networks.3

Approach in retrospect

I’m glad I decided to start from scratch without any external libraries, including ones for matrix math.

I do, however, wish I’d watched Andrej Karpathy’s video before getting so deep into Nielsen’s tutorial, especially because of the backprop calculus4, which I struggled with for a while. Karpathy’s REPL-based, algorithmic explanation was much more intuitive for me than the formal mathematical version.

My approach was to:

  1. First, build a neural network for the MNIST problem with no matrix math (nn_01.clj),
  2. Then, create a version with handwritten matrix math,
  3. Eventually, add the neanderthal library for matrix math in a third version,
  4. Finally, enhance performance with batch training in the fourth version.

The training time for one epoch of MNIST was 400 seconds in the first two versions, 5 seconds in the third (on par with the Python sample), and down to 1 second in final version.

I’m glad I broke it down like this. Would do again.

Before implementing the backprop algorithm, I built some unit tests for calculating the weight and bias gradients given starting weights and biases and some training data, and this turned out to be enormously helpful. I used Nielsen’s sample Python code to generate the test vectors.

Finally, invoking numpy via libpython-clj at the REPL was useful for figuring out the equivalent neanderthal expressions.

Basic things that I should have already known but didn’t

  • A neuron in a neural network is just a function [inputs] -> scalar output, where the output is a linear combination of the inputs and the neuron’s weights, summed together with the neuron’s bias, and passed to an activation function.

  • Much of the magic inside of neural network libraries has less to do with cleverer algorithms and more to do with vectorized SIMD instructions and/or being parsimonious with GPU memory usage and communication back and forth with main memory.

  • Neural networks can, theoretically, compute any function. And a more readily believable fact: with linear activation functions, no matter how many layers you add to a neural network, it simplifies to a linear transformation.

  • But, the activation function is not necessarily all that squiggly — ReLU is just max(0, x) and it’s widely used.

  1. Since I used Scittle to embed the Clojure code in this page, you can browse the source file directly. 

  2. And sure, this is a rudimentary network architecture, and there’s a sense in which “the real program” is the magic weights and biases numbers in the w0, w1, b0, and b1 vectors, but it turns out that you can also write the training code from scratch to find those vectors without too much trouble. 

  3. It is definitely an introduction to memory reuse tricks and GPU programming, for someone who already has a strong grasp of linear algebra, and wants to reinforce or deepen existing understanding of neural networks and relevant performance optimization. Which is crucial for deep learning in practice, but is a lot to take in at first. 

  4. Also, on the indexes in Nielsen’s neural network backpropagation algorithm — the style in the Python sample starting on line 101 was hard for me to parse, with negative indexes and iterations using three indexes each. I found it helpful to rewrite like this:

    # compute the difference between the output and the expected output
    # this is the last layer's error
    error = self.cost_derivative(activations[-1], y) * sigmoid_prime(zs[-1])
    
    # weight and bias gradient vectors, same shape as the network layers
    nabla_w = []
    nabla_b = [error]
    
    # the activations list has inputs prepended, so it's longer by 1
    activations_for_layer = lambda layer_idx: activations[layer_idx+1]
    
    # iterate backwards through the layers
    for layer_idx in xrange(len(self.weights), 0, -1):
    
        # compute a change in weights using the previous layer's activation
        prev_activation = activations_for_layer(layer_idx-1)
        nabla_w.insert(0, np.dot(error, prev_activation.transpose()))
    
        # if there is a previous layer, compute its error
        if layer_idx > 0:
            this_layer_weights = self.weights[layer_idx]
            prev_layer_weighted_inputs = zs[layer_idx-1]
    
            sp = sigmoid_prime(prev_layer_weighted_inputs)
            error = np.dot(this_layer_weights.transpose(), error) * sp
            nabla_b.insert(0, error)
    
    return (nabla_b, nabla_w)
    

Permalink

OSS updates May 2023

In this post I'll give updates about open source I worked on during May 2023.

To see previous OSS updates, go here.

Babashka-conf

Babashka-conf is happening June 10th in Berlin. Only a few tickets left!

Sponsors

I'd like to thank all the sponsors and contributors that make this work possible! Open the details section for more info.

Sponsor info Top sponsors:

If you want to ensure that the projects I work on are sustainably maintained, you can sponsor this work in the following ways. Thank you!

If you're used to sponsoring through some other means which isn't listed above, please get in touch.

On to the projects that I've been working on!

Updates

The following projects had updates in the last month. Note that only highlights are mentioned and not a full overview of all changes. See the project's changelogs for all changes.

  • Preparations for babashka conf are in full swing and I'm preparing a talk with the title 'Growing an ecosystem'.
  • This month I've had the honor to visit the JUXT 10 year anniversary in London and met a lot of fellow Clojurians over there.
  • Babashka and SCI will be featured at the last iteration of Strange Loop!
  • My OSS work is funded by Clojurists Together in Q2
  • clj-kondo: static analyzer and linter for Clojure code that sparks joy.
    • Version 2023.05.18 - 2023.05.26 were released. Full changelogs here. Highlights:
    • Linter :uninitialized-var moved from default :level :off to :warning
    • #2065: new linter :equals-true: suggest using (true? x) over (= true x) (defaults to :level :off).
    • #2066: new linters :plus-one and :minus-one: suggest using (inc x) over (+ x 1) (and similarly for dec and -, defaults to :level :off)
    • #2058: warn about #() and #"" in .edn files
  • babashka: native, fast starting Clojure interpreter for scripting.
    • Released 1.3.180, mostly a maintenance release
    • See the complete CHANGELOG
  • scittle: Execute Clojure(Script) directly from browser script tags via SCI
    • #58: build system for creating scittle distribution with custom libraries. See plugins/demo.
    • Use window.location.hostname for WebSocket connection instead of hardcoding "localhost" (@pyrmont)
    • Upgrade sci.configs to "33bd51e53700b224b4cb5bda59eb21b62f962745"
    • Update nREPL implementation: implement eldoc (info, lookup) (@benjamin-asdf)
  • deps.clj: A faithful port of the clojure CLI bash script to Clojure
    • Introduce all new programmatic API
    • Automatically use file when exceeding Windows argument length
  • deps.add-lib: Clojure 1.12's add-lib feature for leiningen and/or other environments without a specific version of the clojure CLI
    • First clojars release
  • nbb: Scripting in Clojure on Node.js using SCI
    • Nbb is now compatible with bun. To run nbb in a bun project, use bunx --bun nbb.
  • squint: CLJS syntax to JS compiler
    • Fix import with $default
  • cherry Experimental ClojureScript to ES6 module compiler
    • Support with-out-str
  • http-client: Babashka's http-client
    • Add :authenticator option
  • sci.configs: A collection of ready to be used SCI configs.
    • This project now has a configuration for datascript, for anyone who wants to use SCI together with datascript. See this commit.
  • SCI: Configurable Clojure/Script interpreter suitable for scripting and Clojure DSLs
    • Support :require-macros
  • process: Clojure library for shelling out / spawning sub-processes
    • Support :pre-start-fn in exec
    • Allow passing :cmd in map argument
    • Better testing for exec by @lread
  • fs - File system utility library for Clojure
    • :paths argument for fs/which by @lread
    • Support inputstream in fs/copy
    • Add fs/owner to return owner of file

Other projects

These are (some of the) other projects I'm involved with but little to no activity happened in the past month.

Click for more details

  • edamame: Configurable EDN/Clojure parser with location metadata
    • Improvements for reading namespaced maps
  • babashka.book: Babashka manual
    • Several corrections
    • Dynamic :exec-args
    • Script-adjacent bb.edn docs
  • CLI: Turn Clojure functions into CLIs!
    • Support --no-option and parse as {:option false}
    • Support grouped aliase like -ome as {:o true, :m true, :e true}
  • pod-babashka-go-sqlite3: A babashka pod for interacting with sqlite3
    • Better error message when connection is not a string
  • instaparse-bb
    • Add transform function
  • jet: CLI to transform between JSON, EDN, YAML and Transit using Clojure
    • Add option to elide commas
  • quickdoc: Quick and minimal API doc generation for Clojure
  • rewrite-clj: Rewrite Clojure code and edn
  • sql pods: babashka pods for SQL databases
  • pod-babashka-buddy: A pod around buddy core (Cryptographic Api for Clojure).
  • gh-release-artifact: Upload artifacts to Github releases idempotently
  • neil: A CLI to add common aliases and features to deps.edn-based projects
  • cljs-showcase: Showcase CLJS libs using SCI
  • carve - Remove unused Clojure vars
  • grasp: Grep Clojure code using clojure.spec regexes
  • quickblog: Light-weight static blog engine for Clojure and babashka
  • lein2deps: leiningen to deps.edn converter
  • 4ever-clojure - Pure CLJS version of 4clojure, meant to run forever!
  • pod-babashka-lanterna: Interact with clojure-lanterna from babashka
  • pod-babashka-fswatcher: babashka filewatcher pod
  • joyride: VSCode CLJS scripting and REPL (via SCI)
  • clj2el: transpile Clojure to elisp
  • deflet: make let-expressions REPL-friendly!
  • babashka.json: babashka JSON library/adapter

Permalink

Using extensions in ClojureDart

Introduction

We love extensions in Dart, which help us reuse and apply some code to an existing class widget.

But how to use it in ClojureDart ? We&aposll see how to do it.

The result we will achieve in this article:

0:00
/

The Dart part

We still need some Dart code, for example, here we will create an extension to refresh a ListView.

Based on the last article we will change some code to apply for the extension but first, let&aposs create the Dart file. We will use pull_to_refresh package for this widget:

Add it to your pubspec.yaml with the command flutter add pub pull_to_refresh then create the list_view_refreshable.dart:

import &apospackage:flutter/widgets.dart&apos;
import &apospackage:pull_to_refresh/pull_to_refresh.dart&apos;

extension ListViewRefreshable on ListView {
  Widget refreshable(Future<void> Function() onRefresh) => RefreshableWidget(
        builder: (refreshController) => SmartRefresher(
          controller: refreshController,
          onRefresh: () async {
            await onRefresh.call();
            refreshController.refreshCompleted();
          },
          child: this,
        ),
      );
}

class RefreshableWidget extends StatefulWidget {
  const RefreshableWidget({super.key, required this.builder});
  final Widget Function(RefreshController refreshController) builder;
  @override
  State<RefreshableWidget> createState() => _RefreshableWidgetState();
}

class _RefreshableWidgetState extends State<RefreshableWidget> {
  final RefreshController refreshController = RefreshController();

  @override
  Widget build(BuildContext context) {
    return widget.builder(refreshController);
  }
}
list_view_refreshable.dart

Here it&aposs a simple extension to apply in every ListView a refreshable method.

Now let&aposs come back to Clojure.

Subscribe to Etienne Théodore

Don’t miss out on the latest articles. Only one email by week about Flutter/Dart (no-spam)
your@email.com
Subscribe

Clojure Part

Let&aposs require some stuff before we deep dive into the code, change the main.cljd to import the extension file:

(ns acme.main
  (:require
   [api.albums :as api]
   [pages.album_detail :as album-detail]
   ["list_view_refreshable.dart" :as ext-refresh]; our extension
   ["package:flutter/material.dart" :as m]
   [cljd.flutter :as f]))
main.cljd

Then we need a state to save the list of albums to perform the GET call.

(defonce ^:private app-state
  (atom {:albums []}))

Then we create the call function:

(defn- fetch-albums []
  (swap! app-state assoc :albums (await (api.albums/get-albums))))

This function will await the GET call from the API and then use the result to change the value of the app-state albums  thanks to swap! function and assoc.

Finally, we apply the refreshable function from the extension we have created:

(defn- build-list-items [albums]
  (f/widget
   :get [m/Navigator]
   (-> (m/ListView.builder
        .itemCount (count albums)
        .itemBuilder
        (f/build
         #_{:clj-kondo/ignore [:unresolved-symbol]}
         [idx]
         (let [album (get-in albums #_{:clj-kondo/ignore [:unresolved-symbol]} [idx])]
           (m/ListTile
            .onTap (fn [] (navigate navigator
                                    (album-detail/view album)
                                    (str "/album-detail/" (.-id album))))
            .title (m/Text (.-title album))))))
       ext-refresh/ListViewRefreshable
       (.refreshable ; here the refresh
        (fn []
          (await (fetch-albums)))))))

The critical part here is the -> macro that allows us to use the .refreshable  extension just after the ListView. And of course the .refreshable itself to refresh the list via fetch-albums.

If you use VSCode you can extract some functions to refacto some parts of the code to be simpler to understand (on mac cmd+.).

For example, here we could extract this function to a named function like so:

alt
(defn- list-view-albums [albums navigator]
  (m/ListView.builder
   .itemCount (count albums)
   .itemBuilder
   (f/build
    #_{:clj-kondo/ignore [:unresolved-symbol]}
    [idx]
    (let [album (get-in albums #_{:clj-kondo/ignore [:unresolved-symbol]} [idx])]
      (m/ListTile
       .onTap (fn [] (navigate navigator
                               (album-detail/view album)
                               (str "/album-detail/" (.-id album))))
       .title (m/Text (.-title album)))))))

(defn- build-list-items [albums]
  (f/widget
   :get [m/Navigator]
   (-> (list-view-albums albums navigator)
       ext-refresh/ListViewRefreshable
       (.refreshable ; here the refresh
        (fn []
          (await (fetch-albums)))))))
main.cljd

Conclusion

We saw how to use extensions in ClojureDart, with an example to pull to refresh any ListView. Happy coding!

Like always feel free to fork/clone/reuse the code I use for this article:

GitHub - Kiruel/fetch-data-list at extensions
ClojureDart example to fetch a simple list. Contribute to Kiruel/fetch-data-list development by creating an account on GitHub.
alt

Permalink

Q2 2023 Funding Announcement

Greetings all. We’re excited to announce the projects we’ll be funding for Q2 2023. (8 projects $44K USD in total). There were a lot of great projects - so making a decision was, as usual, difficult. The good news, however, is that proposals will automatically be reconsidered for the next 2 cycles. We have supported many projects in the past that did not make it in the initial round - so stay tuned! Thanks to everyone for your thoughtful proposals - this is the kind of thinking and sharing that makes the Clojurists Together community so awesome!

Projects Awarded $9K USD

Emmy: Sam Ritchie
Clj-kondo, babashka, cherry, SCI: Michiel Borkent
Clojure Camp: Daniel Higginbotham
Neanderthal, Clojure CUDA, Deep Diamond: Dragan Duric

Projects Awarded $2K USD

clj-nix: José Luis Lafuente
Jank: Jeaye Wilkerson
Lucene Grep: Dainius Jocas
Portfolio: Christian Johansen

LARGER PROJECTS AWARDED $9K USD


Emmy: Sam Ritchie

I want to make the SICMUtils computer algebra system into a best-in-class, modular set of tools for doing serious work and exploration in math and physics, and sharing that work in the browser. When used with NextJournal’s Clerk, https://maria.cloud/, and the components I built with last quarter’s funding, I think SICMUtils could take the place of Mathematica for students and educators and help cement Clojure as a serious contender for top data science language.

Concretely, I would use the quarter to:

  • Rebrand SICMUtils as “Emmy” after Emmy Noether
  • split the 30kloc project into modules, to make the components easy to use from other projects. Examples of useful components include:
  • A battle-tested symbolic expression simplifier and pattern-matching rules
  • An extensible symbolic expression compiler, capable of emitting LaTeX, fast numerical functions, fast matrix-valued functions, etc
  • Forward- and reverse-mode automatic differentiation
  • Modules for mathematical objects like polynomials, rational functions, quaternions, and geometric algebra
  • Differential geometry code used in general relativity research

With last quarter’s funding, I produced a number of components for creating and interacting with 2d and 3d visualizations, as well as tooling for NextJournal’s Clerk to make it easy to create interactive documents embedding these components.

I’ll extend this work by building out a library of Clerk viewers that make it easy to visualize and work with interactive versions of all of the important mathematical objects in the SICMUtils library.

Finally, I’ll use the system to produce a series of interactive essays that explore mathematics and physics, as a way of showing off the power of the system.
A stretch goal would be to ship these interactive essays both using Clerk and using https://www.maria.cloud/, so that all text and code would become interactive as well.

As an example of what is possible, see this interactive visualization of a (p, q) torus knot.

I was able to use the SICMUtils automatic differentiation and function compilation features to turn the following inefficient description of the curve for one of these knots into a highly optimized JavaScript function that ran 10-20x faster than the hand-tuned JavaScript of the example I was porting: ```clojure (defn torus-knot “Returns a function of theta that produces a 3-vector of the XYZ coordinates of a (p, q) torus knot wrapped around a torus (donut) with major radius R and minor radius r.”


Clj-kondo, babashka, cherry, SCI: Michiel Borkent

Clj-kondo:

  • Support an arbitrary amount of reader conditionals in .cljc files via configuration. This is mostly useful for ClojureDart (:cljd),
  • babashka (:bb), nbb (:org.babashka/nbb) and shadow-cljs targets like :node. It’s often not selfevident if people are writing both for JVM Clojure + ClojureScript (which is how clj-kondo by default understands .cljc files) or for other combinations of Clojure dialects. I want to add support for configuring what Clojure dialects apply to your project and what Clojure dialects should be linted as (e.g. :node should just be linted as :cljs). If necessary I’ll also add support for specific dialects like ClojureDart.
  • Help with clj-kondo configuration for Electric Clojure
  • Numerous other issues, there’s always plenty to do! See the clj-kondo project board: and see here for a list of most upvoted issues:
  • As clj-kondo is the foundation for the static analysis in clojure-lsp: improvements in the interaction between the two, most specifically the analysis data.

Babashka

  • Make it easier to use scripts in projects from anywhere on the system. bbin is already helping a lot with this. However, if we allow scripts that are on the path to be combined with their relative bb.edn file to be invoked while taking into account the dependencies from bb.edn, this would be a lower bar of entry for having system wide scripts and might also simplify the implementation of bbin itself. Eventually I’d like to include some of bbin’s features into babashka itself, as a bb install command and this would be the first step towards that.

Port Clojure 1.12 features to babashka

Cherry

  • Cherry is a CLJS compiler that can be used as a standalone tool but also to (partially) replace self-hosted CLJS in exising projects. This mode is known as cherry.embed: The embed mode is already used in malli as an alternative to SCI: instead of interpreted CLJS we now get compiled CLJS for serializable functions. To be more generally useful, cherry needs to emit namespaced functions which it currently doesn’t. Eventually something like scittle may be implemented using cherry which would give the same convenience but a smaller bundle and better performance.

SCI:


Clojure Camp: Daniel Higginbotham

Vision
At Clojure Camp, we are creating a free learning community to help new developers of all backgrounds build a secure foundation for professional programming success — with Clojure! We hope to model a positive tech environment through a commitment to diversity and through our emphasis on creating a fun, friendly, supportive, and kind environment.

Clojure Camp is rooted in an equity ethos: it’s open to all while prioritizing the experience of those who face the most hurdles in tech. Our Community is a place where Women, BIPOC folks, and LGBTQ+ folks belong.

We believe that learning Clojure is a great long-term investment because it helps people learn a better way of approaching programming, and because it brings creative satisfaction. We want to make it accessible to more people.

We believe this will have a long-term positive impact on the Clojure community by improving our reputation as a welcoming and helpful community, and by increasing the pool of Clojure developers.

Product
For the next six months, we are focusing on two “products”: a study group matchmaking service and a metacircular project incubator.

Study Group Matchmaking
We have created a study group format and matchmaking service that’s proving to be effective with our initial cohort of learners. About five learners are paired with two mentors who guide the learners during weekly video calls, reviewing the work the learners have done over the previous week. In between video calls, learners use a tool to match them with other learners in their group for pair programming and support.

We want to continue building out this model by refining the curriculum and developing other avenues of learning and collaboration. Also, we want to start doing some outreach to attract new developers to participate. We’ve received a lot of interest from potential mentors; now we just need to find some students for them!

Metacircular Project Incubator
We’ve designed Clojure Camp to be a kind of project incubator, where motivated volunteers can propose their learning projects and get our support. The kinds of support we provide include:
Implementation feedback. We are available to brainstorm ideas or otherwise spend time chatting with project leaders to help them refine their offering.
Networking. We can refer project leaders to others who might be able to help, and we run a discord server for project leaders to reach out to each other.
Moral Support. Part of our motivation for creating this structure is that we want to help people avoid burnout. It’s a lot easier to do this sustainably when you’re in community.
Advertising. We help project leaders get their projects in front of a qualified audience, allowing them to focus on what they do best: helping people learn!

Our core team already has a number of projects we’ve begun working on in this spirit, including:

  • Speaker coaching
  • Email newsletter


Neanderthal, Clojure CUDA, Deep Diamond: Dragan Duric

In short, I propose to implement:

  • a port of Neanderthal’s MKL engines and CPU related stuff to JavaCPP instead of neanderthal-native (for sure).
  • a port of ClojureCUDA to JavaCPP’s CUDA. (probably, depending how 1 goes) - a port of Neanderthal’s GPU matrix engine to new, JavaCPP-based ClojureCUDA (almost sure, if ClojureCUDA port goes well)
  • update Deep Diamond to use new infrastructure.
  • improve relevant internal parts of Neanderthal code with more macrology (double-edged sword, I know, but also concentrates bugs at one place each).
  • TESTS! to make sure everything works as well as before (of course!)

EXPERIMENTAL OR SHORT-TERM PROJECTS AWARDED $2K USD


clj-nix: José Luis Lafuente

Why is this project important to the Clojure community? Nix is a great tool to build and deploy software, but Clojure is not well supported in the Nix ecosystem. clj-nix makes easier to integrate Clojure on a Nix project.

What are you wanting to achieve with this funding?
I want to refactor the CLI to be more granular about the dependencies used at build time. Currently, all dependencies in all aliases are added to the lock file. I want to add some flags to the CLI to allow users to manually define the deps.edn files, and the aliases to use. Once CLI refactor is done, I want to add support for calling Babashka at build time, adding the possibility to use Babashka, and the dependencies defined in the bb.edn file, in the Nix build phase.


Jank: Jeaye Wilkerson

jank is the only native Clojure dialect which has its own JIT and embraces interactive programming. It unlocks REPL-driven development for a whole ecosystem previously unserved by Clojure.

Clojurists have demonstrated their desire for native executables with their use of GraalVM. Compared to Graal, jank will offer REPL support, better native interop, smaller binaries, and faster compilation times.

Clojure’s object model is intense. In fact, representing it 1:1 from Java to C++ is impossible, since C++ has stricter rules are duplicate base classes and bases with same-name fields and different types. Furthermore, C++, even with the Boehm GC, is much slower at creating objects than the JVM is. It’s bread and butter work for the JVM. Also, while everything is an Object in the JVM, C++ doesn’t have the same notion. If every jank class were to inherit from the same Object class, it would have very serious performance implications when it comes to allocations.

So jank has two key problems here:

  1. Creating new boxed values is slow, compared to the JVM
  2. Not every jank type can actually be turned into an object, which sometimes means doing some weird dancing to get from one type, through another, and finally to the object base; this generally requires virtual function calls

So far, I’ve worked around the first one by optimizing other things, so jank can be faster than Clojure in a given program for example, but when measuring just the object creations, Clojure is still around twice as fast. I want to fix this.

This task would entail implementing and benchmarking a few different solutions, all of which move jank’s object model away from C++ inheritance and toward something more dataoriented. This gets us around C++’s virtual inheritance allocation costs, but it can also allow every jank type to be treated as an object, which will not only simplify jank’s runtime code, but will itself be a key optimization.

Right now, I have two key design ideas:

  1. A very template-heavy approach, which uses bit fields to keep track of which behaviors an object has, as well as which data storage it has
  2. An ECS-based approach, which separates object identity from storage, which would aid in cache performance and data locality issues

So far, I have prototyped the first approach and found object creation is nearly twice as fast. This funding would allow me to implement both of these solutions fully, benchmark them, and research further ways to improve them. Finally, I will integrate the fastest solution into jank and reap the benefits.


Lucene Grep: Dainius Jocas

What are you wanting to achieve with this funding? I want to tear the lucene-grep project apart into several reusable idiomatic Clojure libraries/tools/tutorials that would help the Clojure community to leverage the Apache Lucene library, e.g. a Clojure library whose single purpose is to create Lucene Analyser in a data driven fashion https://github.com/dainiusjocas/lucene-custom-analyzer.

Also, I want to combine the knowledge encoded in other Clojure projects when it comes to using Apache Lucene and either contribute to existing or fill in the gaps by creating several single purpose idiomatic Clojure libraries.

Why is this project important to the Clojure community? Apache Lucene is a treasure trove of a battle tested functionality for building search and data analysis applications. As of now, there are few ok’ish Lucene wrapper options e.g. https://github.com/jaju/lucene-clj. But Apache Lucene is already a dependency in multiple Clojure systems such as Datomic, Crux, etc. I wan’t to spend some quality time going through the use cases and expose modularise the functionality.


Portfolio: Christian Johansen

What are you wanting to achieve with this funding? I will use the funding to improve the experience for new users. Specifically I will add some self-hosted documentation such that Portfolio can guide users via its UI once set up. I will also improve Portofolio’s error handling and make it provide more helpful pointers when things go wrong. If time permits, I will keep working on a recently started accessibility testing feature for Portfolio.

Why is this project important to the Clojure community? The community has voiced a desire for more developer tooling. Portfolio is a developer tool for frontend ClojureScript work. While it is good that Clojure(Script) makes it easy to use tools built for the underlying platform (e.g. Storybook), strong native CLJ(S) tooling is necessary to fully harness the power of our amazing language and runtime, and make it an attractive alternative. My hope is that Portfolio eventually is one piece of what makes ClojureScript the greatest frontend toolkit out there.

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.