Herder: Automagic scheduling of a convention

About once a year I go off with about 30 of my friends and go and play role playing games and board games for a week in a big house out in the country. This is kinda awesome, but as we’ve increased the numbers we’ve been having fun with the scheduling. Namely, we need to decide when the different games are happening (especially for the 6+ hour board games…), and a week quite frankly almost isn’t enough to cram everything in. Couple that with the actual scheduling being done literally at the last minute, and last year we ran into the limits of what can sensibly be done by hand.

But wait! Last year I did some work with OptaPlanner, which is a constraint satisfaction solver explicitly designed for this sort of problem. However, that would require writing a bunch of Java code, and I’m not that much of a fan of having to do that if I don’t have to. Although, I could try writing this in Clojure instead and leverage the magic of gen-class to do this right? Turns out the answer to that is: mostly.

Enter Herder, a web-based tool for doing just this. You define a convention over some set of days, add time slots per day, some people, some events (and say who’s coming along on those events), and it generates a schedule. You can also add some options, allowing people to only attend some days of the convention, or define multi-day events (we had a use case for something that runs for 5 sessions on separate days), or even have a preferred time slot. It tries to generate a “perfect” schedule i.e. no clashes, everything nicely spaced out, but also gives you feedback on what the problems are if some of the constraints had to be violated.

Screen Shot 2016-04-30 at 11.53.49 pm

Behind the scenes, it’s a single page app using Reagent and ClojureScript for the frontend, talking over a mix of REST HTTP and websockets. This talks to the API section of the app in Clojure, which is mostly a CRUD app (using Korma for the SQL bits). When changes are made that could potentially alter the solution (fixing the spelling of someone’s name doesn’t change things, but adding a new event does), the API section tells the solver to generate a new solution (and then feed that back into the database for use by the CRUD app). The solver itself is almost entirely Clojure, except for two tiny little Java classes that define an abstract implementation of a generic interface, which apparently you can’t do in gen-class. I almost got stuck in a situation where I had a Clojure class that would be referenced by the Java code, which would then be in turn referenced by other Clojure code, but for once Java’s type erasure turns out to be useful, and I could just define the ref in the Java class as Object and everything was pretty much fine.

Along the way, I wrote some Boot code to support cljfmt that I really should split out at some point, wrote my first System components, and submitted jquery-timepicker to CLJSJS.

Early beta testing of this with last years data is now done, and I’m about to unleash it on the people going to this year’s event for further testing before potential live use later this year!


Improving test failures feedback by extending IPrintWithWriter protocol in ClojureScript

When Francesc and I decided to directly store ClojureScript +, -, * and / functions in our Operation record instead of keywords representing them our code got simpler.

However the feedback we received when a test failed got much worse because the whole function got printed in the test failure output:

The same happened when we were working on the REPL.

So we decided to extend the IPrintWithWriter protocol to improve the test failures feedback.

First, we tried to do it in the record declaration as we were doing for the Object protocol:

Surprisingly, it didn't work. We were still getting the ugly output shown before.

We were also getting this warning:

So we tried with extend-protocol:

This worked as we expected and we got a more useful failure feedback:

It also worked using extend-type:

Yet, we didn't understand why it was working as we expected with extend-type and extend-protocol but not in the declaration of the Operation record.

So we decided to ask it on the Clojurians slack.

There thanks to Alejandro Gómez and Nicolás Berger we understood what was happening. As Nicolás said:

That's the reason why it wasn't working and why we were getting that warning.

Check the code Nicolas is talking about in ClojureScript repository.

In the end, it was a nice problem to have, because we not only got a nicer and more useful failure feedback which was our initial goal, but also, learned more about extending core protocols in ClojureScript.

PS: Thank you very much to Alejandro Gómez and Nicolás Berger for their help.


Clojure Newsletter #5

I love to play music and have been an upright bass player in several bands. However, the logistics and economics of a band are … difficult. Still, I’d love to have a creative outlet so plan on looking into Overtone and Sonic Pi. When I saw Smack My Lazer – live coding from overtone-recipes.github.io I […]

The post Clojure Newsletter #5 appeared first on E-String.


Datascript 101 - Chapter 1

Assuming that you have Datascript :required in as d, initialize a database:

(def conn (d/create-conn {})

The {} is the schema where you define how certain attributes (think fields) behave: “Are they references?”, “Are they arrays?” etc.

(def schema {:car/maker {:db/type :db.type/ref}
             :car/colors {:db/cardinality :db.cardinality/many}}

(def conn (d/create-conn schema})

Now we have a database conn with some schema, we’re saying:

“I will have a :car/maker attribute which will be a reference to some other entity (think record), so do the needful when I ask you to handle it (e.g. de-refing). Also, :car/colors is going to be an array.”

Basically, to simplify a bit, you can think of schema as hints to Datascript to help make our lives easier.

Now, lets insert some schnitzel:

Level 1 - Insertion

(d/transact! conn [{:maker/name "Honda"
                    :maker/country "Japan"}]

What’s going on here?

  • transact! means we’re going to transact something (insert/delete/update).
  • conn is our database (or database connection if you will).
  • The weird looking array is what we intend to transact. There are several ways this can be specified as we’ll learn in future lessons. Here we are simply asking Datascript to insert these two “attributes” about “some entity” into the database.

As you can see, I didn’t add any of the attributes I mentioned in my schema, you only need to define stuff that you want to specifically control. Lets do one more.

Level 2 - Insertion

(d/transact! conn [{:db/id -1
                    :maker/name "BMW"
                    :maker/country "Germany"}
                   {:car/maker -1
                    :car/name "i525"
                    :car/colors ["red" "green" "blue"]}])

What in the …?

The new things here:

  • We’re transacting multiple things (since the array now has two maps).
  • There’s a weird :db/id field set to -1 in our maker insertion.
  • There’s a :car/maker attribute set to -1 in our car insertion.
  • The :cars/colors attribute is an array, it will be handled correctly for us because we mentioned that in the schema, it has many wow cardinaleeteez.

You’re saying:

“Insert two things: a maker and a car made by that maker. Give the maker an id -1 (because I am going to refer to it later), then add a car and set the car’s maker ref as the maker I just inserted (identified by -1).”

I wonder if it still works if I switch the insertion order around.

The -1 is resolved to a real entity id when the transaction happens and the :car/maker is correctly set to it. The -1 here is just a temporary id for us to connect stuff up as we insert it, without us having to do multiple transactions (insert maker first, get id, then insert car).

Level 1 - Querying

Now to fetch these values out of the database:

(d/q '[:find ?name
       [?e :maker/name "BMW"]
       [?c :car/maker ?e]
       [?c :car/name ?name]]

This should give you #{["i525"]}.

That looks like an awful way to do whatever the heck its doing. Fear not, fear is the mind-killer.

I don’t want to go into how datascript stores stuff internally as datoms and all the stuff that goes on (since I don’t completely understand it yet). For now lets’s keep it simple:

  • Anything that starts with ? is a variable which will hold a value for us as we process the :where rules.
  • The thing after :find is what the query will return. In this case whatever ends up being in ?name comes out.
  • Each :where rule has a specific format, for our purposes for now its: [<entity-id> <attribute> <value>].
  • A variable in a rule is assigned any values that satisfies that rule. E.g. when we say [?m :maker/name "BMW"] the variable ?m is at the <entity-id> position, so once this rule is processed, ?m holds the entity-id for “BMW” maker.

So as we go down the rules:

  • [?m :maker/name "BMW"] - ?m ends up the entity-id of the “BMW” maker.
  • [?c :car/maker ?m] - ?c ends up with the entity-id of the car which has its maker as ?m (which is “BMW”).
  • [?c :car/name ?name] - ?name ends up with the value of the :car/name attribute of the ?c entity-id, which in this case is"i525".
  • The rule processing finishes and we endup with "i525" in ?name which is finally returned from the query.

You could have also done this:

(let [car-entity (ffirst
                  (d/q '[:find ?c
                         [?e :maker/name "BMW"]
                         [?c :car/maker ?e]]
  (:car/name (d/entity @conn car-entity)))

As far as I know there are better ways of doing things that I (and may be ?you) will eventually learn about.

Things to try

  • Insert some more cars and makers.
  • What do you get from our query when a maker has multiple cars?

Things to think about

  • How do I get an entity-id if I want to insert a car later down the line and I’ve already inserted the maker? In other words, how do I insert a car which is made by “Honda”? For now may be make two queries? First to get the entity-id for “Honda” followed by a transact to do the insertion?

All of the stuff here is available as a gist here.

I will see you next time.


Clojure/ClojureScript Web Developer at Kira Systems (Full-time)

Kira Inc. is a Toronto-based startup using machine learning to automate legal work. We’re looking for a developer to work on our Clojure and ClojureScript web application. Our stack includes reactive single-page web client code and a distributed backend to handle internal computations.

Our team is small, pragmatic, and inquisitive; we love learning new technologies and balance adoption with good analysis. To us, agile is a verb, not a noun, we adopt what works for us rather than strictly following a particular methodology. We prefer to hire near our downtown Toronto office, but also welcome remote work in a time zone within North America.

Technologies we use:

HTML, CSS, SASS, Clojure, ClojureScript, React.js, SQL, PostgreSQL, Java, RabbitMQ, ElasticSearch, Docker

Things you could work on:

Client-side UI code. Server-side REST routes. Platform and core APIs. Document and report generation. Distributed processing architecture. Scaling and performance optimizations.

You should have knowledge of some of these. Most of all we look for those interested in learning.

Please send us:

  • cover letter — tell us why you’re interested and give us a reason to get excited to hire you
  • resume/linkedin profile
  • link to github projects or other work samples

We want to hear from you. Apply online at https://kirasystems.com/careers or email your resume to kirasystems [at] applications [dot] recruiterbox [dot] com with "Web Developer" in the subject line.

Get information on how to apply for this position.


Luminus Workflow

I recently presented at the Philly ETE conference, and it was a really great experience. The conference was well organized, there were lots of great talks, and I got to meet a bunch of interesting people.

My talk focused on the workflow using Luminus and Reagent. During the talk I built a simple app from scratch. The app illustrates how to work with a relational database, create a documented service API, and build a UI for it using Reagent. The live demo portion stats around the 11 minute mark.

If you're interested in my workflow using Luminus and Cursive, then I definitely recommend watching the talk.


Resize Images with Clojurescript

It's another week. And you know what that means? Another excuse for writing a clojurescript widget!

This time, I wanted an easy/quick way to resize some jpeg images.

Here's the result: Image Resize Tool

And here's a peak "under the hood" showing the underlying data structures: Image Resize Tool Devcard

Resize Images in Javascript (and cljs)

Since I have a lot more experience with java and the server side, I almost started writing a java/clojure program.

But then on a whim, I looked into whether it would be possible to use pure javascript. And, it turns out to be really easy using html5 canvas. In fact, this is pretty much all you need to resize images:

(defn resize-image 
  [img width height]
  (let [canvas (doto (js/document.createElement "canvas")
                 (aset "width" width)
                 (aset "height" height))
        ctx  (.getContext canvas "2d")
        _    (.drawImage ctx img 0 0 width height)]

Better Quality

Using the code above doesn't always produce the greatest quality thumbnails. But luckily there are a few tricks to generate higher quality images. For example, rather than resizing directly to the desired size, if you resize in a few different steps, the quality of the resized image will be better. So, for my widget, I use several steps, resizing by 50% of the original image size, until the desired size is reached.

HTML5 Canvas and data URL's

While writing this widget, I learned a lot about canvas objects and the methods used display the results of a canvas inside other html elements. I never knew there was so many options for converting images to/from different formats within browsers.

The way this widget works is that it first resizes an image by using the canvas.drawImage method. Then, it uses canvas.toDataURL to produce a url that can then be used inside an img tag. The same data URL can also be used as the href in anchor tags.

As long as the data url is not too big, everything works great.

But I did run into an annoying limitation. Each browser has different limits to how long a data url can be. If a really long data url is used inside the href of an anchor tag, then when you click on the link, nothing happens?! No errors or anything; which makes it difficult to make a nice UI.

Fortunately, regardless of the thumbnail size, you can still right click inside an img tag and choose "Save As". It's then possible to download and save the image as a png. It's not ideal, but at least it works.


This first version of the resize tool doesn't try to display photos right side up. It simply displays them the same way you happened to be holding your camera when you took the picture.

Photos use EXIF data to suggest the orientation of the photo. However, browsers don't all work the same way. Here's a link to a great write up in case you're curious to learn more.

I might take a stab later at updating this widget to display the photo correctly. But for now, the image is just displayed however the browser chooses to render it.

So much more to play with

All in all, I'm happy how this widget turned out and I'm sure it'll come in handy for quick image resizing in the future.

I'm also planning to reuse this code inside a bigger app I'm working on that will help me manage all my photos.

There's so much more I want to learn and experiment with related the canvas, html5, svg, etc, etc. It's incredible what can be done right inside the browser these days. Lots of fun stuff.

Time to get started on next week's widget ... right now I'm leaning toward a photo orientation fixer ;-)


Datascript 101

I hate writing a ton of text, so I’ll get straight to it.

This is Datascript 101! A series of short tutorials that will help me (and probably you) understand and document some of the things I’d like to do with Datascript.

Bonus: Most of it also applies to Datomic!

I would assume you’re set and ready to go (aka not disucssing installation, IDE setup, Clojure(script) syntax etc.).

If you’ve always dreaded Datascript/Datomic/Datalog like me, here’s something to get you started:

“I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain.” - Frank Herbert, Dune



Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.