Clojure in London: Gower Street

As we’ve grown as a business we’re now using Clojure for almost everything…​

Distributing Movies

Gower Street Analytics.

Gower Street Analytics is an analytics company working in the film industry. Gower Street provides a platform for major distributors of films to manage release dates, helping them to decide exactly when a film should be released.

They use complex algorithms built in Clojure to simulate the performance of box offices around the world, trying out various release dates for movies. These simulations take into account population demographics, release dates and the expected performance of other movies.

I chatted with Gower Street co-founder Dimitrios Mitsinikos and his team to discuss their decision to use Clojure to power their business.

Jon: What is the goal of Gower Street?

Dimitrios: We want to build the analytics platform that we think is missing in the industry. There are many companies doing film analytics, but most are only doing 'green light' comparisons; looking at how well a film will do based on which actors are in it, for example.

We instead look at what films are being released around the same time as others, alongside the likelihood of a film hitting its estimate based on a variety of data points. We also monitor the market closely, improving the accuracy of our predictions as we get closer to a film’s release date.

All the large studios are now using our software, and we are in the process adding more countries to the platform.

Jon: How did Gower Street get started?

Dimitrios: We have a lot of experience in the movie industry - for the last 15 years I’ve worked in analytics at Universal Studios. We started two years ago, spending a lot of time on research and development, processing data feeds to track how movies are performing across the world, and figuring out exactly what platform the industry might need.

Jon: So if a studio wants to put out a film, then they can go to you to find out when would be the optimum time?

Dimitrios: Yes, that’s right. A studio can go online using our system and see what the film estimates are for the next 12-18 months. They can play with scenarios, for example: "Let’s swap Star Wars with Suicide Squad and see how much more or less each film might make."

We create algorithms that accurately model the box office, so customers can see the impact of choosing different release dates and monitor what is happening in a very competitive environment. For example our models show that one film could have made an extra $2.5m by releasing on the date that our system suggested.

It’s fascinating that even in a mature market like the UK, you can still significantly increase revenue by using analytics in this way.

gower st

Technology choices

Jon: Why did you choose Clojure to power the business?

Dimitrios: The core of our business is the simulation of the global box office. It’s all about processing complex and constantly changing data as fast as possible. When we first started our simulations could take five hours. It’s now about a minute. We have found functional programming a great fit for our team, especially those with a scientific background.

Team: Clojure might seem an unusual choice for performance-intensive simulation work; but it’s working out well. It’s a great fit for the problem, performant enough for our needs and it’s extremely expressive and readable, allowing us to iterate quickly on complex simulation algorithms.

Jon: How do you get it so fast?

Dimitrios: Through lots of optimisation and the ability to easily deploy using Amazon AWS.

Team: We’ve had to think outside the box, just throwing more machines at the problem might help, but then we’d have more problems with managing a distributed processing infrastructure e.g. a Hadoop cluster. To say more would be to enter a larger discussion of simulation and stochastic models that we probably don’t have time for here!

Jon: How did you go about adopting Clojure?

Dimitrios: Initially we started with Racket. Racket was nice and functional, but it was lacking support in certain areas. The developers came up with the idea to move to Clojure.

Team: Because the webapp for a previous iteration of our product was built in Racket, we were no strangers to parentheses. We tried Clojure for certain parts of the system at first, and it worked out very well. It has a larger ecosystem than Racket, so we’ve ended up adopting it across our stack. The usual reason of 'it works well with our existing Java systems' wasn’t a factor for us.

Dimitrios: As we’ve grown as a business we’re now using Clojure for almost everything - plain Clojure for back-end services and ClojureScript on the front-end.

Team: While we’ve gotten a lot from Clojure, we’re not obsessive, and recognise that it isn’t the best tool for every job. We’ve been experimenting with Serverless recently and this is one area where a lighter-weight solution may work better.

Jon: What is your team size?

Dimitrios: Our development team is currently five people and will be six shortly. We’re a fully remote team, scattered across the UK.

Team: You might think that remote would be an issue, but we’ve developed a way of collaboratively working together using appear.in video conferencing, Slack and tmux - we pair-program remotely much of the time and although we’re close to XP we’d prefer to refer to our team as agile with a small 'a'.

We get together for two days a month on-site in London as a team to work, eat and play - it helps us see each other as people rather than just co-workers. We have a flat-hierarchy and use a consensus-based decision-making model, so we’re not exactly your normal corporate sweatshop in this regard; we very much agree with Dan Pink’s ideas about Autonomy, Mastery and Purpose.

Hiring

Jon: Tell me about hiring?

Dimitrios: The team has grown organically, and based on word of mouth we’ve hired people. From my experience, it’s best for the development team to grow slowly, there’s a risk with teams of growing big too quickly.

Team: We’re actively creating an environment which promotes employee well-being and we take intentional steps to reduce inherent bias and to promote diversity. We focus on people and robust discussions rather than processes or tools.

We don’t require people to know Clojure to join us; just to have valuable experience to contribute and a willingness to learn and be open to new things. We’ve found that Clojure is a big draw for us - it attracts the kind of people who are more likely to see things as a holistic system rather than a collection of classes.

Permalink

Timeout Effect in Re-frame

Sometimes you need to delay an Event a certain amount of time. Let's say you want an alert to appear for twenty seconds before disappearing. We can store the alert message in the Database, but how do we make it disappear after the right time? We could set a timeout, using setTimeout(), but Event handlers should be pure to make them more testable. We need to turn setting a timeout into an Effect.

Let's make the shell of an Effect:

(rf/reg-fx
  :timeout
  (fn [{}]
    ))

We can pass in a time and an Event to dispatch. We then call setTimeout().

(rf/reg-fx
  :timeout
  (fn [{:keys [event time]}]
    (js/setTimeout
      (fn []
        (rf/dispatch event))
      time)))

That's great! Another feature of the JavaScript timeouts is the ability to cancel them with clearTimeout(). setTimeout() will return an id, which we later pass to clearTimeout().

We cancel timeouts if we store the id somewhere. Let's make a new atom to store these ids.

(defonce timeouts (reagent/atom {}))

We can use our own id when we create the Event so we can refer to it later:

(rf/reg-fx
  :timeout
  (fn [{:keys [id event time]}]

If we already have a timeout for this id, let's cancel it:

(rf/reg-fx
  :timeout
  (fn [{:keys [id event time]}]
    (when-some [existing (get-in @timeouts id)]
      (js/clearTimeout existing))

Then, we dissoc it from the timeouts atom.

(rf/reg-fx
  :timeout
  (fn [{:keys [id event time]}]
    (when-some [existing (get-in @timeouts id)]
      (js/clearTimeout existing)
      (swap! timeouts dissoc id))
    (when (some? event)
      (swap! timeouts assoc id
        (js/setTimeout
          (fn []
            (rf/dispatch event))
          time)))))

There! Now we can use it like the following. We can set the alert message and trigger its removal after 20 seconds.

(rf/reg-event-fx
  :set-alert
  (fn [cfx [_ msg]]
    {:db (assoc (:db cfx) :alert-message msg)
     :timeout {:id :alert-message
               :event [:remove-alert]
               :time 20000}}))

(rf/reg-event-db
  :remove-alert
  (fn [db]
    (dissoc db :alert-message)))

;; ...

(rf/dispatch [:alert-message "Please have a nice day!!"])

Try it out!

Permalink

Business benefits of Clojure SYNC

Does your manager know how many benefits their programmers will get from Clojure SYNC? And do they know how direct those benefits will be?

In this video, I go over the talks that are directly relevant to using Clojure at work–from hiring to improving the maintainability of our code. I also discuss the networking opportunities.

The post Business benefits of Clojure SYNC appeared first on Clojure SYNC.

Permalink

Optimistic Update in Re-frame

We’ve got a little bit of a problem on the web, especially with frontend apps. Our users are enjoying the fast, interactive experience of the app, which is running in the browser. Then some change needs to be propagated on the server. The server has to be the source of truth, since the browser tab is ephemeral. It could be reloaded at any time! Let’s say we click the “Like” button on a picture on social media. Obviously, we need to tell the server. The server knows all the likes and until it gets to the server, no other user can see the like. However, we want the user to know that the like happened or is at least in the process of happening.

In the ideal case, we send a message to the server, get a successful response, and update the UI; and all of that happens so fast the user never wonders whether it happens.

When the user clicks the like button, we dispatch the :like event, which sends a POST to the server. Then on success, we store some state for that post id in a hash map. If there's an error, we put an error message in the database.

(rf/reg-event-fx :like
  (fn [cofx [_ post-id]]
    {:http-xhrio {:url (str "http://server.com/" post-id "/like")
                  :method :post
                  :on-success [:confirm-like post-id]
                  :on-failure [:show-error "Could not like."]}}))

(rf/reg-event-db :confirm-like
  (fn [db [_ post-id]]
    (assoc-in db [:likes post-id] true)))

(rf/reg-event-db :show-error
  (fn [db [_ msg]]
    (assoc db :last-error msg)))

However, this ideal is rare. The message to the server and back is too slow. The user wants instant feedback. What can we do?

Pessimistic update

Well, one option is to save something to the database saying that we're trying to confirm the like. That way, we can show a loading spinner. In addition to sending the POST, we can store that we're trying. Let's create a new structure to house all of the state we'll need to keep track of. Let's use a map. It will have a status and a value.

{:status :stable
 :value  true}

 {:status    :loading
 :value      true
 :next-value false}

There are two statuses: :stable and :loading. Stable means we don't anticipate a response from the server. The value is what it is and is not changing. Loading means we are awaiting a response from the server. This is the case where we'd show a loading spinner.

We can make three operations on this new map. The first one, begin-load, we call when we start the Ajax request. It stores the value we're hoping it will change to under :next-value.

There's also an operation for succeeding, succeed-load, which sets the status to stable and updates the value.

And for failure responses, fail-load sets it back to stable, but leaves the value alone. I've also defined some Events that call these on a certain path in the Database.

(defn begin-load [state next-value]
  (cond
    (nil? state) ;; consider nil a stable value
    {:status :loading
     :value nil
     :next-value next-value}

    (= :stable (:status state))
    {:status :loading
     :value (:value state)
     :next-value next-value}

    (= :loading (:status state))
    (assoc state :next-value next-value)))

(defn succeed-load [state]
  (cond
    (nil? state)
    state

    (= :stable (:status state))
    state

    (= :loading (:status state))
    {:status :stable
     :value (:next-value state)}))

(defn fail-load [state]
  (cond
    (nil? state)
    state

    (= :stable (:status state))
    state

    (= :loading (:status state))
    {:status :stable
     :value (:value state)}))

(rf/reg-event-db :succeed-load
  (fn [db [_ path]]
    (update-in db path succeed-load)))

(rf/reg-event-db :fail-load
  (fn [db [_ path msg]]
    (-> db
      (update-in path fail-load)
      (assoc :last-error msg))))

Alright! Now how do we use this little bit of machinery? We use begin-load to start it off, telling it we're trying to make it true. We can use the generic :succeed-load and :fail-load events, giving them the path that they'll need when they are dispatched.

(rf/reg-event-fx :like
  (fn [cofx [_ post-id]]
    ;; let's start the ajax request
    {:http-xhrio {:url (str "http://server.com/" post-id "/like")
                  :method :post
                  ;; we can call our new Events
                  :on-success [:succeed-load [:likes post-id]]
                  :on-failure [:fail-load    [:likes post-id] "Failed to like post."]}
     ;; and let's begin the load with next value true
     :db (update-in (:db cofx) [:likes post-id] begin-load true)}))

In our Component, if the status is stable, we show the value. Otherwise, we show a spinner. That's one option. We show exactly the truth: we're loading. The actual state of things is in limbo.

Optimistic update

But there's another way, which is to tell a 90% truth. 90% of the time, the Ajax request will succeed with no problem. So 90% of the time, you can just pretend like it's going to work, showing what it would look like if it were a success immediately. You then you have to deal with those 10% of cases where it didn't work--you actually lied to the user. But the lucky thing is that your users will get feedback immediately. That feedback is correct the vast majority of the time. And sometimes, that's good enough.

In this app, liking something is not an essential feature, so we can update it optimistically. With just a few small changes, we can do an optimistic load. The first thing is to immediately set the value to the next value, and store the old value.

{:status :stable
 :value  true}

 {:status    :loading
 :value      false
 :old-value true}

We change our begin-load function so it doesn't save the current value and the next value anymore. Now, it will save the current value, which it optimistically sets to next-value, and the old value, which we'll need in case of failure. Success means we throw away the old value. And failure means we set the current value to the old value.

(defn begin-load [state next-value]
  (cond
    (nil? state)
    {:status :loading
     :value next-value
     :old-value nil}

    (= :stable (:status state))
    {:status :loading
     :value next-value
     :old-value (:value state)}

    (= :loading (:status state))
    (assoc state :value next-value)))

(defn succeed-load [state]
  (cond
    (nil? state)
    state

    (= :stable (:status state))
    state

    (= :loading (:status state))
    {:status :stable
     :value (:value state)}))

(defn fail-load [state]
  (cond
    (nil? state)
    state

    (= :stable (:status state))
    state

    (= :loading (:status state))
    {:status :stable
     :value (:old-value state)}))

I didn't repeat the Events because they're the same. We just had to change these three operations. Well, we also have to change the view. It should never show the loading spinner now. It should just show the current value and there should still be a message when there's a failure.

Permalink

Gettin’ schwifty with Clojure’s core.async

A few months ago, somebody in the office pointed us to an interesting  job offer from CartoDB which looked as follows: What follows is a technical test for this job offer at CARTO: https://boards.greenhouse.io/cartodb/jobs/705852#.WSvORxOGPUI Build the following and make it run as fast as you possibly can using Python 3 (vanilla). The faster it runs, the ...

Permalink

Step inside :cljsbuild

In an ongoing effort to document more of the Clojure ecosystem, I’m going to share with you a day’s headache around the Clojurescript build process. Don’t get me wrong! I adore Clojurescript. However it’s Leiningen configuration is opaque and lends to confusion. What’s worse, the mystery is papered over with calls to lein new and we hope for the best!

No more! I shall rescue you. Here’s a bucket. We’ll save the Titanic yet!

For interested parties, I wanted to build a Clojure/Clojurescript portable library. It was important that the test suite execute in both environments as well as being easy for the programmer to execute.

profile.clj

Here’s the interesting bits of the configuration all in one place.

:plugins [[lein-cljsbuild "1.1.7"]]
:source-paths ["src"]
:test-paths ["test"]
:aliases
{"cljs-test" ["cljsbuild" "test" "unit-tests"]
"test-all" ["do" "clean," "test," "cljsbuild" "once"]
"cljs-auto-test" ["cljsbuild" "auto" "tests"]}
:cljsbuild
{:test-commands {"unit-tests" ["node" "target/unit-tests.js"]}
:builds
{:tests
{:source-paths ["src" "test"]
:notify-command ["node" "target/unit-tests.js"]
:compiler {:output-to "target/unit-tests.js"
:optimizations :none
:target :nodejs
:main your.awesome.code.core-test}}
:production
{:source-paths ["src"]
:compiler {:output-to "target/production.js"
:optimizations :advanced}}}}

:plugins

The cljsbuild project is the central control. It empowers Leiningen with commands like lein cljsbuild test. It handles the compiler through the configuration given here.

:aliases

These serve a duel purpose in my projects. On the one hand they make convenient shortcuts for frequent commands. As importantly, it serves as breadcrumbs for others to understand expectations of how the build tools are configured.

In the first instance, we expose cljs-test as a command to power the test and test-command unit-tests. More on :test-command below, but note that it references cljsbuild and node.

test-all is a set of commands that clean the repository, run the Clojure tests, then compile and run the Clojurescript tests. This is accomplished with the leiningen do command and the :notify-command below.

Finally, cljs-auto-test continuously compiles and runs the tests on every source code change.

:cljsbuild

This is the meat-and-potatoes of most Clojurescript projects, and probably the least understood. I won’t pretend it’s easy (I did spend a day pulling my hair out) but it’s understandable in pieces.

:test-commands

One of the most important lines here, and rarely mentioned. While Clojurescript will happily compile all your code and tests, that code must be run against a real-live Javascript engine. The vector is the shell command to invoke the Javascript engine with your code. The path portion used will match the :output-to key below.

All of this is called by lein cljsbuild test <name>. In this case, the name is unit-tests. This compiles our Clojurescript into Javascript, then runs NodeJS with our newly minted Javascript code.

:builds

I have a pet peeve. This key supports both a vector as input and a map input. This leads to chaos and confusion when you look at other people’s code.

Here I use the map form, and I would encourage others as well to stick to this convention. Although I would love to hear thoughtful benefits of the vector form.

Pet peeve aside, the :builds values contain configurations for the Clojurescript compiler and build configurations. This is how we control where and how our Javascript is generated.

:source-paths

Here we point the compiler to all the folders we want to build from. In the :tests config, we want both the source code and test code. In :production the source code is enough.

:notify-command

It’s a callback when the compiler finishes. It calls a shell command of your choice. Here I call NodeJS to run our generated Javascript+Testcode on every change to the source code so I can see my unit-test results.

:output-to

Once your Javascript is built, where should the compiler put it? This is important, as other tools like NodeJS need to know this location to run our test suite.

When building Single Page Applications and other advanced Clojurescript applications, it is critical you know where to place your generated JS files. Often other tools pick up on output from the Clojurescript compiler.

:optimizations

How aggressive is the code optimization and minification? For :tests I have to avoid using :whitespace as NodeJS complains loudly. In :production I use :advanced to get the smallest deployable.

:output-dir

Often builds and compilations will create extra files. This allows you to specify where those go. This must be unique across the different :builds configurations, otherwise incremental builds will break and worse!

:target :nodejs

Normally, you target a browser. That means you have documents you write to. However, NodeJS lacks these browser features, so this key forces the compiler to generate the raw Javascript for NodeJS consumption.

:main

Usually this is the main entrypoint of your application. In the case of the unit tests, this points to the namespace where my test runner starts the unit tests.

You see, because you send your JS to another Javascript Engine to run tests, it must explicitly call the (run-tests) function somewhere. In this case, it is pointed at the test suite’s main namespace, where the function is called directly in the root.

Extra details

In your test suite namespace, you need two functions: enable-console-print! and run-tests.

(ns your.awesome.code.core-test
(:require [cljs.test :refer [run-tests]]
[your.awesome.code.test-set-a]
[your.awesome.code.test-set-b])
; Turn on console printing. Node can't print to *out* without.(enable-console-print!)
; This must be a root level call for Node to pick it up.
(run-tests 'your.awesome.code.test-set-a
'your.awesome.code.test-set-b)

Compile Onward!

I spent quite some time pouring over the documentation and source files to figure out how all these fit together. Hopefully I saved you a bit of time.

More information

If you’re seeing *print-fn* errors.

How to set the clojurescript *print-fn* fn in nodejs

An exhaustive list of :compiler options available.

ClojureScript - Compiler Options

Sample cljsbuild configuration, annotated.

emezeske/lein-cljsbuild

Permalink

JOB: Software Developer

Location: Anywhere within the US
Target Start Date: February 1, 2018
Salary Range: $112,000 to $130,000 per year (depending on experience. Salary band given for our Brooklyn, NY headquarters; remote salaries will vary by cost-of-living adjustment)
Benefits: Vision, dental, & medical insurance; 403(b) retirement savings plan; generous minimum vacation policy; parental leave; long-term disability; employee assistance program
Level: Mid-level to senior

At Democracy Works, we believe voting should fit the way we live. To that end, we build technology for both voters and election administrators that simplifies the process and ensures that no voter should ever have to miss an election.

TurboVote, our first service, helps voters register, stay registered, and cast a ballot in every election, from municipal to national. TurboVote signed up its millionth voter in 2016 by building the largest college, nonprofit, and corporate voter engagement coalition in the country, including 176 campuses, companies like Starbucks, Univision, Facebook, Google, Snapchat, and dozens more. Our other work includes the Voting Information Project, whose polling-place data received 123 million impressions in 2016, an Election Technology Cooperative to provide affordable, voter-centered technology to election administrators, and Ballot Scout, which tracks absentee ballots through the mail, providing transparency in the vote-by-mail process and making it easier to follow up when things go awry.

These products are the work of our eight-person developer team. Most of our development involves writing microservices in Clojure running in Docker containers on Kubernetes (soon) and hosted on AWS. These services communicate over RabbitMQ and store their data in Datomic. Our users primarily interact with web apps written in ClojureScript and re-frame. We also have projects that use JavaScript, Node, React, Python, and PostgreSQL. We hope you have experience with some of these technologies and are excited to get experience with the rest.

We pair program, collaborate with product managers, and make sure our efforts deliver value to voters. We rotate roles and projects on our team so that everyone gets a variety of experience and working relationships and can bring their unique strengths to as wide a swath of our work as possible.

To apply:
Send a short email with resume, addressed to Chris and Wes, at work@democracy.works with the subject line “Will code for democracy” to begin the application process. Please also include how you found this job listing. Qualified candidates who meet the above requirements will have the opportunity to complete an anonymized skills evaluation before we schedule an interview. Based on the application, evaluation results, interviews, and reference checks, one person will be selected for the position.

Applications will be accepted and interviews will be conducted on a rolling basis.

Democracy Works is committed to diversity and inclusion in everything we do and aspires to have a team which is representative of the voters we serve. When hiring, we practice proactive outreach to top talent that’s underrepresented in our sector (including Latinx, Black, AAPI, and Indigenous candidates), and we offer every candidate an anonymized skills evaluation, to reduce implicit bias and resume-dependency in our process. We're a woman- and gay-founded startup, and promote an inclusive culture that stands against racism, sexism, homophobia, and ableism (to name a few). To be explicit, we strongly encourage applicants of all races, ethnicities, political party associations, religions (or lack thereof), national origins, sexual orientations, genders, sexes, ages, abilities, and branches of military service. Feel free to contact work@democracy.works if you have any questions about our commitment to inclusion or about general hiring practices.

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.