Full-Stack Engineer at ICX Media (Full-time)

  • Do you want to work to solve big problems with great people?
  • Do you want to change the way media is consumed?
  • Do you want to question norms?
  • Do you want to deploy solutions to production hourly?

ICX Media is a product and technology startup that is building success for content creators through imaginative use of data and technology. We are building our product for digital content creators (filmmakers, directors, producers) to help their growing need to find the right audience, understand that audience, and monetize their creations in the best way possible.

We are using data, analytics and machine learning to create a visually brilliant application for independent content creators. To pull this off we will be using tools such as Clojure, DynamoDB, Docker and AngularJS to allow us to solve problems quickly and push traditional development boundaries. In addition to full-time employees, we are always interested in opening our doors to Summer interns and filling out our 2016 hiring roster.

NOTABLE BENEFITS

  • Enterprise-level Healthcare
  • Commuter
  • Mobile Phone Reimbursement
  • Unlimited Vacation
  • Remote Collaboration
  • Flexible Work Schedule

OUR COMMITMENT TO YOU

  • Dedicated and knowledgeable peers
  • Work with cutting-edge technology
  • No forbidden technologies
  • Always question why we are doing things
  • Define your own work rhythm

YOUR COMMITMENT TO US

  • Getting the job done
  • Openness to a diverse set of ideas
  • Willingness to do anything to be successful
  • Play with others, both physical and virtual
  • Situational awareness and common sense

POLYGLOT

  • Functional programming in Clojure, Scala, Ruby, Pyhton, and/or other
  • Big Data & NoSQL databases technologies such as Datomic, Spark and Cassandra
  • Variety of front-end technologies such as AngularJS, React, D3

Get information on how to apply for this position.

Permalink

Clojure for Fun and Profit

When we decided to build our Shutl iOS application we had some decisions to make. One of these decisions was the technology to use to build the back end.

clojure-logo-120bThis back end is a pretty simple service. First, it provides an API that combines and transforms data retrieved from different services so the client can present it. Secondly, it listens to events and notifies the clients accordingly. Both of these use cases could benefit from or require asynchronous processing.

At Shutl, Ruby is our default programming language, so usually it would be a very straightforward decision to choose Sinatra and Sidekiq. But the team is always willing to try new things, and Clojure has caught our attention for some time. Our back-end project sounded like a good opportunity to test out Clojure. It wasn’t a critical component, but it could potentially grow bigger. The logic would be pretty simple, mostly about transforming data, and our design goals — concurrency and parallelization — played to Clojure’s strengths. So we decided to give it a shot.

First choices

As complete beginners, we took a look at the Clojure ecosystem. Although some web frameworks exist in Clojure, the general approach seems to be combining small libraries that do one thing well.

After some research, this was our initial setup:

One big advantage of using Clojure is that it runs on the JVM and offers idiomatic forms to invoke Java code. Clojure therefore makes it very easy to use Java libraries and gives you access to the great ecosystem of Java libraries, like Logback. For some of them (I am looking at you, Hystrix), wrappers also allow a more idiomatic usage.

How we worked

Coming from a Ruby perspective into Clojure, we needed a different mindset but we were halfway there. We embraced immutability a long time ago, functions as first-class citizens are a familiar concept, and Ruby is also dynamically typed.

There were only two things we had to get used to: zero objects and lots of parentheses. The first one was easier than expected; it is kind of easy to get used to the way you write code in Clojure. You write small, concise functions and then you combine them. So simple, so easy. But what about the syntax?

I am not going to lie. It can be overwhelming at the beginning. Then, as you use it, you discover one of the things I love the most about Clojure: it is a very simple language. But simple doesn’t mean powerless, just the opposite. Being a dialect of Lisp, the core of the language is very small but the standard library is complete and well documented.

Clojure allows you to focus on what really matters — what your code is supposed to do — rather than on learning how to do things. This simplicity, combined with the use of the REPL, allows you to easily test and increase your knowledge.

And a tiny, little detail: it is fun.

Epilogue

A few months later, the application went live and everybody was happy. It has been running in production since the fall of 2015, we haven’t had any issue, and it is performing really well.

Obviously, we still have lots to learn. We didn’t use every single feature in the language (did somebody say writing macros?), but just because we didn’t need them. We are very proud of what we built.

All in all the experience was very positive. It was so positive that recently we decided to use Clojure as the back-end language for a new product… stay tuned!

Permalink

Event Stream Processing - Managing time series with core.async

Introduction

In my last post I went through the code for a data flow using a product stock level tracker by combining channels and maintaining state around the go-loop

In this post I want to show how to segregate data from a stream into time series.

Simple stock management

In this use case we want to notify the supplier if we have a spike in demand. This spike example is fairly trivial but we will see that core.async can handle many millions of events per minute and provide fine grained time series with very little code.

Data first

As usual in this series we will outline the simple data we use in our model:

; Order
{:item-id A123 :description "Windscreen wiper" :quantity 2 :customer-id C234}

Timing out of the box

core.async does not provide any models for managing time in the library.

The addition of this support however is relatively simple and suprisngly concise. Well, maybe not that surprising by now.

The development of this model was prompted by a short conversation with @jgdavey in the core.async channel on Slack

If you were to implement yourself, you could make a go that simply pulls from a channel, and adds to another as a [timestamp item] pair, then finally pushes into an unbounded chan that has a transducer that filters based on age of that timestamp.

Joshua Davey

I couldn’t get my head around the suggestion at first but I decided to give it a try - what could possibly go wrong?

Oh, and Joshua has his own implementation which is interesting in its own right.

Order and delivery generation

(defn gen-timed-orders [frequency-millis coll]
  (let [out (chan)]
    (go-loop []
      (do
        (>! out [(t/now) (rand-nth coll)])
        (<! (timeout frequency-millis))
        (recur)))
    out))

(defn gen-random-deliveries [max-wait-millis coll]
  (let [out (chan)]
    (go-loop []
      (do
        (>! out [(t/now) (rand-nth coll)])
        (<! (timeout (rand-int max-wait-millis)))
        (recur)))
    out))

Here we generate infinite streams of data with time stamps. That will be used in later examples to aggregate data into time series.

They differ only in the way that orders are consistent and deliveries are random. This is not a perfect model of the real world (yeah, I know) but is good enough for this purpose.

Blocking without Thread/sleep == SCAAAAALE!

One aspect to note in the above code is the timeout function. It reads from a virtual channel and effects a pause on the generation like Thread/sleep but does not block.

Not blocking means that it is OK to start 100s or 1000s of go blocks with very little CPU or RAM overhead. This is similar in effect to the way that NGINX is architected.

Don’t change that channel

I’m going to show this next function in to forms. The first form shows how to use conditionals based on channel

(defn stock-levels [orders deliveries]
  "Stock levels - also includes demand (negative stock numbers)"
  (let [out (chan)
        dec-stock (partial modify-stock dec)
        inc-stock (partial modify-stock inc)]
    (go-loop [stock {}]
      (if-let [[data chan] (alts! [orders deliveries])]
        (let [[_ item] data
              update-operation (condp = chan
                                 orders dec-stock
                                 deliveries inc-stock)]
          (do
            (>! out (update-operation stock item))          ; does not work - need to manage 'stock'
            (recur stock)))
        (close! out)))
    out))

In this first showing of the code we see the ability to read many channels at once using alts! which receives the data and the channel that was read.

In this example we use condp to run differing code based on the channel that was read. In this case we set the method to adjust stock.

#Re-Stating our intentions

The problem we have here is that we need to retain the latest value of the stock otherwise we will only ever report 0 or 1. That might might be useful if have an external aggregator but we can also aggregate in the go-loop directly

(defn stock-levels [orders deliveries]
  "Stock levels - also includes demand (negative stock numbers)"
  (let [out (chan)
        dec-stock (partial modify-stock dec)
        inc-stock (partial modify-stock inc)]
    (go-loop [stock (into #{} (map #(assoc {} :id (:id %) :count 0) parts))]
      (if-let [[data chan] (alts! [orders deliveries])]
        (let [[_ item] data
              update-operation (condp = chan
                                 orders dec-stock
                                 deliveries inc-stock)]
          (if-let [[modified-stock updated-item] (update-operation stock item)]
            (do
              (>! out updated-item)
              (recur modified-stock))))
        (close! out)))
    out))

Using the parameters to the go-loop we can initiate state and then maintain it via recur.

In this case we do the simplest possible setting for stock to 0 when our function starts. One can imagine more complex initialisations!

In the loop we then pass the stock collection to the modify-stock function which can return a new version of the stock collection.

Here we exploit the fact that Clojure collections are very efficient with respect to minimising the costs of each new version.

This stock list could easily be scaled to tens of thousands of parts without adding any latency costs (barring side effects of swapping although luckily we are no longer limited to 32k, 32Mb or even 32Gb of memory like in the good old days).

Modify without side effects

To finalise the example here is the code for modify-stock which runs the provided function to obtain the new value using the existing count on the item.

(defn modify-stock [count-adjust-fn stock item]
  (let [current-value (first (clojure.set/select #(= (:id %) (:id item)) stock))
        new-value (assoc current-value :count (count-adjust-fn (:count current-value)))
        updated-stock (conj stock new-value)]
    [updated-stock new-value]))

One nice aspect of using Clojure’s set data structure is that we can use conj as a succinct upsert.

Summary

In this example we saw how to read multiple channels in one go-loop and distinguish data from those channels. Further we saw that we can maintain aggregations and state around the go-loop to increase both power and simplicity.

Next next next

In the final example of this short series I will show how to track data within series of time.

And finally - Thanks!

Thanks for making it through. I have a better understanding of core.async after writing this and I hope that’s true for you too!

Zing me or ping me if this was useful via Twitter or in the comments below.

<noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript>

Permalink

Clojure/Clojurescript Web Developer, Ontario

From: https://kirasystems.com/careers#op-103338-clojureclojurescript-web-developer

We’re looking for a developer to work on our Clojure/ClojureScript/Om web stack. Our team is small, pragmatic, and inquisitive; we love learning new technologies and balance adoption with good analysis. Strong preference for someone who lives in or can commute to the Toronto area and come into our offices on a daily basis. Ideally, we’d like to hire someone that might be interested in assuming a leadership role in our development team over time.

Web technology can be built better. If single-page web design driven by a reactive data model sounds interesting to you, get in touch!

Technologies we use:

HTML, CSS, SASS,
Clojure, ClojureScript,
SQL, PostgreSQL, Java,
and experience with web app security.
You should have knowledge of some of these. Most of all we look for those interested in learning.

This position starts immediately.


Permalink

Fairy Tale Word Vectors

http://c2.staticflickr.com/2/1558/24654386380_bda44419a8_n.jpg">

This post continues our exploration from the last blog post Why Hyperdimensional Socks Never Match. We are still working our way through Kanerva’s paper. This time, with the basics of hypervectors under our belts, we’re ready to explore how words can be expressed as context vectors. Once in a high dimensional form, you can compare two words to see how similar they are and even perform reasoning.

To kick off our word vector adventure, we need some words. Preferring whimsy over the Google news, our text will be taken from ten freely available fairy tale books on http://www.gutenberg.org/.

Gather ye nouns

Our goal is to assemble a frequency matrix, with all the different nouns as the rows and the columns will be the counts of if the word appears or not in the document. Our matrix will be binary with just 1s and 0s. The document will be a sentence or fragment of words. A small visualization is below.

Noun Doc1 Doc2
flower 1 0
king 0 1
fairy 1 0
gold 1 1

The size of the matrix will be big enough to support hypervector behavior, but not so big as to make computation too annoyingly slow. It will be nouns x 10,000.

The first task is to get a set of nouns to fill out the rows. Although, there are numerous online sources for linguistic nouns, they unfortunately do not cover the same language spectrum as old fairy tale books. So we are going to collect our own. Using Stanford CoreNLP, we can collect a set of nouns using Grimm’s Book as a guide. There are about 2500 nouns there to give us a nice sample to play with. This makes our total matrix size ~ 2500 x 10,000.

Now that we have our nouns, let’s get down to business. We want to create an index to row to make a noun-idx and then create a sparse matrix for our word frequency matrix.

```clojure (ns hyperdimensional-playground.context-vectors (:require [clojure.core.matrix :as m]

        [clojure.core.matrix.linear :as ml]
        [clojure.string :as string]
        [hyperdimensional-playground.core :refer [rand-hv cosine-sim mean-add inverse xor-mul]]
        [hyperdimensional-playground.fairytale-nouns :refer [fairy-tales-nouns book-list]]))

(m/set-current-implementation :vectorz) ;; size of the hypervectors and freq matrix columns (def sz 10000) ;; The nouns come from a sampling of Grimm’s fairy tale nouns these will ;; make up the rows in the frequency matrix (def noun-idx (zipmap fairy-tales-nouns (range))) (def freq-matrix (m/new-sparse-array [(count fairy-tales-nouns) sz])) ```

The next thing we need to do is to have some functions to take a book, read it in, split it into documents and then update the frequency matrix.

Random indexing for the win

The interesting thing about the update method is that we can use random indexing. We don’t need to worry about having a column for each document. Because of the nature of hyperdimensions, we can randomly assign 10 columns for each document.

```clojure (defn update-doc! “Given a document – upate the frequency matrix using random indexing” [doc] (let [known-nouns (clojure.set/intersection fairy-tales-nouns (set doc))]

; use positive random indexing
(doall (repeatedly 10 #(doseq [noun known-nouns]
                   (m/mset! freq-matrix (get noun-idx noun) (rand-int sz) 1))))))

```

The whole book is processed by slurping in the contents and using a regex to split it up into docs to update the matrix.

```clojure (defn process-book “Load a book and break it into sentence like documents and update the frequency matrix” [book-str] (let [book-text (slurp book-str)

    docs (partition 25 (map string/lower-case
                            (string/split book-text #"\s|\.|\,|\;|\!|\?")))
    doc-count (count docs)]
(println "processing:" book-str "(with" doc-count "docs)")
(doall (map-indexed (fn [idx doc]
                      (when (zero? (mod idx 1000)) (println "doc:" idx))
                      (update-doc! doc))
                    docs))
(println "DONE with " book-str)))

```

We can now run the whole processing with:

```clojure (doseq [book book-list]

(process-book book))

```

On my system, it only takes about 3 seconds.

Great! Now we have hypervectors associated with word frequencies. They are now context word vectors. What can we do with them.

How close is a king to a goat?

One of the things that we can do with them is find out a measure of how closely related the context of two words are by a measure of their cosine similarity. First, we need a handy function to turn a string word into a word vector by getting it out of our frequency matrix.

```clojure (defn wv [word] “Get a hypervector for the word from the frequency matrix” (let [i (get noun-idx word)]

(assert (not (nil? i)) (str word " not found"))
(m/slice freq-matrix i)))

```

Then we can make another nice function to compare two words and give a informational map back.

```clojure (defn compare-wvs “Compare two words and give the cosine distance info map” [word1 word2] (let [wv1 (wv word1)

    wv2 (wv word2)]
(when (not= word1 word2)
  {:word1 word1
   :word2 word2
   :cosine (cosine-sim wv1 wv2)})))

```

Let’s take a look at the similarities of some words to king.

```clojure (sort-by :cosine[(compare-wvs “king” “queen”)

             (compare-wvs "king" "prince")
             (compare-wvs "king" "princess")
             (compare-wvs "king" "guard")
             (compare-wvs "king" "goat")])

;; ({:word1 “king”, :word2 “goat”, :cosine 0.1509151478896664} ;; {:word1 “king”, :word2 “guard”, :cosine 0.16098893367403827} ;; {:word1 “king”, :word2 “queen”, :cosine 0.49470535530616655} ;; {:word1 “king”, :word2 “prince”, :cosine 0.5832521795716931} ;; {:word1 “king”, :word2 “princess”, :cosine 0.5836922474743367}) ```

As expected, the royal family is closer to the king then a guard or goat is.

One of the interesting things is that now we can do addition and subtraction with these word vectors and see how it affects the relation with other words.

Boy + Gold = King, Boy + Giant = Jack

We can take a look at how close boy and king are together by themselves.

clojure (cosine-sim (wv "boy") (wv "king")) ;=> 0.42996397142253145

Now we can add some gold to the boy and that new word vector will be closer to king than boy was alone.

```clojure (cosine-sim (mean-add (wv “boy”) (wv “gold”))

          (wv "king")) ;=> 0.5876251031366048

```

Doing the same for boy and jack, we find that adding a giant moves the context closer.

```clojure (cosine-sim (wv “boy”) (wv “jack”)) ;=> 0.33102858702785953 ;; boy + giant = jack (cosine-sim (mean-add (wv “giant”) (wv “boy”))

          (wv "jack")) ;=>0.4491473187787431

```

Amusingly, a frog and a princess make a prince.

clojure ;;; frog + princess = prince (cosine-sim (wv-add "frog" "princess") (wv "prince")) ;=> 0.5231641991974249

We can take this even farther by subtracting words and adding others. For example a similarity to the word queen can be obtained by subtracting man from king and adding woman.

```clojure ;;; queen= (king-man) + woman (cosine-sim (wv “queen”)

          (mean-add (wv "woman") (wv-subtract "king" "man"))) ;=>0.5659832204544486

```

Similarly, a contextual closeness to father can be gotten from subtracting woman from mother and adding man.

```clojure (cosine-sim (wv “father”)

          (mean-add (wv "man") (wv-subtract "mother" "woman"))) ;=>0.5959841177719538

```

But wait, that’s not all. We can also do express facts with these word vectors and reason about them.

Reasoning with word vector with the database as a hyperdimensional value

The curious nature of hypervectors allows the storage of multiple entity, attributes in it and allow the retrieval of the likeness of them later by simple linear math – using only xor multiplication and addition. This gives us the database as a value in the form of a high dimensional vector.

For an example, say we want to express the fact that Hansel is a brother of Gretel. We can do this by adding the xor product of brother with hansel and the product of brother with Gretel.

```clojure ;; hansel is the brother of gretel ;; BH + BG (def hansel-brother-of-gretel (mean-add

(xor-mul (wv "brother") (wv "hansel"))
(xor-mul (wv "brother") (wv "gretel"))))

```

Also we can express that Jack is a brother of Hansel.

```clojure (def jack-brother-of-hansel (mean-add

 (xor-mul (wv "brother") (wv "jack"))
 (xor-mul (wv "brother") (wv "hansel"))))

```

We can add these two facts together to make a new hypervector value.

```clojure (def facts (mean-add hansel-brother-of-gretel

                   jack-brother-of-hansel))

```

Now we can actually reason about them and ask questions. Is Jack a brother of Hansel? With a high cosine similarity, we can assume the answer is likely.

```clojure ;; is jack the brother of hansel? (cosine-sim (wv “jack”) (xor-mul (mean-add (wv “brother”) (wv “gretel”))

        facts)) ;=>0.8095270629815969

```

What about someone unrelated. Is Cinderella the brother of Gretel? – No

```clojure ;; is cinderella the brother of gretel ? (cosine-sim (wv “cinderella”) (xor-mul (mean-add (wv “brother”) (wv “gretel”))

        facts)) ;=>0.1451799916656951

```

Is Jack the brother of Gretel – Yes

```clojure ;; is jack the brother of gretel ? (cosine-sim (wv “jack”) (xor-mul (mean-add (wv “brother”) (wv “gretel”))

        facts)) ;=> 0.8095270629815969

```

We can take this further by adding more facts and inventing a relation of our own.

Siblings in Hyperspace

Let’s invent a new word vector that is not in our nouns – siblings. We are going to create new random hypervector to represent it.

clojure (def siblings (rand-hv))

We will define it in terms of word vectors that we already have. That is, siblings will be a the sum of brother + sister. We XOR multiply it by siblings to associate it with the hypervector.

```clojure (def siblings-brother-sister

(mean-add (xor-mul siblings (wv "brother")) (xor-mul siblings (wv "sister"))))

```

Now we can add some more facts. Gretel is a sister of Hansel.

```clojure ;; gretel is the sister of hansel ;; SG + SH (def gretel-sister-of-hansel

(mean-add
 (xor-mul (wv "sister") (wv "gretel"))
 (xor-mul (wv "sister") (wv "hansel"))))

```

Gretel is also a sister of Jack.

```clojure ;; gretel is the sister of jack ; SG + SH (def gretel-sister-of-jack

(mean-add
 (xor-mul (wv "sister") (wv "gretel"))
 (xor-mul (wv "sister") (wv "jack"))))

```

Collecting all of our facts into one hypervector (as a database).

```clojure (def facts (mean-add hansel-brother-of-gretel

                   jack-brother-of-hansel
                   gretel-sister-of-jack
                   gretel-sister-of-hansel
                   siblings-brother-sister))

```

Now we can ask some for questions.

Are Hansel and Gretel siblings? – Yes

clojure ;; are hansel and gretel siblings? (cosine-sim (mean-add (wv "hansel") (wv "gretel")) (xor-mul siblings facts)) ;=>0.627015379034067

Are John and Roland siblings – No

clojure ;; are john and roland siblings? (cosine-sim (mean-add (wv "roland") (wv "john")) (xor-mul siblings facts)) ;=> 0.1984017637065277

Are Jack and Hansel siblings? – Yes

```clojure (cosine-sim

(mean-add (wv "jack") (wv "hansel"))
(xor-mul siblings facts)) ;=>0.48003572523507465

```

It is interesting to think of that nothing is stopping us at this point from retracting facts by simply subtracting the fact encoded word vectors from our “database” value and making a new value from it.

Conclusions

In this fun, but casual exploration of word vector we have seen the potential for reasoning about language in a way that uses nothing more complicated than addition and multiplication. The ability to store dense information in hypervectors, extract it with simple methods, and flexibly collect it randomly, shows its versatility and power. Hyperdimensional vectors might hold the key to unlocking a deeper understanding of cognitive computing or perhaps even true artificial intelligence.

It is interesting to note that this technique is not limited to words. Other applications can be done the same way. For example a video recommendation using a hypervector with movie titles. Or perhaps even anomaly detection using sensor readings over a regular weekly time period.

Looking over our journey with word vectors. At the beginning it seemed that word vectors were magical. Now, after an understanding of the basics, it still seems like magic.

If you are interested in exploring further, feel free to use my github hyperdimensional-playground as a starting point.

Permalink

Senior Functional Web Engineer at Front Row Education (Full-time)

Position

Senior Functional Web Engineer to join fast-growing education startup transforming the way 3+ million K-8 students learn Math and English.

What you will be doing

Architect, design and develop new applications, tools and distributed systems for the Front Row ecosystem in Haskell, Flow, PostgreSQL, Ansible and many others. You will get to work on your deliverable end-to-end, from the UX to the deployment logic.

Once you're an integral part of the team you will act as Dev Lead and oversee the success of your team

Mentor and support more junior developers in the organization

Create, improve and refine workflows and processes for delivering quality software on time and without incurring debt

Work at our offices in San Francisco as part of a very small (there's literally half a dozen of us!), world-class team of engineers with a track record of rapidly delivering valuable software to millions of users.

Work closely with Front Row educators, product managers, customer support representatives and account executives to help the business move fast and efficiently through relentless automation.

Why you should join Front Row

Our mission is important to us, and we want it to be important to you as well: millions of students learn math using Front Row every month. Our early results show students improve twice as much while using Front Row than their peers who aren’t using the program.

You’ll be THE first Senior Engineer ever at Front Row, which means you’ll have an immense impact on our company, product, and culture; you’ll have a ton of autonomy and responsibility; you’ll have equity to match the weight of this role. If you're looking for an opportunity to both grow and do meaningful work, surrounded and supported by like-minded professionals, this is THE place for you.

You will be working side by side with many well known world-class personalities in the Haskell and Functional Programming community whose work you've likely used. Front Row is an active participant to the Open Source community and contributor to some of the most popular Haskell libraries.

A lot of flexibility: while we all work towards the same goals, you’ll have a lot of autonomy in what you work on. You can work from home up to one day a week, and we have a very flexible untracked vacation days policy

The company and its revenue are growing at a rocketship pace. Front Row is projected to make a massive impact on the world of education in the next few years. It's a once in a lifetime opportunity to join a small organization with great odds of becoming the Next Big Thing.

Must haves

  • You have experience doing full-stack web development. You understand HTTP, networking, databases and the world of distributed systems.
  • You have functional programming experience.
  • Extreme hustle: you’ll be solving a lot of problems you haven’t faced before without the resources and the support of a giant organization. You must thrive on getting things done, whatever the cost.
  • Soft skills: we want you to move into a leadership position, so you must be an expert communicator

Nice-to-haves

  • You have led a software development team before
  • You have familiarity with a functional stack (Haskell / Clojure / Scala / OCaml etc)
  • You understand and have worked all around the stack before, from infrastructure automation all the way to the frontend
  • You're comfortable with the Behavior-Driven Development style
  • You have worked at a very small startup before: you thrive on having a lot of responsibility and little oversight
  • You have worked in small and effective Agile/XP teams before
  • You have delivered working software to large numbers of users before

Benefits

  • Competitive salary
  • Generous equity option grants
  • Medical, Dental, and Vision
  • Catered lunch and dinner 4 times a week
  • Equipment budget
  • One flexible work day per week
  • Working from downtown SF, very accessible location
  • Professional yet casual work environment

Get information on how to apply for this position.

Permalink

New library: mount-lite

Just enough structure

As many of the Clojurists will know, the reloaded workflow has been a popular way of developing Clojure(Script) applications. A library that supports this for more than two years now is Component. While I have used Component successfully, I was happy to learn about a new kid on the block a few months ago, called mount.

There have always been proponents and opponents to Component - as it is a fairly opinionated library. And whether the way mount tackles the reloaded workflow is any improvement is being discussed as well. I for one like it very much. For me it delivers on what it set out to do: keeping Clojure fun. It feels more Clojuresque, but I find it has also the following important properties:

  • It is small (at least from a user’s perspective), less opinionated, and provides just enough structure (oddly enough) and - more importantly - nothing more.
  • It still allows for protocols and records, if that’s applicable.
  • It forces you to put state where it belongs - in the application namespaces, not in library namespaces.
  • You don’t need to pass the stateful parts around everywhere explicitly.
  • It is more flexible in (re)starting or stopping parts of the system.
  • It is REPL friendly.

I don’t want to say Component is bad, and I also don’t want to repeat the already great discussions on that subject. I am here to tell you that I like mount, why I created a spin-off from scratch called mount-lite, and why you might like it.

Why mount-lite was created

As I said, I discovered mount quite early. While checking it out, I saw it has some nice options on how to and which states to start or stop. I quickly noticed though that those options do not compose well. I tried to steer the project to have somewhat better composable API, by starting an discussion in this mount issue early on, but the project had other priorities, which is perfectly fine of course.

After waiting to see what direction mount would go in, I decided it was best to develop mount-lite, if only to realize my own idea of a data driven and composable API. The library is very suitable to have a simple internal model of what to start/stop and how. Therefore, the internal start* function, simply takes the following map:

{#'state-var {:start ... :stop ...}
 ...}

It is simply a map of all the state vars that need to be started and their respective :start and :stop functions. Those functions might come directly from the state var, or from some substitution. The start* function does not care. It is the public start function that takes all kinds of options, and brings it down to the above map. This is in contrast to the original mount, where each option has its own function and logic. Mount-lite’s design makes it simple to compose the options and add new options, without disrupting the public API or most of the internals.

The options to start are specified via one or more option maps. These option maps can be generated by (combinator-like) builder functions. For instance, if one wants to start only the state #'foo and substitute its behaviour, one could call start with data directly:

(start {:only [#'foo]
        :substitute {#'foo {:start (constantly "bar")}}})

Or one could use the builder functions, for a more convenient REPL experience:

(start (only #'foo)
       (substitute #'foo (state :start "bar")))

One can also thread these builder functions together, to create an option map:

(def opts (-> (only #'foo)
              (substitute #'foo (state :start "bar"))))

(start opts)

Whatever the style you prefer, it can be done, because ultimately it is data driven.

Alright, experiment succeeded. I had a small mount like library, having a better API (in my opinion). But hey, now that I have my own library, I can add more ideas to the mix!

What makes mount-lite unique

No suspending

Right from the start, I did not implement the possibility to “suspend” and “resume” states. This has long been a feature of the original mount, but has also been removed there some days ago. The currently available start and stop options (for both mount and mount-lite) are enough.

Cleaner substitutions

Also right from the start, I had a different idea about how to handle “substitutions”, i.e. starting a state with different (mock) lifecycle functions. With mount, one would write (start-with {#'app/foo #'app.test/test-foo}), meaning that when #'app/foo is started, it should use the lifecycle functions of the state #'app.test/test-foo. The latter is also defined with defstate, meaning extra care must be taken that it is not started if not used as a substitute.

In mount-lite, substitutions are anonymous states, or at least not defined with defstate (they can be bound to a var of course). So, as we have seen already, in mount-lite one would write (start (substitute #'app/foo app.test/test-foo)), meaning it will use the value (a map) of app.test/test-foo for the lifecycle functions. This way, substitution are just that, not global states. It also means that the lifecycle map can be inline, such as (start (substitute #'app/foo {:start ...})). Simple, yet effective.

Dependency graph

Instead of building a sequence of ordered states, mount-lite builds a dependency graph of them. It does this by leveraging the tools.namespace library. By default, a state var depends on previously defined state vars in the same namespace and transitively required namespaces. This can be influenced however, for power users, by adding :dependencies to the meta data of the state var. This dependency graph provides the basis for the next three unique features.

Start and stop “up-to”

One of the start/stop options that can be supplied, is :up-to. The value of that option is a state var. It means that, when starting, all dependencies of that state are started first, and lastly the state itself is started. This way one can bring up only a part of the system, for instance when testing. Below is a figure of a simple dependency graph of four states. The arrows in the diagram mean “depends on”. In the figure, the white states have been started, after issuing the following:

(start (up-to #'b))

Mount start up to example

When supplying the :up-to option to stop, first all the dependents of the given state are stopped, and lastly the state itself is stopped. This safely brings down only the necessary part of the system, without any dependents suddenly missing a dependency. Again, the gray states in the figure below show which states have been stopped, after the following expressions:

(start)
(stop (up-to #'b))

Mount stop up to example

Cascading stop on redefinition, or not

Whenever you are hacking away in a REPL on your project, and redefine an existing defstate, the default behaviour is to stop that current state first. This will nicely clean up any resources first. The original mount does this as well. But in mount-lite, this stop is actually using the :up-to option we just discussed. We call this a “cascading” stop. Only the parts of your system that is being redefined is stopped cleanly, and the rest keeps running. Simply call start and the full system is back up in no time.

Of course, there are situations that it is not desired to automatically stop all the states up to the redefined state. Because of this, the reloading behaviour can be altered globally, by calling the on-reload function. More details on the various on-reload modes can be found in the documentation of mount-lite.

Parallel start and stop

Another great feature that is possible because of the internal dependency graph, is being able to start and stop independent states in parallel. Simply supply the number of threads one desires to use via the :parallel start/stop option, and an internal executor service will do the rest for you. This may significantly reduce the start and stop times of your application. For example, the red states in the figure below are started in parallel.

(start (parallel 2))

Mount parallel start example

If state b finishes starting before state c finishes, state a might be starting in parallel with c as well. The next figure shows which states are stopped in parallel.

(stop (parallel 2))

Mount parallel stop example

Again, if state a finishes stopping before state c finishes, state b might be stopping in parallel with c as well.

Of course, one does not have to use the parallel functionality. If the option is left out, the current thread and the simple sequential ordering is used to start and stop the states.

There you have it

In the end, mount-lite turned out not to be so “lite” after all. Yet, for me at least, these are nice features to have. I showed mount-lite to Anatoly (the author of the original mount) first. He liked the result, and thought mount and mount-lite can co-exist very well together. Thus, I open sourced it, for all of you who might enjoy it. Have questions or suggestions? I’d love to hear them, for example on twitter or #mount channel on Slack.

Permalink

Project: SendGrid for Clojure

TLDR: If you need to send emails via SendGrid from a Clojure application, check out the sendgrid library I just released, it’s pretty swell.

This is another simple library that just aims to do one thing well, but if you’re sending transactional emails from your webapp, you should be using something like this. There are a couple other Clojure libraries out there, but I checked them out and ran into a number of issues and inconveniences. (On that topic, I’m making it a habit to include a Motivations section in my projects' READMEs as I know one of my first questions when evaluating a library is to ask “OK, but how is this one different from the alternatives?”)

Sending emails is simply a matter of passing API credentials and a map representing the email to send:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(require &lsquo;[sendgrid.core :as sendgrid])</p>

<p>(def sg-config
  {:api-user &ldquo;camdez&rdquo;
   :api-key  &ldquo;d34db33f&rdquo;})</p>

<p>; Send a plain-text email
(sendgrid/send-email sg-config</p>

<pre><code>                 {:to      "email@example.com"
                  :from    "email2@example.com"
                  :subject "SendGrid Test"
                  :text    "Email body here."})
</code></pre>

<p>;; => {:message &ldquo;success&rdquo;}

There’s nothing novel here but it works smoothly.

Actually the only particularly interesting piece to implement was attachments (simply pass a sequence under the :attachments key), for which I had to learn a bit more about multipart encoding (incidentally this is also the reason I didn’t use my preferred Aleph for the HTTP client—it doesn’t yet have multipart support). The payoff to that research is that the library can handle attachments in the full variety of formats offered by the underlying library: File, InputStream, ByteArray, or String. This means you can easy attach a file on disk or generate one dynamically.

Lastly I’ll point out that there are lots of other calls in the SendGrid API, but I didn’t implement them since I didn’t need them. But adding new calls is trivial, so please feel free to either submit a PR or open an issue on the GitHub page if you need support for something else.

I hope this is helpful to someone. Please contact me with any thoughts, questions, or concerns.

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.