I started reading a book called Generative Art a few weeks ago and the first exercise, drawing a circle using trig functions, reminded me of my favorite After Dark screen saver, Rose. I have always wanted to understand how Rose worked and could see that this exercise had started to send me down the right path. A few weeks later — and with some help from Rose’s author! — and I now have a framework for building HTML5 Canvas-based generative artworks in ClojureScript.
Cassaforte is a new Clojure client for Apache Cassandra 1.2+. It is built around CQL 3 and focuses on ease of use. You will likely find that using Cassandra from Clojure has never been so easy.
ClojureWerkz projects are mostly known for our data store clients. Data processing is a sweet spot for Clojure and that’s what we primarily use it for.
Over time we’ve developed and released a number of data store clients:
and a few others.
Today we are adding another client to this family, Cassaforte.
Cassaforte was started about 11 months ago as an experiment. Back then we needed a database well suited for time series data, and Cassandra is a good choice. However, dealing with existing clients, namely Hector and Astyanax, was quite a bit of a pain.
In the end, using Cassandra’s low level Thrift client worked well and we have been improving the codebase bit by bit, while working on an event collection and analytics system.
A few months ago, DataStax released a new Java driver for Cassandra that was a lot better suited to what Cassaforte needed to power it than any other alternative.
We were able to switch Cassaforte to it in a couple of weeks. About the same time, a great fellow Max Penet released a Clojure DSL for generating CQL, Hayt.
Hayt makes working with CQL as nice as the new clojure.java.jdbc DSL:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | |
1 2 3 4 5 6 7 8 9 10 | |
1 2 3 4 5 6 7 8 | |
Integrating Hayt took a few weeks as well. From there, it took some time to write initial documentation and add some polish.
Major props to the DataStax team for releasing such a nice and focused Java driver and Max for making querying Cassandra from Clojure such a good experience.
Cassaforte is young but already offers all the key features you’d expect from such a client:
Needless to say, as more features are added to Cassandra and the DataStax core Java driver, Cassaforte will soon to follow their lead.
Cassaforte documentation is far from finished but it follows the “if it is not documented, it is not part of ClojureWerkz” rule. To get started, take a look at the Getting Started guide.
The rest of the guides are being worked on.
API reference is also available.
Cassaforte was started by Michael but ended up being a brain child of Alex.
We would like to thank Max Penet for providing a lot useful feedback, sharing Hayt with us and helping us define what we want Cassaforte to be and feel like.
New releases and updates are announced on Twitter. Cassaforte also has a mailing list, feel free to ask questions and report issues there.
Cassaforte is part of the group of libraries known as ClojureWerkz, together with
and several others. If you like Cassaforte, you may also like our other projects.
Let us know what you think on Twitter or on the Clojure mailing list.
Michael on behalf of the ClojureWerkz Team.
Clojure Gazette 1.42
|
I have finally managed to understand one of the most unusual databases of today, Datomic, and would like to share it with you. Thanks to Stuart Halloway and his workshop! Why? Why?!? As we shall see shortly, Datomic is very different from the traditional RDBMS databases as well as the various NoSQL databases. It even […]
This post will explain how our team was able to greatly reduce the friction in writing documentation guides by standardizing on a toolchain that other projects can use.
Before there was ClojureWerkz, myself and Alex were fairly active in the open source community. I maintain several RabbitMQ clients and needed a new documentation site for one of them.
So I put together a doc site based on Jekyll and Alex helped me with styles. This toolchain has evolved a bit and when we were looking at writing documentation for Monger and Langohr in late 2011, we decided to just base them on what we already had and knew: that Jekyll site amqp gem was using.
ClojureWerkz grew and we kept reusing the same thing over and over again. By June 2013, ClojureWerkz has accumulated about a dozen of projects that have substantial documentation guides. Writing, editing and maintaining these docs took time and in the process, we’ve found ourselves repeating a lot of things between sites.
It’s easy to notice a lot of similarities between, say, Elastisch docs and Langohr docs. We also try to take good ideas from doc sites developed later and feed them back into older sites.
So we needed a standard toolchain that all these documentation projects could use. In particular, we wanted it to give us a very fast way of writing the first couple of guides before we would worry about finer details and editing.
Docslate is a relatively unknown project. We don’t write about it on this blog. It does not get updates very often. It does, however, play a major role in ClojureWerkz: powering our documentation guides and making writing them a more pleasant and straightforward process.
Docslate is a Jekyll site with predefined structure and integrated Twitter Bootstrap. Once you have it going, with just a few edits you are ready for the work that really matters: writing helpful documentation guides.
Docslate lets you get to writing quickly without spending too much time on issues such as
and so on.
It’s dead easy to put together a pretty good documentation site with Docslate and then work on copy and styling once your first version is ready to go live.
Since Docslate uses Jekyll, you’ll need Ruby (any version, although we typically use 2.0 or 1.9.3 these days) and RubyGems installed.
Then install Bundler with
gem install bundler
Clone Doclsate Git repository with
git clone git://github.com/clojurewerkz/docslate.git my-project-guides
Change to your local clone, remove .git directory and re-init the repo
(Docslate is simply a starting point, your doc site should really be in a new
repo):
cd my-project-guides
rm -rf ./.git
git init
git add .
git commit -m "Initial commit: Docslate"
Then install dependencies with Bundler:
bundle install --binstubs
and you are ready to run a development server with
./bin/jekyll serve --watch
then navigate to localhost:4000.
The site you’ll see is very similar in structure and styles to Elastisch docs, Monger docs, and so on.
Docslate assumes your documentation guides have an index page, a ToC page and one or more guides.
The index page is in HTML and can be found in index.html in the repository root.
The ToC page is in Markdown and can be found with individual guides under articles.
Docslate contains 3 pages for you to start with:
guides.md
getting_started.md
community.md
guides.md is the ToC page. getting_started.md contains the Getting Started guide.
Finally, community.md gives you a page to list information about the project,
its mailing list, IRC channel, Twitter account, and other things you need to
make your open source project really awesome.
Adding new guides is as straightforward as adding a Markdown file to the articles directory
with the header (title, etc) Jekyll expects and linking to it from the ToC page.
For example, to add a guide on extending your library, create a new file at articles/extensions.md,
add Jekyll header to it, make sure development server is started and navigate to
localhost:4000/articles/extensions.html.
Layout files can be found under _layouts. There are two layouts available:
Article layout is used to display individual articles (guides), default layout is used for everything else.
Styleshsets and JavaScript files can be found under assets/stylesheets and
assets/javascripts, respectively.
Use
./bin/jekyll build
to generate the site for deployment. Site root then will be under _site. This directory
is what you will deploy. It only contains static assets so deployment usually is
as straightforward as rsync-ing files to your server and serving them with Nginx, Apache
or any other Web server of your choice.
Docslate is a starting point. It is meant to be customized for your project’s needs, styling and so on. Feel free to do it, add JavaScript libraries you need, upgrade and customize Twitter Bootstrap, and so on.
Docslate was born out of a dozen documentation sites we have for ClojureWerkz projects. It’s a battle tested, easy to use toolchain that provides you a good starting point and is completely open sourced under a reasonable license.
Now you don’t have an excuse to not document your open source project well!
Michael on behalf of the ClojureWerkz Team.
Constructing Expressions
Gsoc finally starts monday. And for the start I want to have a good idea of how the user interface of expresso should look like, for this I want to discuss the best way of aktually constructing expresso expressions :).
Because core.logic currently doesn't play nicely with custom types and algebraic expressions fit nicely with normal clojure s-expressions I plan to represent algebraic expressions just as that. However, that doesn't mean it is straightforward to construct them - expresso has to know the full qualified function symbol to differentiate between - for example - clojure.core/* and the core.matrix * which behave differently. Also other mathematical properties should be stored about the functions - I plan to use metadata for it.
Hopefully as seamless as possible.
One approach would be a macro - create-expression. It would construct the enhanced form of the s-expression from the s-expression it gets. This, however has some drawbacks. First the user either needs to use ~ to aktually get the data in the expressions. Or you have to have map as argument which would map from symbols to values. Further than that, because it is a macro, it can't be used in higher order functions.
(let [matrix [[1 0 1]
[0 1 0]
[1 0 1]]]
(simplify (construct-expression (* 5 (determinant ~matrix))))
;or
(simplify (construct-expression (* 5 (determinant x)) {'x matrix})))
The other approach would be to have construction functions in an own namespace, with concise names, just like core.matrix did with the operators namespace.
(ns test
(:use [expresso.construction])
(:refer-clojure :exclude [* - + == /]))
(let [matrix [[1 0 1]
[0 1 0]
[1 0 1]]]
(simplify (* 5 (determinant matrix))))
Mike Anderson also posted some interesting usage examples (with a possible syntax) on the clojure mailing list:
; 1. Solving simultaneous equations expressed in arrays:
(solve [x y] (== [x y] (* [[3 0] [0 10]] [y 1])))
=> [30 10]
;2a. Derivatives of array expressions:
(derivative [x] (* [x 2] [3 (* x x)]))
=> [3 (* 4 x)]
;2b. Derivatives that are themselves vectors / arrays:
(derivative [[x y]] (* [x 2] [3 (* x x)]))
=> [[3 (* 4 x)] 0]
;3. Simplification of computations on arrays:
(simplify (trace [[x y] [z (- 5 x)]]))
=> 5
;4. Simplification of array-level expressions
(simplify (* 10 A (some-identity-matrix) (inverse A) B))
=> (* 10 B)
What would be your favorite way of using expresso?
Considering the close relation between expresso and core.matrix it is very likely that expresso (especially the optimizer) will be used on core.matrix expressions. Mike's examples also point in this direction. What about making expresso an implementation of core.matrix ? It would not calculate the results of matrix operations, but create the expressions under the hood. That way one could just switch the implementation of core.matrix to make the matrix expression available to expresso.
For the core.matrix folks: Do you think that is possible? There are certainly a few caveats in it but it seems to be the perfect way of using expresso in a core.matrix context.
Early this year I wrote a simple to-do planning web app in Clojurescript mostly for use on my smartphone. As a learning experiment with client/server side Javascript and Meteor I rewrote it to support real time collaboration: our-foc.us
This was an interesting learning experience, and I must say that Meteor if both easy to use and provides impressive functionality “out of the box.”
We're happy to announce today transparent integrated support for memcached in Datomic Pro Edition.
One of the nice things about the Datomic architecture is that the index segments kept in storage are immutable. That enables them to be cached extensively. Currently that caching happens inside the peers, which keep segments they have needed thus far in the application process heap.
While this is great for process-local working sets, there is only so much a single machine can cache. So, we've added support for an optional second tier of distributed, shared cache, leveraging a memcached cluster. This tier of cache can be as large as you wish, and is shared between all the peers.
The entire use of memcached is automatic and integrated - just provide the endpoints of your memcached cluster in configuration. The peer protocols will automatically both look in it, and populate it on cache misses. Being based upon immutability, there are no cache coherence problems nor expiration policy woes.
The architecture incorporating memcached looks like this:
During QCon New York 2013, Rich Hickey gave a talk on functional databases. Hickey is well known for creating the Clojure programming language and is currently developing Datomic, which is a functional database. During his talk, Hickey argued that the useful properties of functional languages: data as values and pure functions, are just as useful in the context of databases. By Zef Hemel
Jim Duey surveys several abstraction techniques that can help in writing reusable code in Clojure. By Jim Duey
When:
Tuesday, June 25, 2013 from 7:00 PM to 9:30 PM (BST)
Where:
ThoughtWorks London Office
173 High Holborn
WC1V London
United Kingdom
Hosted By:
London Clojurians
Register for this event now at :
http://london-clojure-dojo-tw-2013-06-tw-rss.eventbrite.com
Event Details:
The goal of the session is to help people learn to start working with Clojure through practical exercises, but we hope that more experienced developers will also come along to help form a bit of a London Clojure community. The dojo is a great place for new and experienced clojure coders to learn more. If you want to know how to run your own dojo or get an idea of what dojos are like you can read more here.
We hope to break up into groups for the dojo. So if you have a laptop with a working clojure environment please bring it along.
We’ll be discussing the meetup on the london-clojurians mailing list
Clojure is a JVM language that has syntactically similarities to Lisp, full integration with Java and its libraries and focuses on providing a solution to the issue of single machine concurrency.
Its small core makes it surprisingly easy for Java developers to pick up and it provides a powerful set of concurrency strategies and data structures designed to make immutable data easy to work with. If you went to Rich Hickey’s LJC talk about creating Clojure you’ll already know this, if not it’s well worth watching the Rich Hickey “Clojure for Java Programmers” video or Stuart Halloway “Radical Simplicity” video .
Problem #157 in 4Clojure.com is listed as a very simple problem. Despite having used Clojure’s map function a lot, I saw the solution to this problem recursively using loop and recur, by walking through each element of the function and determining its count. The following solution came from http://blog.qinjian.me/2012/12/21/4clojure_solutions/ . Look for #157 in the midst of quite a few solutions.
(fn __[coll]
(loop [x (- (count coll) 1)
l []]
(if (>= x 0)
(recur (- x 1)
(conj l (list (nth coll x) x)))
(reverse l))))
This solution makes more sense to me than this solution.
#(map list % (range))
It just doesn’t look as obvious. But the prevailing wisdom is to use Clojure’s sequence functions to handle problems like this.
The bugfix release of Incater was just pushed to Clojars.
It fixes 2 bugs – one nasty bug when functions called from transform-with were working on underlying data, modifying them, and 2nd – when 2-argument version solve thrown an error.
From the webpage:
Clojure Cookbook is coming
And we need your help.
We want this O’Reilly cookbook to be a comprehensive resource containing the collective wisdom of Clojurists from every domain. That’s why we want to write it together, as a community.
Share some code. Explain it. Be a part of Clojure history.
Cool!
GitHub clojure-cookbook.
Today, I want to provide a guided tour through some of the many libraries
available from the Clojure team that don't come distributed with Clojure. Consider
them Clojures standard library. Some came from old clojure.contrib libs, others
are brand-new, but all are great.
I spend a lot of time thinking about how to make Clojure more popular. After all, if there's more code written in Clojure, there's more need for Clojure developers, which means more opportunities to use my favorite language of late in production. One potential barrier to adoption – if only a psychological one – is that Clojure, as a Lisp-derived language, tends to eschew the batteries-included philosophy of other, more mainstream languages. Java, PHP, Python and Ruby come to mind as languages I've used that come with extensive standard libraries by default, and I'm sure C# and Perl and all the other ones I've never really dealt with.
Clojure's default installation takes a different tack from these. Besides core, clojure includes
only the following menagerie of utilities and helpers. Some of these are very useful indeed:
diff, which might come in handy. Used as a namespace later though.seq-zip, xml-zip and vector-zip seem to be the most useful bits at-a-glance.Other libraries included with clojure may be mostly only useful to the clojure.core team themselves,
or people writing tools for clojure:
browse-url, which accepts a url and opens a browser.However, beyond the libraries distributed with clojure, there is a whole world of
libraries maintained by the clojure team, mirroring many of the “batteries” from
other languages. And thanks to Leiningen, these almost-first-class-members of the
clojure experience are just an entry in :dependencies away.
First on our tour, we'll check out 3 libraries available in the clojure.data namespace.
Most projects won't need all of these, but a good number will probably have occasion to use
at least one.
If a library doesn't come with an easy way to deal with csv files, it probably
doesn't care about you and your petty problems. But Clojure loves you, baby,
and clojure.data.csv is how you know it. (Actually, it's Jonas Enlund who loves you, and who wrote this lib)
(require '[clojure.data.csv :as csv]
'[clojure.java.io :as io])
(with-open [in-file (io/reader "in-file.csv")]
(doall
(csv/read-csv in-file)))
(with-open [out-file (io/writer "out-file.csv")]
(csv/write-csv out-file
[["abc" "def"]
["ghi" "jkl"]]))
I mentioned above Rich Hickey's effort, clojure.xml, but here's it's big brother.
clojure.data.xml, maintained and likely written by Ryan Senior.
data.xml's parsing utilities return records of the following description
#clojure.data.xml.Element{:tag :foo,
:attrs {},
:content ...}
You can deal with these parsed records just like maps, as usual. Here's an excerpt from some code I recently wrote to extract items from RSS feeds:
(defmulti get-items :tag)
(defmethod get-items :rss [doc]
(->>
doc
:content
first
:content
(filter-tag :item)
(map :content)))
(defmethod get-items :feed [doc]
(->>
doc
:content
(filter-tag :entry)
(map :content)))
(defmethod get-items :default [_] [])
You can construct XML documents by creating Elements yourself (using the
(element [tag attrs content]) record definition), or you can use
a Hiccup-esque syntax with sexp-as-element:
(sexp-as-element
[:foo {:foo-attr "foo value"}
[:bar {:bar-attr "bar value"}
[:baz {} "The baz value"]]]))
Good stuff.
Like other JSON libraries for languages with literal hash maps,
Stuart Sierra's clojure.data.json
is best thought of as a conversion layer between hash maps and
JSON strings, although this library also provides stream-based i/o.
(json/read-str "{\"a\":1,\"b\":2}")
;;=> {"a" 1, "b" 2}
(json/write-str {:a 1 :b 2})
;;=> "{\"a\":1,\"b\":2}"
What more do you need to know? If you need to deal with json in clojure,
you want clojure.data.json. There's lots more functionality, so be sure
to read the docs.
Adds a few syntactic sugars to macro definitions in clojure:
macrolets, which are locally-scoped macros, and “symbol macros”.
(defsymbolmacro sym (+ a b))
(with-symbol-macros
(+ 1 sym))
;; expands to (+ 1 (+ a b))
Some nice little utils. If your projects don't already have
half-baked implementations of these, you should include clojure.tools.macro before
you do.
clojure.tools.logging, by Alex Taggart, “wraps”, i.e. implements, all of the big Java logging libraries at once.
Write your code once, and pick your implementation later.
The logging functions are exactly what you'd expect:
(defn divide [x y]
(try
(info "dividing" x "by" y)
(/ x y)
(catch Exception ex
(error ex "There was an error in calculation"))))
If you're not using Timbre, you should
be using clojure.tools.logging.
I must admit, when I tried to use clojure.core.cache
(written and maintained by Michael Fogus),
I got a bit confused for a while before I realized that I had to be storing the output
of the cache. Really, I don't mind if the caching library I use is a bit stateful!
But, I'm still happy about core.cache, because it paves the way for…
This library (also Fogus) is great. You use it to wrap a function, and it pops an atom right in the metadata of the function. Your function calls are instantly memoized.
You can pick and choose your caching strategy as per core.cache, too:
(require '[clojure.core.memoize :refer (memo-lu)])
(def id (memo-lu #(do (Thread/sleep 5000) (identity %)) 3))
(id 42)
; ... waits 5 seconds
;=> 42
(id 42)
; instantly
;=> 42
David Nolen's core.match is an
alpha-quality attempt to bring pattern-matching
to clojure. I prefer writing code to words, so let me explain it to you
in the language of my people: Fizzbuzz!
(use '[clojure.core.match :only (match)])
(defn fizzbuzz
"
Print each number from 1 to 100, but for multiples of 3 print 'Fizz'
instead, and for multiples of 5 print 'Buzz'. For multiples of both
print 'FizzBuzz'
"
[]
(doseq [n (range 1 101)]
(println
(match [(mod n 3) (mod n 5)]
[0 0] "FizzBuzz"
[0 _] "Fizz"
[_ 0] "Buzz"
:else n))))
This library has a nice little writeup available for it, so I won't belabour it.
I personally can't wait for this to get out of alpha, and hopefully get absorbed into clojure.core
The core.unify library, another Michael Fogus joint, is
a library that brings unification)
to clojure.
It's not a concept I was particularly familiar with before I started writing this, but I actually found the best examples in the comments section of the main library file from which the following examples are alternately inspired and outright stolen.
There are three primary functions exposed by core.unify.
(unify [expr1 expr2]) attempts to define what symbols (denoted by ?) it can given an expression.
(u/unify {:first '?first
:last '?last
:genre :giallo}
{:first "Dario"
:last "Argento"
:genre :giallo})
;; => {?first "Dario" ?last "Argento"}
Here's most of the table from that wikipedia article, translated into core.unify
(u/unify 'a 'a) ;; => {}
(u/unify 'a 'b) ;; => nil
(u/unify '?x '?x) ;; => {}
(u/unify 'a '?x) ;; => {?x a}
(u/unify '?x 'y) ;; => {?x y}
(u/unify '(f a ?x) '(f a b)) ;; => {?x b}
(u/unify '(f a) '(g a)) ;; => nil
(u/unify '(f ?x) '(f ?y)) ;; => {?x ?y}
(u/unify '(f ?x) '(g ?y)) ;; => nil
(u/unify '(f (g ?x)) '(f ?y)) ;; => {?y (g ?x)}
(u/unify '(f (g ?x) ?x) '(f ?y a)) ;; => {?y (g ?x) ?x a}
(u/unify '?x '(f ?x)) ;; (throws an exception)
(subst [expr bindings]) is the inverse of unify, in a way. Given an expression and
a map of bindings, it returns the expression with the bindings substituted. Symbols
can be nested:
(u/subst
'[1 2 ?x ?y]
'{?x [3 4 ?y 6] ?y 1})
;; => [1 2 [3 4 1 6] 1]
(unifier [expr1 expr2]) performs a unification and then a substitution, resulting
in the most complete version of the provided expressions that can be formed.
(defn my-unifier [expr1 expr2]
(u/subst expr1
(u/unify expr1 expr2)))
(my-unifier '(f ?y) '(f (g ?x))) ;; => (f (g ?x))
(u/unifier '(f ?y) '(f (g ?x))) ;; => (f (g ?x))
The primary motivation for writing unifiers seems to be the creation of type inference systems, but I'm sure there's other cool stuff you can find to do with your newfound learning.
This library, also written/maintained by David Nolen, clojure.core.logic
is an implementation of both
miniKanren
and cKanren (pdf)
in Clojure. You could think of it like core.unify, but fuller-featured.
It lets you write declarative statements, then run them against the logic engine to get a list of results satisfying the provided conditions.
Here's a sample from the primer:
(run* [q]
(membero q [1 2 3])
(membero q [3 4 5]))
;; => (3)
Of course there are better ways to expression set intersection, but there are tons
of applications that clojure.core.logic can help express more concisely.
There is pretty good documentation available on the project's wiki, but I prefer The Magical Island of Kanren as an intro (even if the syntax is mildly outdated).
Also, here's a very recent blog post by the author that I found really helpful.
Statically-typed functional programs have a nice habit of just working, once
you get your types all sorted out. clojure.core.typed,
written by Ambrose Bonnaire-Sergeant,
lets you do the next best thing in Clojure, by adding type annotations to
your functions.
Here's how it works. First, write a function:
(ns example.typed
(:require [clojure.core.typed :as t]))
(t/ann add [Number Number -> String])
(defn add [a b]
(+ a b))
Then, call check-ns from somewhere within that namespace (I keep mine in a comment at the bottom of the file
and run it with fireplace.vim):
(t/check-ns)
And, since I annotated that wrong, we'll see some errors:
Type Error (example.typed:7:3) Return type of static
method clojure.lang.Numbers/add is java.lang.Number, expected java.lang.String.
in: (clojure.lang.Numbers/add a b)
Type Error (example.typed:6) Expected type: String
Actual: Number
in: (clojure.lang.Numbers/add a b)
Type Error (example.typed:6:1) Expected type: (Fn [Number Number -> String])
Actual: (Fn [Number Number -> Number])
in: (def add (fn* #))
You probably won't annotate everything, but it's nice to have core.typed around when
you forsee there might be a typing problem, and it's good to have your code checked
anyhow.
I slacked off and left this out last night, but somebody called me out so now I'm back.
Fogus's clojure.core.contracts is for people
who want their functions to be even more uptight than can be had with static typing. It's basically
a replacement for starting your function with a bunch of assert calls. Here's the example from
the home page:
(use 'clojure.core.contracts)
(def secure-doubler
(with-constraints
(fn [n] (* 2 n))
(contract doubler
"ensures doubling"
[x] [number? => (= (* 2 x) %)]
[x y] [(every? number? [x y])
=>
(= (* 2 (+ x y)) %)])))
(secure-doubler 10)
;=> 20
As you can see, you can specify pre- and post- conditions, and even extend the function itself. Pretty cool.
That's just a quick tour of some of the interesting libraries I found, but it's far from exhaustive. There are a ton more, though, so be sure to check out the Clojure github repo to see what else exists.
As we approach a 1.0.0 of Immutant, we've also been focusing on getting our Leiningen plugin ready for a 1.0.0 release as well.
Today, we're as pleased as a punch bug player to announce the first 1.0.0 beta release of the plugin: 1.0.0.beta1.
For 1.0.0.beta1, we've made quite a few changes to the plugin, some of which are not backwards compatible. We'll talk about all of the changes, starting with the breaking ones.
When we wrote the initial version of the plugin, we used
.lein/immutant as base directory for storing Immutant installs and
other files. But as the plugin has grown, we've realized that's not
the best place for storing those files, since they aren't Leiningen
specific. Therefore, we've moved the base directory to ~/.immutant.
In addition, we've reorganized the layout of ~/.immutant/releases to
be a bit cleaner.
Since the new plugin will be looking in ~/.immutant, it won't see
the prior installs in ~/.lein/immutant. You'll need to re-install
the Immutant versions you need, and can safely delete
~/.lein/immutant.
Pre-1.0.0.beta1, the plugin installed the latest incremental build
when you executed lein immutant install. As we approach 1.0.0 of
Immutant, we've changed the plugin to install the latest stable
release instead.
So, to install the latest stable release:
lein immutant install
If you want to install the latest incremental build:
lein immutant install LATEST
You can still specify any stable release version or incremental build number to get an exact version.
A couple of releases ago, we added the lein immutant list command
that lists currently deployed applications. In 1.0.0.beta1, this
command has been extended with a --installs flag (or -i) to list
all of the installed Immutant versions instead of deployments:
$ lein immutant list --installs
The following versions of Immutant are installed to /home/tobias/.immutant
(* is currently active via /home/tobias/.immutant/current):
0.10.0 (type: full)
0.10.0 (type: slim)
0.6.0 (type: full)
0.9.0 (type: slim)
0.9.0 (type: full)
* 1.0.0.beta1 (type: slim)
1.x.incremental.879 (type: slim)
1.x.incremental.882 (type: slim)
You'll also now see this list after every lein immutant install invocation.
In addition to moving the base directory to ~/.immutant, we've added
the option to override its location, either per-project or globally.
You can override it on a per-project basis by setting :lein-immutant
{:base-dir "/path"} in your project.clj
You can override it globally two ways:
:lein-immutant {:base-dir "/path"} in your user profile
in .lein/profiles.clj
$LEIN_IMMUTANT_BASE_DIR
Setting the base directory in project.clj will override the setting
in .lein/profiles.clj. Using the environment variable will override
both.
If you're already using lein-immutant you probably already know how
to do this, but just in case - to install it, add it to the :plugins
list in your :user profile in ~/.lein/profiles.clj:
{:user {:plugins [[lein-immutant "1.0.0.beta1"]]}}
We realize this release has quite a few changes, and we may not have all of the kinks worked out. If you have any questions, issues, or other feedback, you can always find us on #immutant on freenode or you can file an issue on lein-immutant.
We are almost as excited to announce our very first beta release of Immutant as this guy is to see a train. And like that train, we expect our beta cycle to be short; our plan is to have 1.0.0 out by the end of the summer, after which we'll begin to incorporate new features from JBoss Wildfly for the next release. As always, view our road map here.
Immutant is an application server for Clojure. It's an integrated platform built on JBoss AS7 that aims to reduce the inherent incidental complexity in real world applications.
We haven't changed a whole lot since 0.10.0. We wanted to spruce up our docs and make sure we didn't have any glaring stability issues. We made a couple of minor enhancements to the jobs and caching namespaces, but nothing that isn't backwards compatible.
Cluster configuration should be simpler since we're now taking advantage of HornetQ's ability to use JGroups. We added a Production Setup chapter to our manual which we hope to evolve as more Immutants are put in production. Speaking of that, check out Damion's excellent writeup of his experiences clustering Immutant and TorqueBox on Amazon's EC2.
See the complete list of changes below.
The simplest way to install or upgrade to 1.0.0.beta1 is via our Leiningen plugin (though we recommend you update your plugin version first):
$ lein immutant install 1.0.0.beta1
See our install page for more details. Once you have it installed, take a look at our tutorials.
If you have any questions, issues, or other feedback about Immutant, you can always find us on #immutant on freenode or our mailing lists.
Messaging has multiple uses in system design. It can aid communication between application components, enable asynchronous processing, and form the basis for a distributed architecture.
My last project involved receiving financial data in multiple formats, transforming it into a single canonical form, analyzing it for patterns, and forwarding findings to a web app. This mapped naturally to a pipeline design.
By implementing each step as its own subsystem and having them all communicate through a messaging layer, a number of benefits were achieved. New formats were easy to support. Scaling was a matter of adding more processes and machines. Any application which needed transformed data, such as a dashboard, could get it simply by subscribing to the right component.
This messaging was implemented using ZeroMQ.
In this post, I’ll give you a quick introduction to ZeroMQ through Clojure and help you get up and running with it. By the end, I hope to show you how simple it can be to add basic messaging capabilities to your own projects.
Tutorial Overview
What we’ll cover
What we won’t cover:
ZeroMQ Overview
ZeroMQ is an embeddable networking library. It gives you an enhanced socket which encapsulates and handles much of the complexity that typically comes with connecting programs. Specifically, these sockets:
One of ZeroMQ’s goal is to make building distributed software akin to Lego. In that vein, connecting two application components is simply a matter of instantiating sockets on both ends with a configuration of your choice, and letting ZeroMQ handle the rest. Much like Clojure, creating a ZeroMQ-based topology means starting with basic communication patterns, and then composing them into higher-level systems suited to your goal.
ZeroMQ Sockets vs TCP Sockets
ZeroMQ sockets exhibit a few differences from typical TCP sockets:
In addition, ZeroMQ messages can go across multiple transports (inproc, ipc, tcp, pgm, epgm), bound endpoints accept connections automatically, and rather than working directly with connections (which are encapsulated under the socket), you work with the socket instead.
ZeroMQ vs Other Messaging Tools
From a user’s perspective, the biggest difference between ZeroMQ and other popular messaging solutions (such as RabbitMQ) is the lack of an external component to provide easy routing, persistence, and load-balancing; features which RabbitMQ’s broker architecture lets you get quite easily.
In contrast, ZeroMQ is designed for high-throughput/low-latency applications. It works at a lower-level as a socket library and is designed to be embedded in your application. Its brokerless nature is also the origin of ‘Zero’ in the library’s name. Advanced messaging schemes are then built by combining pieces of the framework.
Setup
ZeroMQ is written in C++ and offers bindings for a number of languages, including Java through JNI. In the past, I used the Java bindings (JZMQ) but there is now a native Java implementation called JeroMQ. For simplicity, that’s what we’ll use here today. The examples we’ll cover are simple enough that we’ll use Java interop rather than wrap the calls in Clojure functions.
Start by creating a new Clojure project called clj-zmq. I use Leiningen:
lein new clj-zmq
Next, open project.clj in your project’s root folder and add these to your dependencies:
[com.rmoquin.bundle/jeromq "0.2.0"] [cheshire "5.2.0"]
Cheshire is a JSON encoding library which we’ll use in our examples.
Set main to clj-zmq.core:
:main clj-zmq.core
This is what my project.clj file looks like:
(defproject dist "0.1.0-SNAPSHOT"
:description "Messaging With Clojure and ZeroMQ"
:url "http://patternhatch.com"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.5.1"]
[com.rmoquin.bundle/jeromq "0.2.0"]
[cheshire "5.2.0"]]
:main clj-zmq.core)
Save the file, switch to your project’s root directory, and run:
lein deps
That should set your dependencies.
Open core.clj and set up the namespace:
(ns clj-zmq.core (:import [org.jeromq ZMQ]) (:require (cheshire [core :as c])))
Contexts
The first thing we need to do is create a context, which is what we’ll use to create our sockets. A context is a container for all our sockets. In general, you should create one context per process. Creating multiple contexts in a single process is akin to creating separate ZeroMQ instances.
Continuing in core.clj, let’s create a context:
(def ctx (ZMQ/context 1))
The argument in the context creation call above is the number of threads to use. The ZeroMQ FAQ recommends one thread per gigabyte of data in or out per second. Now that we have our context, we can go over sockets before moving on to our examples.
Sockets
When using ZeroMQ, your application communicates by dealing with the socket, rather than the connection the socket handles. The socket, depending on its type, encapsulates policies for inward and outward routing, queuing, and what other types of sockets it can connect to. These relationships form messaging patterns, which give rise to ZeroMQ’s power and ability to layer additional patterns. The three basic socket pairings are:
To connect two sockets, you perform a bind on one and a connect on the other. In general, the socket with the known network address (such as the server) should bind, and the client(s) with unknown addresses should connect. Let’s see this in action with our first core ZeroMQ pattern.
The Request-Reply Pattern
For the request-reply pattern, we’ll create a simple echo server for our client to connect to. Let’s take a look at the code and go over the salient lines:
(defn echo-server
[]
(let [s (.socket ctx ZMQ/REP)]
(.bind s "tcp://127.0.01:5555")
(loop [msg (.recv s)]
(.send s msg)
(recur (.recv s)))))
Here’s what’s happening:
All in all, pretty straightforward.
One thing to note: ZeroMQ doesn’t know anything about the data we send except its size in bytes. That means we’re responsible for formatting it safely.
In our examples, we’ll be sending strings back and forth. JeroMQ has an overloaded send command which, when called only with a single argument, assumes the data is a string and formats it accordingly for us. For other types of data, we would need to take additional care and perhaps use specialized libraries such as protocol buffers.
Let’s move on to our client:
(defn echo
[msg]
(let [s (.socket ctx ZMQ/REQ)]
(.connect s "tcp://127.0.01:5555")
(.send s msg)
(println "Server replied:" (String. (.recv s)))
(.close s)))
This code looks similar to our server with three differences:
Let’s give it a try. Start your REPL (you should be in the clj-zmq.core namespace). In our examples, rather than starting multiple REPLs, we’ll run certain servers in separate threads. Start the echo server with the following command:
(future-call echo-server)
And now send a message to it using the client:
(echo "Hello!")
You should get back the following response:
Server replied: Hello!
This is the request-reply pattern, probably the simplest way to use ZeroMQ. The client issues a send command and then a receive command. Any other action such as sending multiple messages in a row will result in a -1 return code.
The Publish-Subscribe Pattern
The second pattern we’ll look at is used for one-way data distribution. For our example, we’ll create a server which publishes random stock trades and a client which subscribes to it. Here’s the server:
(defn market-data-publisher
[]
(let [s (.socket ctx ZMQ/PUB)
market-data-event (fn []
{:symbol (rand-nth ["CAT" "UTX"])
:size (rand-int 1000)
:price (format "%.2f" (rand 50.0))})]
(.bind s "tcp://127.0.01:6666")
(while :true
(.send s (c/generate-string (market-data-event))))))
By now, the code probably looks familiar to you. Once again, we use the context to create our PUB socket. We then bind it to port 6666 and begin generating market data events. c/generate-string is a Cheshire command to turn a Clojure map into a JSON string. When no clients are connected, the server silently drops its messages.
Here’s the client:
(defn get-market-data
[num-events]
(let [s (.socket ctx ZMQ/SUB)]
(.subscribe s "")
(.connect s "tcp://127.0.01:6666")
(dotimes [_ num-events]
(println (c/parse-string (String. (.recv s)))))
(.close s)))
Our client connects to the publisher, processes a certain number of market events, and ends. Notice that in addition to creating the SUB socket and connecting it, we issue a subscribe command on Line 4. In this case, issuing a subscribe command with an empty string means we wish to receive everything from the publisher (your client can specify filters on what to receive). Not issuing this command is a common oversight. If you use this pattern and find yourself not receiving anything, check for this.
The c/parse-string command on line 7 converts a JSON string into a Clojure map.
Reload your REPL and give it a try. Start the publisher in a separate thread:
(future-call market-data-publisher)
Now listen to 100 market data events:
(get-market-data 100)
You should get output similar to the following:
{symbol UTX, size 303, price 18.42}
{symbol CAT, size 492, price 41.02}
{symbol CAT, size 971, price 47.48}
.
.
.
Pub-Sub is a one-way, asynchronous line from the server to the client. An error results if either the server tries to receive or the client tries to send.
One issue this example doesn’t address is slow joiner syndrome. When your client initiates a connection, it takes time to step through the connection routine. This may take milliseconds, but during that time, the publisher may have already sent a few hundred or thousand messages which will be missed by the client. This may not matter if the publisher sends data endlessly and the client doesn’t care what transpired before. But if you want to ensure the client is well and ready before the publisher begins sending, then there’s an advanced pattern in the documentation for reliable pub-sub.
The Pipeline Pattern
Our final example covers the most involved of the three core patterns. The pipeline pattern is suited for task distribution and collection. For our example, we’ll set up a dispatcher which generates tasks and sends them to workers in a round-robin fashion. These workers will be connected upstream to the dispatcher and downstream to a collector. The collector will receive output from each worker as they complete.
Here’s the code for the dispatcher:
(defn dispatcher
[jobs]
(let [s (.socket ctx ZMQ/PUSH)]
(.bind s "tcp://127.0.01:7777")
(Thread/sleep 1000)
(dotimes [n jobs]
(.send s (str n)))
(.close s)))
Similar to the previous examples, we create a PUSH socket and bind it to port 7777. But we add an additional step where we wait for one second before dispatching. Why?
Originally, I had the dispatcher prompt the user to hit enter when ready.The reason for this is because we want to ensure all the workers are connected before beginning. This way, the first worker to connect won’t be inundated with an outsized set of tasks. Nevertheless, workers are still free to come and go as they please and the dispatcher will automatically adjust the routing accordingly.
But because certain versions of nREPL have a bug where the read-line command causes an I/O exception in the REPL, I chose to go with this arbitrary timer instead for this example to avoid confusion. For a production scenario, the ZeroMQ guide covers more sophisticated means to signal when dispatch is ready.
Here’s the worker:
(defn worker
[]
(let [rcv (.socket ctx ZMQ/PULL)
snd (.socket ctx ZMQ/PUSH)
id (str (gensym "w"))]
(.connect rcv "tcp://127.0.01:7777")
(.connect snd "tcp://127.0.01:8888")
(while :true
(let [job-id (String. (.recv rcv))
proc-time (rand-int 100)]
(Thread/sleep proc-time)
(.send snd (c/generate-string {:worker-id id
:job-id job-id
:processing-time proc-time}))))))
The worker has two sockets, one pulling from the dispatcher and another pushing to the collector (on a different port). Each worker has a unique ID and simulates performing a task for a random duration. It then forwards information about itself, the job, and how long it took to the collector.
Here’s the code for the collector:
(defn collector
[]
(let [s (.socket ctx ZMQ/PULL)]
(.bind s "tcp://127.0.01:8888")
(while :true
(->> (.recv s)
(String.)
(c/parse-string)
(println "Job completed:")))))
The collector offers an endpoint where it pulls completed tasks from all the workers in a fair-queuing fashion. Let’s give it a try. Reload your REPL, and start your collector:
(future-call collector)
Then start three workers:
(future-call worker) (future-call worker) (future-call worker)
Finally, dispatch 100 jobs:
(dispatcher 100)
You should see output similar to the following:
Job completed: {worker-id w2706, job-id 69, processing-time 53}
Job completed: {worker-id w2712, job-id 79, processing-time 71}
Job completed: {worker-id w2709, job-id 71, processing-time 96}
.
.
.
As you can see, all three workers are processing tasks in parallel and completing them in variable time.
Here’s a complete listing for core.clj:
(ns clj-zmq.core
(:import [org.jeromq ZMQ])
(:require (cheshire [core :as c])))
(def ctx (ZMQ/context 1))
;
; The Request-Reply Pattern
;
; Server
(defn echo-server
[]
(let [s (.socket ctx ZMQ/REP)]
(.bind s "tcp://127.0.01:5555")
(loop [msg (.recv s)]
(.send s msg)
(recur (.recv s)))))
; Client
(defn echo
[msg]
(let [s (.socket ctx ZMQ/REQ)]
(.connect s "tcp://127.0.01:5555")
(.send s msg)
(println "Server replied:" (String. (.recv s)))
(.close s)))
;
; The Publish-Subscribe Pattern
;
; Server
(defn market-data-publisher
[]
(let [s (.socket ctx ZMQ/PUB)
market-data-event (fn []
{:symbol (rand-nth ["CAT" "UTX"])
:size (rand-int 1000)
:price (format "%.2f" (rand 50.0))})]
(.bind s "tcp://127.0.01:6666")
(while :true
(.send s (c/generate-string (market-data-event))))))
; Client
(defn get-market-data
[num-events]
(let [s (.socket ctx ZMQ/SUB)]
(.subscribe s "")
(.connect s "tcp://127.0.01:6666")
(dotimes [_ num-events]
(println (c/parse-string (String. (.recv s)))))
(.close s)))
;
; The Pipeline Pattern
;
; Dispatcher
(defn dispatcher
[jobs]
(let [s (.socket ctx ZMQ/PUSH)]
(.bind s "tcp://127.0.01:7777")
(Thread/sleep 1000)
(dotimes [n jobs]
(.send s (str n)))
(.close s)))
; Worker
(defn worker
[]
(let [rcv (.socket ctx ZMQ/PULL)
snd (.socket ctx ZMQ/PUSH)
id (str (gensym "w"))]
(.connect rcv "tcp://127.0.01:7777")
(.connect snd "tcp://127.0.01:8888")
(while :true
(let [job-id (String. (.recv rcv))
proc-time (rand-int 100)]
(Thread/sleep proc-time)
(.send snd (c/generate-string {:worker-id id
:job-id job-id
:processing-time proc-time}))))))
; Collector
(defn collector
[]
(let [s (.socket ctx ZMQ/PULL)]
(.bind s "tcp://127.0.01:8888")
(while :true
(->> (.recv s)
(String.)
(c/parse-string)
(println "Job completed:")))))
Further Reading
ZeroMQ removes a lot of complexity from creating high-performance, scalable, distributed architectures, and I hope I’ve shown you a glimmer of it in this tutorial. If you want to learn more about advanced patterns, implementing persistence, and other ways to work with ZeroMQ, the first stop would be the excellent ZeroMQ Guide. Much of it is also reflected in a recent (2013) O’Reilly book.
JeroMQ is a fairly new project, so if you prefer using the C++ version with Java bindings, this is a good installation guide.
The code for this tutorial is available on github.
If you have any questions, thoughts, suggestions, or corrections, please leave a comment or email me at nitin (at) patternhatch (dot) com.
I’m always trying to share more, so if you enjoyed this post, then follow me on Twitter. Thanks for reading.
Nitin [About me]
Download this podcast episode.
I've said it before: one of the few downsides to working at Relevance is getting to hear all sorts of cool customer success stories that we can't talk about. Which is why Justin and I were thrilled to record an episode with Kurt Zimmer from Room Key, one of our clients and an enthusiastic user of both Clojure and Datomic. We talked about Relevance's relationship with Room Key, and about Kurt's experience successfully selling cutting-edge technologies into extremely large, conservative organizations.
Kurt is a sharp guy, and it was a pleasure to have him as a guest.
While I've got your attention, I'd also like to mention a few things:
The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.
You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!
Kurt chose "Reboot the Mission" by The Wallflowers as the intro, and Justin chose "Havana Burning" by Dan Reeder to end the show.
Episode Cover Art:
Production Assistance:
Extension without inheritance, Part 1