Planet Clojure

Epitrochoids in ClojureScript

2013-06-18T07:00:00+00:00

I started reading a book called Generative Art a few weeks ago and the first exercise, drawing a circle using trig functions, reminded me of my favorite After Dark screen saver, Rose. I have always wanted to understand how Rose worked and could see that this exercise had started to send me down the right path. A few weeks later — and with some help from Rose’s author! — and I now have a framework for building HTML5 Canvas-based generative artworks in ClojureScript.

Introducing Cassaforte

2013-06-17T19:38:00+00:00

TL;DR

Cassaforte is a new Clojure client for Apache Cassandra 1.2+. It is built around CQL 3 and focuses on ease of use. You will likely find that using Cassandra from Clojure has never been so easy.

Another Brick in the Wall

ClojureWerkz projects are mostly known for our data store clients. Data processing is a sweet spot for Clojure and that’s what we primarily use it for.

Over time we’ve developed and released a number of data store clients:

and a few others.

Today we are adding another client to this family, Cassaforte.

Cassaforte Story

Cassaforte was started about 11 months ago as an experiment. Back then we needed a database well suited for time series data, and Cassandra is a good choice. However, dealing with existing clients, namely Hector and Astyanax, was quite a bit of a pain.

In the end, using Cassandra’s low level Thrift client worked well and we have been improving the codebase bit by bit, while working on an event collection and analytics system.

A few months ago, DataStax released a new Java driver for Cassandra that was a lot better suited to what Cassaforte needed to power it than any other alternative.

We were able to switch Cassaforte to it in a couple of weeks. About the same time, a great fellow Max Penet released a Clojure DSL for generating CQL, Hayt.

Hayt makes working with CQL as nice as the new clojure.java.jdbc DSL:

(require '[clojurewerkz.cassaforte.cql :as cql])

(cql/prepared
   (cql/insert "users" {:name "Alex" :city "Munich")}))
(cql/prepared
   (cql/insert "users" {:name "Robert" :city "Berlin")}))

(cql/select "users" (where :name "Alex"))
;; => [{:name "Alex", :city "Munich"}]

(cql/select "users"
            (where :name [:in ["Alex" "Robert"]]))
;; => [{:name "Alex", :city "Munich"}
       {:name "Robert", :city "Berlin"}]

(require '[clojurewerkz.cassaforte.cql :as cql])

;; For clarity, we select :post_id column only
(cql/select "user_posts"
            (columns :post_id)
            (where :username "Alex")
            (order-by [:post_id :desc]))
;; => [{:post_id "post3"}
       {:post_id "post2"}
       {:post_id "post1"}]

(require '[clojurewerkz.cassaforte.cql :as cql])

(cql/select "user_posts"
            (columns :post_id)
            (where :username "Alex"
                   :post_id [> "post1"]
                   :post_id [< "post3"]))
;; => [{:post_id "post2"}]

Integrating Hayt took a few weeks as well. From there, it took some time to write initial documentation and add some polish.

Major props to the DataStax team for releasing such a nice and focused Java driver and Max for making querying Cassandra from Clojure such a good experience.

What’s In The Box

Cassaforte is young but already offers all the key features you’d expect from such a client:

Schema (keyspaces, tables/column families, indices) manipulation
All CQL operations
Inserts that work with Clojure data structures
CQL 3.0 queries, including queries with placeholders (?, a la JDBC)
Prepared statements
Counters
Multi-node (cluster) connections
Automatic deserialization of column names and values according to the schema
Lazy sequence-based queries over large result sets

Needless to say, as more features are added to Cassandra and the DataStax core Java driver, Cassaforte will soon to follow their lead.

Documentation

Cassaforte documentation is far from finished but it follows the “if it is not documented, it is not part of ClojureWerkz” rule. To get started, take a look at the Getting Started guide.

The rest of the guides are being worked on.

API reference is also available.

Thank You, Contributors

Cassaforte was started by Michael but ended up being a brain child of Alex.

We would like to thank Max Penet for providing a lot useful feedback, sharing Hayt with us and helping us define what we want Cassaforte to be and feel like.

News and Updates

New releases and updates are announced on Twitter. Cassaforte also has a mailing list, feel free to ask questions and report issues there.

Cassaforte is a ClojureWerkz Project

Cassaforte is part of the group of libraries known as ClojureWerkz, together with

Langohr, a Clojure client for RabbitMQ that embraces the AMQP 0.9.1 model
Monger, a Clojure MongoDB client for a more civilized age
Elastisch, a minimalistic Clojure client for ElasticSearch
Welle, a Riak client with batteries included
Neocons, a Clojure client for the Neo4J REST API
Quartzite, a powerful scheduling library

and several others. If you like Cassaforte, you may also like our other projects.

Let us know what you think on Twitter or on the Clojure mailing list.

Michael on behalf of the ClojureWerkz Team.

Clojure Gazette 1.42

2013-06-16T23:18:26+00:00

Clojure Gazette 1.42

Clojure on the rise

Clojure Gazette

Issue 1.42 -- June 16, 2013

Clojure on the rise

Editorial

Clojure manages risk better than any other technology.

-- Paul deGrandis

Clojure is getting exciting. If you weren't already excited, now is the time. There are several great Clojure conferences, a couple of competitions this summer, and Clojure is in the Adopt Group on the ThoughtWorks Technology Radar 2013. And last year Paul deGrandis compelled us to believe that Clojure has important advantages to other platforms. I believe that all of these are true and we'll see steady and strong growth in the next few years.

Sincerely,
Eric Normand <ericwnormand@gmail.com>

P.S. Feel free to email me any time. I love hearing from readers.

Articles

Comparing Clojure Serialization Libraries

If performance is an issue for you, this article should give you a good idea of what to expect in terms of relative performance of various Clojure serialization libraries.

My Clojure Workflow, Reloaded

Through numerous client projects, Stuart Sierra has developed a workflow that allows him to recompile and restart entire servers from the REPL during development.

How to Learn Emacs: A Hand-drawn One-pager for Beginners

A cute and useful introduction to Emacs for real beginners. It covers most of the important concepts that differ from other computing environments.

Clojurescript over Meteor

Meteor is a full-stack web server/client framework written in Javascript. It lets you write real-time web applications from the client down. Since it's written in Javascript, it's the perfect target for ClojureScript. This article tells you how to get set up.

Nodifying your Clojure

Another article explaining how to set up a Node server running compiled ClojureScript.

Libraries

primitive-math

A library that redefines the arithmetic operators to use only primitive math (no boxing). In theory, this could give you a seamless way to improve mathematical code.

Exploding Fish

A highly usable URL library. Another example of Clojure making built-in types better.

Video

Self and Self: Whys and Wherefores

David Unger recounts many lessons he learned during his career in computer science research. David Unger developed the Self language, which pioneered the prototype object system. He's a very creative thinker and has quite a few good points of view.

The Architecture of core.logic

David Nolen has built a lot of flexibility into core.logic. In this talk, he discusses how the constraints system allows multiple solvers to work at the same time. This is a significant step in the efficient evaluation of logic programs -- it decomplects the strategy for constraining the problem from the declarative specification of the problem.

The Data-Reader's Guide to the Galaxy

Steve Miner discusses reading tagged data literals in Clojure -- new in Clojure 1.5 -- and how they make Edn a significant player in the universal data format space.

Copyright © 2013 LispCast, All rights reserved.
unsubscribe from this list update subscription preferences

Making Sense Out of Datomic, The Revolutionary Non-NoSQL Database

2013-06-16T21:39:43+00:00

I have finally managed to understand one of the most unusual databases of today, Datomic, and would like to share it with you. Thanks to Stuart Halloway and his workshop! Why? Why?!? As we shall see shortly, Datomic is very different from the traditional RDBMS databases as well as the various NoSQL databases. It even […]

How We Document Our Projects with ClojureWerkz Docslate

2013-06-15T11:18:00+00:00

This post will explain how our team was able to greatly reduce the friction in writing documentation guides by standardizing on a toolchain that other projects can use.

A Bit of History

Before there was ClojureWerkz, myself and Alex were fairly active in the open source community. I maintain several RabbitMQ clients and needed a new documentation site for one of them.

So I put together a doc site based on Jekyll and Alex helped me with styles. This toolchain has evolved a bit and when we were looking at writing documentation for Monger and Langohr in late 2011, we decided to just base them on what we already had and knew: that Jekyll site amqp gem was using.

ClojureWerkz grew and we kept reusing the same thing over and over again. By June 2013, ClojureWerkz has accumulated about a dozen of projects that have substantial documentation guides. Writing, editing and maintaining these docs took time and in the process, we’ve found ourselves repeating a lot of things between sites.

It’s easy to notice a lot of similarities between, say, Elastisch docs and Langohr docs. We also try to take good ideas from doc sites developed later and feed them back into older sites.

So we needed a standard toolchain that all these documentation projects could use. In particular, we wanted it to give us a very fast way of writing the first couple of guides before we would worry about finer details and editing.

Enter ClojureWerkz Docslate

Docslate is a relatively unknown project. We don’t write about it on this blog. It does not get updates very often. It does, however, play a major role in ClojureWerkz: powering our documentation guides and making writing them a more pleasant and straightforward process.

What Is Docslate

Docslate is a Jekyll site with predefined structure and integrated Twitter Bootstrap. Once you have it going, with just a few edits you are ready for the work that really matters: writing helpful documentation guides.

Why You Should Adopt Docslate For Your Project

Docslate lets you get to writing quickly without spending too much time on issues such as

Should I use Jekyll, Awestruct, hack my own thing with Pandoc or use a wiki?
What library should I use for code highlighting?
How do I make the docs look decent?
What licenses should I use for the site and content?

and so on.

It’s dead easy to put together a pretty good documentation site with Docslate and then work on copy and styling once your first version is ready to go live.

How To Get Started With Docslate

Since Docslate uses Jekyll, you’ll need Ruby (any version, although we typically use 2.0 or 1.9.3 these days) and RubyGems installed.

Then install Bundler with

gem install bundler

Clone Doclsate Git repository with

git clone git://github.com/clojurewerkz/docslate.git my-project-guides

Change to your local clone, remove .git directory and re-init the repo (Docslate is simply a starting point, your doc site should really be in a new repo):

cd my-project-guides
rm -rf ./.git
git init
git add .
git commit -m "Initial commit: Docslate"

Then install dependencies with Bundler:

bundle install --binstubs

and you are ready to run a development server with

./bin/jekyll serve --watch

then navigate to localhost:4000.

The site you’ll see is very similar in structure and styles to Elastisch docs, Monger docs, and so on.

How To Add Content

Docslate assumes your documentation guides have an index page, a ToC page and one or more guides. The index page is in HTML and can be found in index.html in the repository root.

The ToC page is in Markdown and can be found with individual guides under articles.

Docslate contains 3 pages for you to start with:

  guides.md
  getting_started.md
  community.md

guides.md is the ToC page. getting_started.md contains the Getting Started guide. Finally, community.md gives you a page to list information about the project, its mailing list, IRC channel, Twitter account, and other things you need to make your open source project really awesome.

Adding new guides is as straightforward as adding a Markdown file to the articles directory with the header (title, etc) Jekyll expects and linking to it from the ToC page.

For example, to add a guide on extending your library, create a new file at articles/extensions.md, add Jekyll header to it, make sure development server is started and navigate to localhost:4000/articles/extensions.html.

How To Customize Layout

Layout files can be found under _layouts. There are two layouts available:

Default layout
Article layout

Article layout is used to display individual articles (guides), default layout is used for everything else.

How To Customize Styles and Assets

Styleshsets and JavaScript files can be found under assets/stylesheets and assets/javascripts, respectively.

How To Generate The Site

Use

./bin/jekyll build

to generate the site for deployment. Site root then will be under _site. This directory is what you will deploy. It only contains static assets so deployment usually is as straightforward as rsync-ing files to your server and serving them with Nginx, Apache or any other Web server of your choice.

Where To Go From Here

Docslate is a starting point. It is meant to be customized for your project’s needs, styling and so on. Feel free to do it, add JavaScript libraries you need, upgrade and customize Twitter Bootstrap, and so on.

Wrapping Up

Docslate was born out of a dozen documentation sites we have for ClojureWerkz projects. It’s a battle tested, easy to use toolchain that provides you a good starting point and is completely open sourced under a reasonable license.

Now you don’t have an excuse to not document your open source project well!

Michael on behalf of the ClojureWerkz Team.

Constructing Algebraic Expressions

2013-06-14T21:14:11+00:00

Constructing Expressions

Gsoc finally starts monday. And for the start I want to have a good idea of how the user interface of expresso should look like, for this I want to discuss the best way of aktually constructing expresso expressions :).

1 Representation of expresso expressions

Because core.logic currently doesn't play nicely with custom types and algebraic expressions fit nicely with normal clojure s-expressions I plan to represent algebraic expressions just as that. However, that doesn't mean it is straightforward to construct them - expresso has to know the full qualified function symbol to differentiate between - for example - clojure.core/* and the core.matrix * which behave differently. Also other mathematical properties should be stored about the functions - I plan to use metadata for it.

2 How should using expresso feel like

Hopefully as seamless as possible.

One approach would be a macro - create-expression. It would construct the enhanced form of the s-expression from the s-expression it gets. This, however has some drawbacks. First the user either needs to use ~ to aktually get the data in the expressions. Or you have to have map as argument which would map from symbols to values. Further than that, because it is a macro, it can't be used in higher order functions.

(let [matrix [[1 0 1]
              [0 1 0]
              [1 0 1]]]
  (simplify (construct-expression (* 5 (determinant ~matrix))))
  ;or 
  (simplify (construct-expression (* 5 (determinant x)) {'x matrix})))

The other approach would be to have construction functions in an own namespace, with concise names, just like core.matrix did with the operators namespace.

(ns test
  (:use [expresso.construction])
  (:refer-clojure :exclude [* - + == /]))
(let [matrix [[1 0 1]
              [0 1 0]
              [1 0 1]]]
 (simplify (* 5 (determinant matrix))))

Mike Anderson also posted some interesting usage examples (with a possible syntax) on the clojure mailing list:

; 1. Solving simultaneous equations expressed in arrays:

(solve [x y] (== [x y] (* [[3 0] [0 10]] [y 1])))
=> [30 10]

;2a. Derivatives of array expressions:

(derivative [x] (* [x 2] [3 (* x x)]))
=> [3 (* 4 x)]

;2b. Derivatives that are themselves vectors / arrays:

(derivative [[x y]] (* [x 2] [3 (* x x)]))
=> [[3 (* 4 x)] 0]

;3. Simplification of computations on arrays:

(simplify (trace [[x y] [z (- 5 x)]]))
=> 5

;4. Simplification of array-level expressions

(simplify (* 10 A (some-identity-matrix) (inverse A) B))
=> (* 10 B)

What would be your favorite way of using expresso?

3 expresso on core.matrix expressions

Considering the close relation between expresso and core.matrix it is very likely that expresso (especially the optimizer) will be used on core.matrix expressions. Mike's examples also point in this direction. What about making expresso an implementation of core.matrix ? It would not calculate the results of matrix operations, but create the expressions under the hood. That way one could just switch the implementation of core.matrix to make the matrix expression available to expresso.

For the core.matrix folks: Do you think that is possible? There are certainly a few caveats in it but it seems to be the perfect way of using expresso in a core.matrix context.

Date: 2013-06-14T23:09+0200

Author: Maik Schünemann

Org version 7.8.09 with Emacs version 23

Validate XHTML 1.0

our-foc.us

2013-06-14T15:12:10+00:00

Early this year I wrote a simple to-do planning web app in Clojurescript mostly for use on my smartphone. As a learning experiment with client/server side Javascript and Meteor I rewrote it to support real time collaboration: our-foc.us

This was an interesting learning experience, and I must say that Meteor if both easy to use and provides impressive functionality “out of the box.”

Memcache Support

2013-06-14T14:14:02+00:00

We're happy to announce today transparent integrated support for memcached in Datomic Pro Edition.

One of the nice things about the Datomic architecture is that the index segments kept in storage are immutable. That enables them to be cached extensively. Currently that caching happens inside the peers, which keep segments they have needed thus far in the application process heap.

While this is great for process-local working sets, there is only so much a single machine can cache. So, we've added support for an optional second tier of distributed, shared cache, leveraging a memcached cluster. This tier of cache can be as large as you wish, and is shared between all the peers.

The entire use of memcached is automatic and integrated - just provide the endpoints of your memcached cluster in configuration. The peer protocols will automatically both look in it, and populate it on cache misses. Being based upon immutability, there are no cache coherence problems nor expiration policy woes.

The architecture incorporating memcached looks like this:

The benefits of this are many:

You can get a shared cache of arbitrary size - many deployments will be able to fit their entire database in memcached if desired.
If you are using a storage that is not otherwise distributed (e.g. unclustered PostgreSQL), the memcached tier can both almost entirely remove the read load on the single server and distribute it.
Even when using a distributed storage like DynamoDB, the memcached tier can reduce your read provisioning and increase speed.
Developers can set up a small memcached daemon locally so their DB will always feel 'hot' across process restarts.
The memcached tier will enable hybrid strategies where the peers, transactors and memcached are all local but the storage is remote (e.g. DynamoDB).

Datomic is a good citizen in its use of memcached - it doesn't need to 'own' the cluster, and all of the Datomic keys incorporate UUIDs so they won't conflict with other application-level use of the same memcached cluster.

We hope you enjoy this feature, which is included in the Pro Edition at no extra charge.

The Database as a Value

2013-06-14T13:00:00+00:00

During QCon New York 2013, Rich Hickey gave a talk on functional databases. Hickey is well known for creating the Clojure programming language and is currently developing Datomic, which is a functional database. During his talk, Hickey argued that the useful properties of functional languages: data as values and pure functions, are just as useful in the context of databases. By Zef Hemel

Presentation: Engines of Abstraction

2013-06-14T02:00:00+00:00

Jim Duey surveys several abstraction techniques that can help in writing reusable code in Clojure. By Jim Duey

June 2013 London Clojure Dojo at ThoughtWorks

2013-06-13T23:34:25+00:00

When:
Tuesday, June 25, 2013 from 7:00 PM to 9:30 PM (BST)

Where:
ThoughtWorks London Office
173 High Holborn
WC1V London
United Kingdom

Hosted By:
London Clojurians

Register for this event now at :
http://london-clojure-dojo-tw-2013-06-tw-rss.eventbrite.com

Event Details:

London Clojure Dojo at ThoughtWorks

The goal of the session is to help people learn to start working with Clojure through practical exercises, but we hope that more experienced developers will also come along to help form a bit of a London Clojure community. The dojo is a great place for new and experienced clojure coders to learn more. If you want to know how to run your own dojo or get an idea of what dojos are like you can read more here.

We hope to break up into groups for the dojo. So if you have a laptop with a working clojure environment please bring it along.

We’ll be discussing the meetup on the london-clojurians mailing list

Clojure is a JVM language that has syntactically similarities to Lisp, full integration with Java and its libraries and focuses on providing a solution to the issue of single machine concurrency.

Its small core makes it surprisingly easy for Java developers to pick up and it provides a powerful set of concurrency strategies and data structures designed to make immutable data easy to work with. If you went to Rich Hickey’s LJC talk about creating Clojure you’ll already know this, if not it’s well worth watching the Rich Hickey “Clojure for Java Programmers” video or Stuart Halloway “Radical Simplicity” video .

Transitioning To Sequences

2013-06-13T20:17:34+00:00

Problem #157 in 4Clojure.com is listed as a very simple problem. Despite having used Clojure’s map function a lot, I saw the solution to this problem recursively using loop and recur, by walking through each element of the function and determining its count. The following solution came from http://blog.qinjian.me/2012/12/21/4clojure_solutions/ . Look for #157 in the midst of quite a few solutions.

(fn __[coll]
  (loop [x (- (count coll) 1)
      l []]
  (if (>= x 0)
    (recur (- x 1)
        (conj l (list (nth coll x) x)))
    (reverse l))))

This solution makes more sense to me than this solution.

#(map list % (range))

It just doesn’t look as obvious. But the prevailing wisdom is to use Clojure’s sequence functions to handle problems like this.

Incanter 1.5.1: Bugfix release

2013-06-13T18:46:06+00:00

The bugfix release of Incater was just pushed to Clojars.

It fixes 2 bugs – one nasty bug when functions called from transform-with were working on underlying data, modifying them, and 2nd – when 2-argument version solve thrown an error.

Clojure Cookbook

2013-06-13T14:39:38+00:00

Clojure Cookbook

From the webpage:

Clojure Cookbook is coming

And we need your help.

We want this O’Reilly cookbook to be a comprehensive resource containing the collective wisdom of Clojurists from every domain. That’s why we want to write it together, as a community.

Share some code. Explain it. Be a part of Clojure history.

Cool!

GitHub clojure-cookbook.

Clojure.core: Batteries (almost) included

2013-06-13T00:00:00+00:00

Today, I want to provide a guided tour through some of the many libraries available from the Clojure team that don't come distributed with Clojure. Consider them Clojures standard library. Some came from old clojure.contrib libs, others are brand-new, but all are great.

Foreword

I spend a lot of time thinking about how to make Clojure more popular. After all, if there's more code written in Clojure, there's more need for Clojure developers, which means more opportunities to use my favorite language of late in production. One potential barrier to adoption – if only a psychological one – is that Clojure, as a Lisp-derived language, tends to eschew the batteries-included philosophy of other, more mainstream languages. Java, PHP, Python and Ruby come to mind as languages I've used that come with extensive standard libraries by default, and I'm sure C# and Perl and all the other ones I've never really dealt with.

Clojure's default installation takes a different tack from these. Besides core, clojure includes only the following menagerie of utilities and helpers. Some of these are very useful indeed:

clojure.data - Just one function, diff, which might come in handy. Used as a namespace later though.
clojure.edn - Read edn-formatted data.
clojure.inspector - Graphical inspectors for Clojure data
clojure.java.io - Some very-useful clojurifications of java I/O libraries (streams, etc.)
clojure.java.shell - Exec commands from your environment
clojure.pprint - Pretty-printing
clojure.set - Set operations (union, difference, intersection, and so on).
clojure.string - All the usual string operations (trim, upper/lower case, replace, split, join, etc.) live here
clojure.template - A small utility for templated macroing
clojure.test - Testing utils.
clojure.walk - Some utilities for walking tree structures.
clojure.zip - Zipping functions. seq-zip, xml-zip and vector-zip seem to be the most useful bits at-a-glance.

Other libraries included with clojure may be mostly only useful to the clojure.core team themselves, or people writing tools for clojure:

clojure.instant - Timestamp parsers of no obvious utility
clojure.java.browse - Contains a single function, browse-url, which accepts a url and opens a browser.
clojure.java.javadoc - Quick browser access to javadocs
clojure.main - Functions for actually running Clojure itself
clojure.reflect - Some (apparently unstable) utilities for reflection)
clojure.repl - Useful functions for working with the repl.
clojure.stacktrace - Mostly, print stacktraces.
clojure.xml - RH's xml utils. I'd recommend the XML library to be discussed in a bit.

However, beyond the libraries distributed with clojure, there is a whole world of libraries maintained by the clojure team, mirroring many of the “batteries” from other languages. And thanks to Leiningen, these almost-first-class-members of the clojure experience are just an entry in :dependencies away.

First on our tour, we'll check out 3 libraries available in the clojure.data namespace. Most projects won't need all of these, but a good number will probably have occasion to use at least one.

data.csv

If a library doesn't come with an easy way to deal with csv files, it probably doesn't care about you and your petty problems. But Clojure loves you, baby, and clojure.data.csv is how you know it. (Actually, it's Jonas Enlund who loves you, and who wrote this lib)

(require '[clojure.data.csv :as csv]
         '[clojure.java.io :as io])

(with-open [in-file (io/reader "in-file.csv")]
  (doall
    (csv/read-csv in-file)))

(with-open [out-file (io/writer "out-file.csv")]
  (csv/write-csv out-file
                 [["abc" "def"]
                  ["ghi" "jkl"]]))

data.xml

I mentioned above Rich Hickey's effort, clojure.xml, but here's it's big brother. clojure.data.xml, maintained and likely written by Ryan Senior.

data.xml's parsing utilities return records of the following description

    #clojure.data.xml.Element{:tag :foo,
                              :attrs {},
                              :content ...}

You can deal with these parsed records just like maps, as usual. Here's an excerpt from some code I recently wrote to extract items from RSS feeds:

(defmulti get-items :tag)
(defmethod get-items :rss [doc]
  (->>
    doc
    :content
    first
    :content
    (filter-tag :item)
    (map :content)))

(defmethod get-items :feed [doc]
  (->>
    doc
    :content
    (filter-tag :entry)
    (map :content)))

(defmethod get-items :default [_] [])

You can construct XML documents by creating Elements yourself (using the (element [tag attrs content]) record definition), or you can use a Hiccup-esque syntax with sexp-as-element:

(sexp-as-element
  [:foo {:foo-attr "foo value"}
    [:bar {:bar-attr "bar value"}
      [:baz {} "The baz value"]]]))

Good stuff.

data.json

Like other JSON libraries for languages with literal hash maps, Stuart Sierra's clojure.data.json is best thought of as a conversion layer between hash maps and JSON strings, although this library also provides stream-based i/o.

(json/read-str "{\"a\":1,\"b\":2}")
;;=> {"a" 1, "b" 2}

(json/write-str {:a 1 :b 2})
;;=> "{\"a\":1,\"b\":2}"

What more do you need to know? If you need to deal with json in clojure, you want clojure.data.json. There's lots more functionality, so be sure to read the docs.

tools.macro

Adds a few syntactic sugars to macro definitions in clojure: macrolets, which are locally-scoped macros, and “symbol macros”.

(defsymbolmacro sym (+ a b))
(with-symbol-macros
  (+ 1 sym))
;; expands to (+ 1 (+ a b))

Some nice little utils. If your projects don't already have half-baked implementations of these, you should include clojure.tools.macro before you do.

tools.logging

clojure.tools.logging, by Alex Taggart, “wraps”, i.e. implements, all of the big Java logging libraries at once. Write your code once, and pick your implementation later.

The logging functions are exactly what you'd expect:

(defn divide [x y]
  (try
    (info "dividing" x "by" y)
    (/ x y)
    (catch Exception ex
      (error ex "There was an error in calculation"))))

If you're not using Timbre, you should be using clojure.tools.logging.

core.cache

I must admit, when I tried to use clojure.core.cache (written and maintained by Michael Fogus), I got a bit confused for a while before I realized that I had to be storing the output of the cache. Really, I don't mind if the caching library I use is a bit stateful!

But, I'm still happy about core.cache, because it paves the way for…

core.memoize

This library (also Fogus) is great. You use it to wrap a function, and it pops an atom right in the metadata of the function. Your function calls are instantly memoized.

You can pick and choose your caching strategy as per core.cache, too:

    (require '[clojure.core.memoize :refer (memo-lu)])

    (def id (memo-lu #(do (Thread/sleep 5000) (identity %)) 3))

    (id 42)
    ; ... waits 5 seconds
    ;=> 42

    (id 42)
    ; instantly
    ;=> 42

Go read all about it.

core.match

David Nolen's core.match is an alpha-quality attempt to bring pattern-matching to clojure. I prefer writing code to words, so let me explain it to you in the language of my people: Fizzbuzz!

(use '[clojure.core.match :only (match)])

(defn fizzbuzz
  "
  Print each number from 1 to 100, but for multiples of 3 print 'Fizz'
  instead, and for multiples of 5 print 'Buzz'. For multiples of both
  print 'FizzBuzz'
  "
  []
  (doseq [n (range 1 101)]
    (println
      (match [(mod n 3) (mod n 5)]
        [0 0] "FizzBuzz"
        [0 _] "Fizz"
        [_ 0] "Buzz"
        :else n))))

This library has a nice little writeup available for it, so I won't belabour it.

I personally can't wait for this to get out of alpha, and hopefully get absorbed into clojure.core

core.unify

The core.unify library, another Michael Fogus joint, is a library that brings unification) to clojure.

It's not a concept I was particularly familiar with before I started writing this, but I actually found the best examples in the comments section of the main library file from which the following examples are alternately inspired and outright stolen.

There are three primary functions exposed by core.unify.

(unify [expr1 expr2]) attempts to define what symbols (denoted by ?) it can given an expression.

(u/unify {:first '?first
          :last  '?last
          :genre :giallo}

         {:first "Dario"
          :last  "Argento"
          :genre :giallo})
;; => {?first "Dario" ?last "Argento"}

Here's most of the table from that wikipedia article, translated into core.unify

(u/unify 'a 'a)                     ;; => {}
(u/unify 'a 'b)                     ;; => nil
(u/unify '?x '?x)                   ;; => {}
(u/unify 'a '?x)                    ;; => {?x a}
(u/unify '?x 'y)                    ;; => {?x y}
(u/unify '(f a ?x) '(f a b))        ;; => {?x b}
(u/unify '(f a) '(g a))             ;; => nil
(u/unify '(f ?x) '(f ?y))           ;; => {?x ?y}
(u/unify '(f ?x) '(g ?y))           ;; => nil
(u/unify '(f (g ?x)) '(f ?y))       ;; => {?y (g ?x)}
(u/unify '(f (g ?x) ?x) '(f ?y a))  ;; => {?y (g ?x) ?x a}
(u/unify '?x '(f ?x))               ;; (throws an exception)

(subst [expr bindings]) is the inverse of unify, in a way. Given an expression and a map of bindings, it returns the expression with the bindings substituted. Symbols can be nested:

(u/subst
  '[1 2 ?x ?y]
  '{?x [3 4 ?y 6] ?y 1})
;; => [1 2 [3 4 1 6] 1]

(unifier [expr1 expr2]) performs a unification and then a substitution, resulting in the most complete version of the provided expressions that can be formed.

(defn my-unifier [expr1 expr2]
  (u/subst expr1
  (u/unify expr1 expr2)))
(my-unifier '(f ?y) '(f (g ?x))) ;; => (f (g ?x))
(u/unifier '(f ?y) '(f (g ?x)))  ;; => (f (g ?x))

The primary motivation for writing unifiers seems to be the creation of type inference systems, but I'm sure there's other cool stuff you can find to do with your newfound learning.

core.logic

This library, also written/maintained by David Nolen, clojure.core.logic is an implementation of both miniKanren and cKanren (pdf) in Clojure. You could think of it like core.unify, but fuller-featured.

It lets you write declarative statements, then run them against the logic engine to get a list of results satisfying the provided conditions.

Here's a sample from the primer:

  (run* [q]
    (membero q [1 2 3])
    (membero q [3 4 5]))
  ;; => (3)

Of course there are better ways to expression set intersection, but there are tons of applications that clojure.core.logic can help express more concisely.

There is pretty good documentation available on the project's wiki, but I prefer The Magical Island of Kanren as an intro (even if the syntax is mildly outdated).

Also, here's a very recent blog post by the author that I found really helpful.

core.typed

Statically-typed functional programs have a nice habit of just working, once you get your types all sorted out. clojure.core.typed, written by Ambrose Bonnaire-Sergeant, lets you do the next best thing in Clojure, by adding type annotations to your functions.

Here's how it works. First, write a function:

(ns example.typed
  (:require [clojure.core.typed :as t]))

(t/ann add [Number Number -> String])
(defn add [a b]
  (+ a b))

Then, call check-ns from somewhere within that namespace (I keep mine in a comment at the bottom of the file and run it with fireplace.vim):

(t/check-ns)

And, since I annotated that wrong, we'll see some errors:

Type Error (example.typed:7:3) Return type of static
method clojure.lang.Numbers/add is java.lang.Number, expected java.lang.String.
in: (clojure.lang.Numbers/add a b)


Type Error (example.typed:6) Expected type: String
Actual: Number
in: (clojure.lang.Numbers/add a b)


Type Error (example.typed:6:1) Expected type: (Fn [Number Number -> String])
Actual: (Fn [Number Number -> Number])
in: (def add (fn* #))

You probably won't annotate everything, but it's nice to have core.typed around when you forsee there might be a typing problem, and it's good to have your code checked anyhow.

core.contracts

I slacked off and left this out last night, but somebody called me out so now I'm back.

Fogus's clojure.core.contracts is for people who want their functions to be even more uptight than can be had with static typing. It's basically a replacement for starting your function with a bunch of assert calls. Here's the example from the home page:

(use 'clojure.core.contracts)

    (def secure-doubler
      (with-constraints
        (fn [n] (* 2 n))
        (contract doubler
          "ensures doubling"
          [x] [number? => (= (* 2 x) %)]
          [x y] [(every? number? [x y])
                   =>
                 (= (* 2 (+ x y)) %)])))

    (secure-doubler 10)
    ;=> 20

As you can see, you can specify pre- and post- conditions, and even extend the function itself. Pretty cool.

Conclusion

That's just a quick tour of some of the interesting libraries I found, but it's far from exhaustive. There are a ton more, though, so be sure to check out the Clojure github repo to see what else exists.

lein-immutant Plugin 1.0.0.beta1 Released

2013-06-12T22:45:56+00:00

As we approach a 1.0.0 of Immutant, we've also been focusing on getting our Leiningen plugin ready for a 1.0.0 release as well.

Today, we're as pleased as a punch bug player to announce the first 1.0.0 beta release of the plugin: 1.0.0.beta1.

For 1.0.0.beta1, we've made quite a few changes to the plugin, some of which are not backwards compatible. We'll talk about all of the changes, starting with the breaking ones.

The base install dir location and structure

When we wrote the initial version of the plugin, we used .lein/immutant as base directory for storing Immutant installs and other files. But as the plugin has grown, we've realized that's not the best place for storing those files, since they aren't Leiningen specific. Therefore, we've moved the base directory to ~/.immutant.

In addition, we've reorganized the layout of ~/.immutant/releases to be a bit cleaner.

Since the new plugin will be looking in ~/.immutant, it won't see the prior installs in ~/.lein/immutant. You'll need to re-install the Immutant versions you need, and can safely delete ~/.lein/immutant.

The plugin now installs the latest release by default

Pre-1.0.0.beta1, the plugin installed the latest incremental build when you executed lein immutant install. As we approach 1.0.0 of Immutant, we've changed the plugin to install the latest stable release instead.

So, to install the latest stable release:

lein immutant install

If you want to install the latest incremental build:

lein immutant install LATEST

You can still specify any stable release version or incremental build number to get an exact version.

Listing installed versions

A couple of releases ago, we added the lein immutant list command that lists currently deployed applications. In 1.0.0.beta1, this command has been extended with a --installs flag (or -i) to list all of the installed Immutant versions instead of deployments:

$ lein immutant list --installs
      The following versions of Immutant are installed to /home/tobias/.immutant
      (* is currently active via /home/tobias/.immutant/current):
      
         0.10.0                         (type: full)
         0.10.0                         (type: slim)
         0.6.0                          (type: full)
         0.9.0                          (type: slim)
         0.9.0                          (type: full)
       * 1.0.0.beta1                    (type: slim)
         1.x.incremental.879            (type: slim)
         1.x.incremental.882            (type: slim)

You'll also now see this list after every lein immutant install invocation.

Overriding the base directory

In addition to moving the base directory to ~/.immutant, we've added the option to override its location, either per-project or globally.

You can override it on a per-project basis by setting :lein-immutant {:base-dir "/path"} in your project.clj

You can override it globally two ways:

by setting :lein-immutant {:base-dir "/path"} in your user profile in .lein/profiles.clj
by setting an environment variable: $LEIN_IMMUTANT_BASE_DIR

Setting the base directory in project.clj will override the setting in .lein/profiles.clj. Using the environment variable will override both.

Get it

If you're already using lein-immutant you probably already know how to do this, but just in case - to install it, add it to the :plugins list in your :user profile in ~/.lein/profiles.clj:

{:user {:plugins [[lein-immutant "1.0.0.beta1"]]}}

Get in touch

We realize this release has quite a few changes, and we may not have all of the kinks worked out. If you have any questions, issues, or other feedback, you can always find us on #immutant on freenode or you can file an issue on lein-immutant.

Immutant 1.0.0.beta1 Released

2013-06-12T22:45:56+00:00

We are almost as excited to announce our very first beta release of Immutant as this guy is to see a train. And like that train, we expect our beta cycle to be short; our plan is to have 1.0.0 out by the end of the summer, after which we'll begin to incorporate new features from JBoss Wildfly for the next release. As always, view our road map here.

What is Immutant?

Immutant is an application server for Clojure. It's an integrated platform built on JBoss AS7 that aims to reduce the inherent incidental complexity in real world applications.

What's in this release?

We haven't changed a whole lot since 0.10.0. We wanted to spruce up our docs and make sure we didn't have any glaring stability issues. We made a couple of minor enhancements to the jobs and caching namespaces, but nothing that isn't backwards compatible.

Cluster configuration should be simpler since we're now taking advantage of HornetQ's ability to use JGroups. We added a Production Setup chapter to our manual which we hope to evolve as more Immutants are put in production. Speaking of that, check out Damion's excellent writeup of his experiences clustering Immutant and TorqueBox on Amazon's EC2.

See the complete list of changes below.

Get It

The simplest way to install or upgrade to 1.0.0.beta1 is via our Leiningen plugin (though we recommend you update your plugin version first):

$ lein immutant install 1.0.0.beta1

See our install page for more details. Once you have it installed, take a look at our tutorials.

Get In Touch

If you have any questions, issues, or other feedback about Immutant, you can always find us on #immutant on freenode or our mailing lists.

Issues resolved in 1.0.0.beta1

[IMMUTANT-82] - Add clustering chapter to docs
[IMMUTANT-175] - implement dynapath's add-classpath-url
[IMMUTANT-271] - Evaluate and possibly eliminate reflection in web.session/session-cookie-attributes
[IMMUTANT-274] - sign clojars artifacts before releasing
[IMMUTANT-276] - publish dists with the type in the dir name (slim/full)
[IMMUTANT-277] - Get immutant.xa working with clojure.java.jdbc 0.3.x
[IMMUTANT-278] - Add a util function to return the effective classpath of an app
[IMMUTANT-279] - Add get-queue function to immutant.messaging
[IMMUTANT-281] - Update to nrepl 0.2.3
[IMMUTANT-282] - Remove util/if-in-immutant
[IMMUTANT-283] - Add convenient keyword aliases and military time assumptions to job/schedule
[IMMUTANT-284] - Configure messaging (HornetQ) to cluster via JGroups
[IMMUTANT-286] - Add a note about how dynamic jobs behave in a cluster
[IMMUTANT-287] - proofread and update the docs
[IMMUTANT-288] - Modify cache/create to support :ttl [1 :second] as well as :ttl 1 :units :seconds

Messaging Using Clojure and ZeroMQ

2013-06-12T21:14:49+00:00

Messaging has multiple uses in system design. It can aid communication between application components, enable asynchronous processing, and form the basis for a distributed architecture.

My last project involved receiving financial data in multiple formats, transforming it into a single canonical form, analyzing it for patterns, and forwarding findings to a web app. This mapped naturally to a pipeline design.

By implementing each step as its own subsystem and having them all communicate through a messaging layer, a number of benefits were achieved. New formats were easy to support. Scaling was a matter of adding more processes and machines. Any application which needed transformed data, such as a dashboard, could get it simply by subscribing to the right component.

This messaging was implemented using ZeroMQ.

In this post, I’ll give you a quick introduction to ZeroMQ through Clojure and help you get up and running with it. By the end, I hope to show you how simple it can be to add basic messaging capabilities to your own projects.

Tutorial Overview
What we’ll cover

What we won’t cover:

High-level patterns. These patterns address reliability, persistence, slow joiners, and load-balancing among others. Please see further reading for more information.
Advanced features: the High Water Mark, message envelopes, etc.
The different varieties of transport outside of TCP.

ZeroMQ Overview
ZeroMQ is an embeddable networking library. It gives you an enhanced socket which encapsulates and handles much of the complexity that typically comes with connecting programs. Specifically, these sockets:

Handle I/O asynchronously.
Queue messages automatically.
Enable components to come and go dynamically. ZeroMQ handles automatically reconnecting.
Don’t impose a message format.

One of ZeroMQ’s goal is to make building distributed software akin to Lego. In that vein, connecting two application components is simply a matter of instantiating sockets on both ends with a configuration of your choice, and letting ZeroMQ handle the rest. Much like Clojure, creating a ZeroMQ-based topology means starting with basic communication patterns, and then composing them into higher-level systems suited to your goal.

ZeroMQ Sockets vs TCP Sockets
ZeroMQ sockets exhibit a few differences from typical TCP sockets:

ZeroMQ sockets carry messages rather than a stream of bytes. A ZeroMQ message is length-specified binary data.
ZeroMQ I/O is automatically handled in a background thread, where output is enqueued and input is dequeued from local queues as your application goes about its business.
ZeroMQ sockets come in a variety of types, a few of which have 1-N routing built-in.

In addition, ZeroMQ messages can go across multiple transports (inproc, ipc, tcp, pgm, epgm), bound endpoints accept connections automatically, and rather than working directly with connections (which are encapsulated under the socket), you work with the socket instead.

ZeroMQ vs Other Messaging Tools
From a user’s perspective, the biggest difference between ZeroMQ and other popular messaging solutions (such as RabbitMQ) is the lack of an external component to provide easy routing, persistence, and load-balancing; features which RabbitMQ’s broker architecture lets you get quite easily.

In contrast, ZeroMQ is designed for high-throughput/low-latency applications. It works at a lower-level as a socket library and is designed to be embedded in your application. Its brokerless nature is also the origin of ‘Zero’ in the library’s name. Advanced messaging schemes are then built by combining pieces of the framework.

Setup
ZeroMQ is written in C++ and offers bindings for a number of languages, including Java through JNI. In the past, I used the Java bindings (JZMQ) but there is now a native Java implementation called JeroMQ. For simplicity, that’s what we’ll use here today. The examples we’ll cover are simple enough that we’ll use Java interop rather than wrap the calls in Clojure functions.

Start by creating a new Clojure project called clj-zmq. I use Leiningen:

lein new clj-zmq

Next, open project.clj in your project’s root folder and add these to your dependencies:

[com.rmoquin.bundle/jeromq "0.2.0"]
[cheshire "5.2.0"]

Cheshire is a JSON encoding library which we’ll use in our examples.

Set main to clj-zmq.core:

:main clj-zmq.core

This is what my project.clj file looks like:

(defproject dist "0.1.0-SNAPSHOT"
  :description "Messaging With Clojure and ZeroMQ"
  :url "http://patternhatch.com"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.5.1"]
                 [com.rmoquin.bundle/jeromq "0.2.0"]
                 [cheshire "5.2.0"]]
  :main clj-zmq.core)

Save the file, switch to your project’s root directory, and run:

lein deps

That should set your dependencies.

Open core.clj and set up the namespace:

(ns clj-zmq.core
  (:import [org.jeromq ZMQ])
  (:require (cheshire [core :as c])))

Contexts
The first thing we need to do is create a context, which is what we’ll use to create our sockets. A context is a container for all our sockets. In general, you should create one context per process. Creating multiple contexts in a single process is akin to creating separate ZeroMQ instances.

Continuing in core.clj, let’s create a context:

(def ctx (ZMQ/context 1))

The argument in the context creation call above is the number of threads to use. The ZeroMQ FAQ recommends one thread per gigabyte of data in or out per second. Now that we have our context, we can go over sockets before moving on to our examples.

Sockets
When using ZeroMQ, your application communicates by dealing with the socket, rather than the connection the socket handles. The socket, depending on its type, encapsulates policies for inward and outward routing, queuing, and what other types of sockets it can connect to. These relationships form messaging patterns, which give rise to ZeroMQ’s power and ability to layer additional patterns. The three basic socket pairings are:

REQ/REP [Request-Reply]
PUB/SUB [Publish-Subscribe]
PUSH/PULL [Pipeline]

To connect two sockets, you perform a bind on one and a connect on the other. In general, the socket with the known network address (such as the server) should bind, and the client(s) with unknown addresses should connect. Let’s see this in action with our first core ZeroMQ pattern.

The Request-Reply Pattern
For the request-reply pattern, we’ll create a simple echo server for our client to connect to. Let’s take a look at the code and go over the salient lines:

(defn echo-server
  []
  (let [s (.socket ctx ZMQ/REP)]
    (.bind s "tcp://127.0.01:5555")
    (loop [msg (.recv s)]
      (.send s msg)
      (recur (.recv s)))))

Here’s what’s happening:

On line 3, we use our context to create a REP socket.
On line 4, we bind our socket describing the transport mechanism (tcp), the IP, and the port number
On line 5, we wait for input using our socket’s recv command.
On line 6, upon receiving a message from the client, we send it right back using our socket’s send command

All in all, pretty straightforward.

One thing to note: ZeroMQ doesn’t know anything about the data we send except its size in bytes. That means we’re responsible for formatting it safely.

In our examples, we’ll be sending strings back and forth. JeroMQ has an overloaded send command which, when called only with a single argument, assumes the data is a string and formats it accordingly for us. For other types of data, we would need to take additional care and perhaps use specialized libraries such as protocol buffers.

Let’s move on to our client:

(defn echo
  [msg]
  (let [s (.socket ctx ZMQ/REQ)]
    (.connect s "tcp://127.0.01:5555")
    (.send s msg)
    (println "Server replied:" (String. (.recv s)))
    (.close s)))

This code looks similar to our server with three differences:

On line 3, we create a REQ socket.
On line 4, we use connect rather than bind.
On line 7, we close our socket once the server replies.

Let’s give it a try. Start your REPL (you should be in the clj-zmq.core namespace). In our examples, rather than starting multiple REPLs, we’ll run certain servers in separate threads. Start the echo server with the following command:

(future-call echo-server)

And now send a message to it using the client:

(echo "Hello!")

You should get back the following response:

Server replied: Hello!

This is the request-reply pattern, probably the simplest way to use ZeroMQ. The client issues a send command and then a receive command. Any other action such as sending multiple messages in a row will result in a -1 return code.

The Publish-Subscribe Pattern
The second pattern we’ll look at is used for one-way data distribution. For our example, we’ll create a server which publishes random stock trades and a client which subscribes to it. Here’s the server:

(defn market-data-publisher
  []
  (let [s (.socket ctx ZMQ/PUB)
        market-data-event (fn []
                            {:symbol (rand-nth ["CAT" "UTX"])
                             :size (rand-int 1000)
                             :price (format "%.2f" (rand 50.0))})]
    (.bind s "tcp://127.0.01:6666")
    (while :true
      (.send s (c/generate-string (market-data-event))))))

By now, the code probably looks familiar to you. Once again, we use the context to create our PUB socket. We then bind it to port 6666 and begin generating market data events. c/generate-string is a Cheshire command to turn a Clojure map into a JSON string. When no clients are connected, the server silently drops its messages.

Here’s the client:

(defn get-market-data
  [num-events]
  (let [s (.socket ctx ZMQ/SUB)]
    (.subscribe s "")
    (.connect s "tcp://127.0.01:6666")
    (dotimes [_ num-events]
      (println (c/parse-string (String. (.recv s)))))
    (.close s)))

Our client connects to the publisher, processes a certain number of market events, and ends. Notice that in addition to creating the SUB socket and connecting it, we issue a subscribe command on Line 4. In this case, issuing a subscribe command with an empty string means we wish to receive everything from the publisher (your client can specify filters on what to receive). Not issuing this command is a common oversight. If you use this pattern and find yourself not receiving anything, check for this.

The c/parse-string command on line 7 converts a JSON string into a Clojure map.

Reload your REPL and give it a try. Start the publisher in a separate thread:

  (future-call market-data-publisher)

Now listen to 100 market data events:

  (get-market-data 100)

You should get output similar to the following:

{symbol UTX, size 303, price 18.42}
{symbol CAT, size 492, price 41.02}
{symbol CAT, size 971, price 47.48}
.
.
.

Pub-Sub is a one-way, asynchronous line from the server to the client. An error results if either the server tries to receive or the client tries to send.

One issue this example doesn’t address is slow joiner syndrome. When your client initiates a connection, it takes time to step through the connection routine. This may take milliseconds, but during that time, the publisher may have already sent a few hundred or thousand messages which will be missed by the client. This may not matter if the publisher sends data endlessly and the client doesn’t care what transpired before. But if you want to ensure the client is well and ready before the publisher begins sending, then there’s an advanced pattern in the documentation for reliable pub-sub.

The Pipeline Pattern
Our final example covers the most involved of the three core patterns. The pipeline pattern is suited for task distribution and collection. For our example, we’ll set up a dispatcher which generates tasks and sends them to workers in a round-robin fashion. These workers will be connected upstream to the dispatcher and downstream to a collector. The collector will receive output from each worker as they complete.

Here’s the code for the dispatcher:

(defn dispatcher
  [jobs]
  (let [s (.socket ctx ZMQ/PUSH)]
    (.bind s "tcp://127.0.01:7777")
    (Thread/sleep 1000)
    (dotimes [n jobs]
      (.send s (str n)))
    (.close s)))

Similar to the previous examples, we create a PUSH socket and bind it to port 7777. But we add an additional step where we wait for one second before dispatching. Why?

Originally, I had the dispatcher prompt the user to hit enter when ready.The reason for this is because we want to ensure all the workers are connected before beginning. This way, the first worker to connect won’t be inundated with an outsized set of tasks. Nevertheless, workers are still free to come and go as they please and the dispatcher will automatically adjust the routing accordingly.

But because certain versions of nREPL have a bug where the read-line command causes an I/O exception in the REPL, I chose to go with this arbitrary timer instead for this example to avoid confusion. For a production scenario, the ZeroMQ guide covers more sophisticated means to signal when dispatch is ready.

Here’s the worker:

(defn worker
  []
  (let [rcv (.socket ctx ZMQ/PULL)
        snd (.socket ctx ZMQ/PUSH)
        id (str (gensym "w"))]
    (.connect rcv "tcp://127.0.01:7777")
    (.connect snd "tcp://127.0.01:8888")
    (while :true
      (let [job-id (String. (.recv rcv))
            proc-time (rand-int 100)]
        (Thread/sleep proc-time)
        (.send snd (c/generate-string {:worker-id id
                                       :job-id job-id
                                       :processing-time proc-time}))))))

The worker has two sockets, one pulling from the dispatcher and another pushing to the collector (on a different port). Each worker has a unique ID and simulates performing a task for a random duration. It then forwards information about itself, the job, and how long it took to the collector.

Here’s the code for the collector:

(defn collector
  []
  (let [s (.socket ctx ZMQ/PULL)]
    (.bind s "tcp://127.0.01:8888")
    (while :true
      (->> (.recv s)
           (String.)
           (c/parse-string)
           (println "Job completed:")))))

The collector offers an endpoint where it pulls completed tasks from all the workers in a fair-queuing fashion. Let’s give it a try. Reload your REPL, and start your collector:

(future-call collector)

Then start three workers:

(future-call worker)
(future-call worker)
(future-call worker)

Finally, dispatch 100 jobs:

(dispatcher 100)

You should see output similar to the following:

Job completed: {worker-id w2706, job-id 69, processing-time 53}
Job completed: {worker-id w2712, job-id 79, processing-time 71}
Job completed: {worker-id w2709, job-id 71, processing-time 96}
.
.
.

As you can see, all three workers are processing tasks in parallel and completing them in variable time.

Here’s a complete listing for core.clj:

(ns clj-zmq.core
  (:import [org.jeromq ZMQ])
  (:require (cheshire [core :as c])))

(def ctx (ZMQ/context 1))

;
; The Request-Reply Pattern
;
; Server
(defn echo-server
  []
  (let [s (.socket ctx ZMQ/REP)]
    (.bind s "tcp://127.0.01:5555")
    (loop [msg (.recv s)]
      (.send s msg)
      (recur (.recv s)))))

; Client
(defn echo
  [msg]
  (let [s (.socket ctx ZMQ/REQ)]
    (.connect s "tcp://127.0.01:5555")
    (.send s msg)
    (println "Server replied:" (String. (.recv s)))
    (.close s)))

;
; The Publish-Subscribe Pattern
;
; Server
(defn market-data-publisher
  []
  (let [s (.socket ctx ZMQ/PUB)
        market-data-event (fn []
                            {:symbol (rand-nth ["CAT" "UTX"])
                             :size (rand-int 1000)
                             :price (format "%.2f" (rand 50.0))})]
    (.bind s "tcp://127.0.01:6666")
    (while :true
      (.send s (c/generate-string (market-data-event))))))

; Client
(defn get-market-data
  [num-events]
  (let [s (.socket ctx ZMQ/SUB)]
    (.subscribe s "")
    (.connect s "tcp://127.0.01:6666")
    (dotimes [_ num-events]
      (println (c/parse-string (String. (.recv s)))))
    (.close s)))

;
; The Pipeline Pattern
;
; Dispatcher
(defn dispatcher
  [jobs]
  (let [s (.socket ctx ZMQ/PUSH)]
    (.bind s "tcp://127.0.01:7777")
    (Thread/sleep 1000)    
    (dotimes [n jobs]
      (.send s (str n)))
    (.close s)))

; Worker
(defn worker
  []
  (let [rcv (.socket ctx ZMQ/PULL)
        snd (.socket ctx ZMQ/PUSH)
        id (str (gensym "w"))]
    (.connect rcv "tcp://127.0.01:7777")
    (.connect snd "tcp://127.0.01:8888")
    (while :true
      (let [job-id (String. (.recv rcv))
            proc-time (rand-int 100)]
        (Thread/sleep proc-time)
        (.send snd (c/generate-string {:worker-id id
                                       :job-id job-id
                                       :processing-time proc-time}))))))

; Collector
(defn collector
  []
  (let [s (.socket ctx ZMQ/PULL)]
    (.bind s "tcp://127.0.01:8888")
    (while :true
      (->> (.recv s)
           (String.)
           (c/parse-string)
           (println "Job completed:")))))

Further Reading
ZeroMQ removes a lot of complexity from creating high-performance, scalable, distributed architectures, and I hope I’ve shown you a glimmer of it in this tutorial. If you want to learn more about advanced patterns, implementing persistence, and other ways to work with ZeroMQ, the first stop would be the excellent ZeroMQ Guide. Much of it is also reflected in a recent (2013) O’Reilly book.

JeroMQ is a fairly new project, so if you prefer using the C++ version with Java bindings, this is a good installation guide.

The code for this tutorial is available on github.

If you have any questions, thoughts, suggestions, or corrections, please leave a comment or email me at nitin (at) patternhatch (dot) com.

I’m always trying to share more, so if you enjoyed this post, then follow me on Twitter. Thanks for reading.

Nitin [About me]

Kurt Zimmer of Room Key - Podcast Episode 033

2013-06-12T19:37:00+00:00

Download this podcast episode.

I've said it before: one of the few downsides to working at Relevance is getting to hear all sorts of cool customer success stories that we can't talk about. Which is why Justin and I were thrilled to record an episode with Kurt Zimmer from Room Key, one of our clients and an enthusiastic user of both Clojure and Datomic. We talked about Relevance's relationship with Room Key, and about Kurt's experience successfully selling cutting-edge technologies into extremely large, conservative organizations.

Kurt is a sharp guy, and it was a pleasure to have him as a guest.

While I've got your attention, I'd also like to mention a few things:

Clojure/conj registration is open! Head on over to clojure-conj.org to get ticket information.
We will be offering 3 days of Clojure-related training just prior to the conference, with one day each of ClojureScript, Pedestal, and Datomic.
We are pleased to announce that the Scheme and Functional Programming Conference will be co-located with the Conj this year on Wednesday, November 13.
Lambda Jam Chicago is happening July 8-10, 2013. Stuart Sierra will be there speaking about Data, Visibility & Abstraction. The conference is being put on by our good friend Alex Miller, who has been kind enough to give us the offer code RELPOD, good for a $50 discount off your ticket. Go to http://regonline.com/lambdajam2013 to redeem it!

The show is available on iTunes! You can also subscribe to the podcast using our podcast feed.

You can send feedback about the show to podcast@thinkrelevance.com, or leave a comment here on the blog. Thanks for listening!

Our Guests, Kurt Zimmer and Justin Gehtland

Kurt Zimmer
- on the Room Key website
- on LinkedIn
Justin Gehtland
- on Github
- on Twitter

Music

Kurt chose "Reboot the Mission" by The Wallflowers as the intro, and Justin chose "Havana Burning" by Dan Reeder to end the show.

Credits

Episode Cover Art:

Michael Parenteau

Production Assistance:

Java.next: Extension without inheritance, Part 1

2013-06-12T18:00:10+00:00

Extension without inheritance, Part 1

Planet Clojure

Epitrochoids in ClojureScript

Introducing Cassaforte

TL;DR

Another Brick in the Wall

Cassaforte Story

What’s In The Box

Documentation

Thank You, Contributors

News and Updates

Cassaforte is a ClojureWerkz Project

Clojure Gazette 1.42

Clojure Gazette

Issue 1.42 -- June 16, 2013

Clojure on the rise

Editorial

Articles

Libraries

Video

Making Sense Out of Datomic, The Revolutionary Non-NoSQL Database

How We Document Our Projects with ClojureWerkz Docslate

A Bit of History

Enter ClojureWerkz Docslate

What Is Docslate

Why You Should Adopt Docslate For Your Project

How To Get Started With Docslate

How To Add Content

How To Customize Layout

How To Customize Styles and Assets

How To Generate The Site

Where To Go From Here

Wrapping Up

Constructing Algebraic Expressions

Constructing Expressions

Table of Contents

1 Representation of expresso expressions

2 How should using expresso feel like

3 expresso on core.matrix expressions

our-foc.us

Memcache Support

The Database as a Value

Presentation: Engines of Abstraction

June 2013 London Clojure Dojo at ThoughtWorks

London Clojure Dojo at ThoughtWorks

Transitioning To Sequences

Incanter 1.5.1: Bugfix release

Clojure Cookbook

Clojure.core: Batteries (almost) included

Foreword

data.csv

data.xml

data.json

tools.macro

tools.logging

core.cache

core.memoize

core.match

core.unify

core.logic

core.typed

core.contracts

Conclusion

lein-immutant Plugin 1.0.0.beta1 Released

The base install dir location and structure

The plugin now installs the latest release by default

Listing installed versions

Overriding the base directory

Get it

Get in touch

Immutant 1.0.0.beta1 Released

What is Immutant?

What's in this release?

Get It

Get In Touch

Issues resolved in 1.0.0.beta1

Messaging Using Clojure and ZeroMQ

Kurt Zimmer of Room Key - Podcast Episode 033

Our Guests, Kurt Zimmer and Justin Gehtland

Music

Links