Planet Clojure

memoize done right

2010-03-15T22:53:00+00:00

My recent post on how to make memoize more flexible has sparked an interesting discussion about the actual implementation. Christophe Grand and Eugen Dück joined to discuss almost everything Clojure gives you as a tool to handle concurrency. Don't miss the evolution of a quite simple function in a non-concurrent world to a quite complex function in case multiple threads of execution are involved.

Building Clojure with Maven

2010-03-15T18:48:51+00:00

Just a quick reminder on the steps I needed to take to build Clojure and install it in my local repository:

download the Maven Ant tasks, version 2.1.0 at the time of writing;
move them to $ANT_HOME/lib (this is /usr/share/ant/lib on Mac OS X);
cd to my Clojure download folder, run git pull if needed;
run ant -Dsnapshot.repo.dir=~/.m2/repository clean nightly-build, this will perform a clean build and install it into the supplied repo; this can be anywhere, the default though is /var/www/maven-snapshot-repository which is probably no good;
to also install clojure-contrib just change to it’s download directory and run mvn install.

Is simples!

This weekend in the Intertweets (March 14th Ed)

2010-03-15T08:13:32+00:00

Talked with two devs (1 java, 1 C#) interested in clojure but repulsed by emacs – no one had yet pointed them at the various IDE plugins (via @cemerick) — This post started a long thread of tweets. Clearly newcomers to Clojure need to know about all existing IDEs earlier in their research. We do our part.
- Adding clojure to an existing slime setup in emacs: The current recommended setup of emacs and… (here, via @mohegskunkworks) — This is an example of why Emacs might scare new Clojure adopters away…
InfoQ Interview: Stuart Halloway on Clojure and Functional Programming (here, via @infoq) — Stuart covers a lot of ground in this interview.
Variations on a parametrized memoize in Clojure (here, via @cgrand) — This is a follow-up response to @kotka’s article on how to write modified versions of memoize with different caching strategies
Google App Engine (GAE) Blog mentions Clojure as a programming language for GAE. (here, via @smartrevolution)
clj-native: see a C function being passed a clojure fn as a callback, great stuff (here, via @kyleburton) — You can quickly see how it works with this example. Now you can use any native library from Clojure.
New blog post: Yahoo Finance in Clojure (here, via @pjlegato) — This is a re-implementation of a Ruby gem in Clojure by its own author. This plugin fetches financial data from Yahoo Finance. It is still slower than its Ruby counterpart, but the author thinks this is mainly because the Ruby version is using a native HTTP client library while his Clojure implementation is using Apache HttpClient.
Mar 17: Clojure by Stu Halloway and Rich Hickey (here, via @DavidValeri) — If you are near Reston, VA on March 17 then you should definitely plan to attend.
A little blog post showing how to integrate Rich Hickey into your Emacs buffer (here, via @LauJensen) — Some notes on Emacs configuration, especially the IRC client (ERC)
An in-depth tour of Enlive (here, via @swannodette) — Enlive is a selector-based templating and transformation system for HTML in Clojure. This is a very powerful library, but the documentation has always been rather sparse. This tutorial fixes this situation by providing an in-depth tour through all Enlive features.
Russian speaking #clojure community at LiveJournal (here, via @alexott_en)
Reflections on leaving Haskell (to clojure) (here, via @jneira) — A glimpse on what brings other FP developers over to Clojure (hint: it’s pragmatism)
Setting Clojure’s Log Level (here, via @pjlegato) — “Clojure.contrib.logging doesn’t have any way to set the log level. This is obviously a problem if you want to make use of various log levels (debug, warn, etc.) to separate different logging depths. Here’s a function to set the logging level on my default clojure.contrib.logging setup”

Approaching Productivity

2010-03-13T21:29:01+00:00

Time is scarce these days — and with no prospect of any extraordinary reduction in the number of tasks I need to work on my level of productivity. Here are some thoughts on getting more out of Emacs.

Preface

Productivity is many things, starting with the way we think, through the habits we end up adapting, to the tools we use. Many years of software development has made clear to me the effect of having a good IDE and I have thus defined 3 factors which greatly improve productivity and 3 factors which inhibit it.

The three factors which make you more productive:

Emacs Emacs Emacs

The three factors which slow you down:

Vim Vim Vim

A question which came up recently in #clojure was exactly that of editors, is Emacs the only way to go ? Answer: I’m not sure. Vim-Clojure is pretty powerful, but oddly enough it only works with Vim. For NetBeans we have Enclojure, which broke during installation of the latest version from Github, but I have previously had luck installing earlier versions. And finally there’s talk of an IntelliJ plugin called LaClojure which looks very promising as well — If you’re not approaching Clojure from an Emacs background, try out those 2 if Emacs is too big a mouthful.

Emacs

Emacs has a great pull on developers using all languages and it’s no different for the Clojurian. Emacs has editing capabilities which aid producitivty quite a bit, but its primary pull comes from its vast integration capabilities with other systems. Here are a few key components of my setup:

Color-Themes which aid reading and reduce strain on eyes
Git integration, which makes pushing/pulling/merging/rebasing/stashing/committing etc a breeze
Gist integration which lets me share snippets in a second
ORG mode which lets me manage time, appointments and above all TODO lists
Clojure-mode/SLIME which lets me do REPL coding, reflections, source surfing etc
ERC which lets me chat on IRC networks directly in a buffer

Color-Themes

Normally there’s not a lot to say about this — You have a number of themes to pick from, select one you like and thats it. Except for me its not. I use htmlize whenever I post code on this blog, meaning I mark a region in my blog, hit M-x blog and the code is converted to colored HTML which is then sent to my clipboard. That means when I change color-themes, the blog changes looks. Earlier today etate pasted a home-grown theme on #clojure which I immediately adopted:

(defun color-theme-dark-bliss ()
  ""
  (interactive)
  (color-theme-install
   '(color-theme-dark-bliss
     ((foreground-color . "#eeeeee")
      (background-color . "#001122")
      (background-mode . dark)
      (cursor-color . "#ccffcc"))
     (bold ((t (:bold t))))
     (bold-italic ((t (:italic t :bold t))))
     (default ((t (nil))))

     (font-lock-builtin-face ((t (:foreground "#f0f0aa"))))
     (font-lock-comment-face ((t (:italic t :foreground "#aaccaa"))))
     (font-lock-delimiter-face ((t (:foreground "#aaccaa"))))
     (font-lock-constant-face ((t (:bold t :foreground "#ffaa88"))))
     (font-lock-doc-string-face ((t (:foreground "#eeccaa"))))
     (font-lock-doc-face ((t (:foreground "#eeccaa"))))
     (font-lock-reference-face ((t (:foreground "#aa99cc"))))
     (font-lock-function-name-face ((t (:foreground "#ffbb66"))))
     (font-lock-keyword-face ((t (:foreground "#ccffaa"))))
     (font-lock-preprocessor-face ((t (:foreground "#aaffee"))))
     (font-lock-string-face ((t (:foreground "#bbbbff")))))))

Call that function from anywhere in your .emacs and you’re set.

ERC

I’ve gone back and forth on this a lot, but today I’ve decided that having your IRC contacts directly in your buffers make sense, so I thought I’d share. To get started, grap ERC and add this to your .emacs:

(add-to-list 'load-path "~/.emacs.d/erc")
(require 'erc)

That will load the basic functionality. Then to set up ERC to track your nickname(s) do something like this:

;; Only track my nick(s)
(defadvice erc-track-find-face (around erc-track-find-face-promote-query activate)
  (if (erc-query-buffer-p)
      (setq ad-return-value (intern "erc-current-nick-face"))
    ad-do-it))

(setq erc-keywords '("Lau" "lau" "ljensen"))

(setq erc-track-exclude-types '("JOIN" "NICK" "PART" "QUIT" "MODE"
                                "324" "329" "332" "333" "353" "477"))

That will also keep ERC from annoying you with JOINS/PARTS. Then as a little something extra, I added OSD notifications for all you Ubuntu users:

;; Use libnotify

(defun clean-message (s)
  (setq s (replace-regexp-in-string "'" "&apos;"
  (replace-regexp-in-string "\"" "&quot;"
  (replace-regexp-in-string "&" "&"
  (replace-regexp-in-string "<" "&lt;"
  (replace-regexp-in-string ">" "&gt;" s)))))))

(defun call-libnotify (matched-type nick msg)
  (let* ((cmsg  (split-string (clean-message msg)))
        (nick   (first (split-string nick "!")))
        (msg    (mapconcat 'identity (rest cmsg) " ")))
    (shell-command-to-string
     (format "notify-send -u critical '%s says:' '%s'" nick msg))))

(add-hook 'erc-text-matched-hook 'call-libnotify)

So now when somebody highlights you, you’ll get something like this:

While chatting, people throw many links which don’t exactly follow the http://www.dom.com syntax, so to agressively identify those links and make them clickable, add:

;; Be overly eager to identify URLs

 (setq erc-button-url-regexp
      "\\([-a-zA-Z0-9_=!?#$@~`%&*+\\/:;,]+\\.\\)+[-a-zA-Z0-9_=!?#$@~`%&*+\\/:;,]*[-a-zA-Z0-9\\/]")

I looted that from EmacsWiki and it seems to work, although I’d hate to debug it.

;; Enable logging

(setq erc-log-channels-directory "~/.erc/logs/")
(setq erc-save-buffer-on-part t)
(defadvice save-buffers-kill-emacs (before save-logs (arg) activate)
(save-some-buffers t (lambda () (when (eq major-mode 'erc-mode) t))))

That’ll make sure you get everything logged correctly, so all we need to do now, is launch ship:

;; Launch

(setq erc-autojoin-channels-alist
      '(("freenode.net" "#clojure")))

(erc-select :server "irc.freenode.net" :port 6667 :nick "yournick")

There we go! Now you have Rich Hickey and the gang right in your favorite IDE and you can copy/paste snippets directly from #clojure to your REPL — Hows that for productive? If you really want to get going with ERC, make sure you stop by the EmacsWiki entry, especially the logging can be improved I think.

More?

If I had more time to blog about Emacs extensions, Org-Mode would be at the top of my list. Its a very powerful mode which lets to integrate task planning seamlessly with the rest of your business in Emacs, I recommend you check it out.

Regarding Raynes’ quote, we were discussing meta-programming, namely macros in Ruby and seemed to be the case that Ruby macros are so far from Lisp macros, that claiming a connection is a stretch — any Rubyists who can elaborate on why Ruby is called a “Lisp chainsaw” ?

Leiningen (unknown severity): help task fails; "java.util.zip.ZipException: error in opening zip file"

2010-03-13T00:00:00+00:00

Leiningen (unknown severity): help task fails; "java.util.zip.ZipException: error in opening zip file"

Today in the Intertweets (March 11th Ed)

2010-03-12T07:11:03+00:00

Europe’s Largest Ad Targeting Platform Uses Clojure & Hadoop (here, via @ghoseb) — Clojure is used within Hadoop.
Introduction to Clojure – 14 Apr – Independence, USA (here, via @developerevents) — That’s in Independence, OH. Organized by the Cleveland Java Meetup Group.
Clojure Buch: a German Clojure Book (here, via @ajlopez) — Another book on Clojure! This time in German :) Scheduled for September 2010
Showing off @hugoduncan’s swank-clojure changes to the team. You guys are in for a treat once this stuff gets merged. (via @technomancy) — Hmmm. Please tell me more…
Clutch v0.2 with #CouchDB change notifications support is up (also on Leiningen). Updated API and syntax (here, via @ashafa)

Today in the Intertweets (March 10th Ed)

2010-03-11T07:50:37+00:00

Michael Fogus talks to RubyLearning’s #Clojure Course Participants (here, via @IndianGuru) — Includes a mug shot! (@fogus is a co-author in “The Joy of Clojure“)
Simple clojure memoization “API” w/ multiple cache strategies (was reminded of this by @kotarak’s recent article) (here, via @cemerick) — Another set of ‘memoize’ alternatives.
I’m heading to the #montreal #clojure group meeting today at 1.00pm (here, via @hugoduncan) — I know I am posting this too late, but this is to highlight that there is a Clojure User Group in Montreal: Bonjure
Joy of Clojure TOC updates (here, via @fogus) — Some chapter reorganization + new goodies.

Leiningen (unknown severity): Leiningen strangely broken

2010-03-11T00:00:00+00:00

Leiningen (unknown severity): Leiningen strangely broken

Joy of Clojure TOC Updates

2010-03-10T20:51:22+00:00

Chouser and I are working hard to get the material for the upcoming (next week, covering chapters 1-7) technical review in order. However, we hope to have an update to the MEAP available sooner rather than later. However, the http://joyofclojure.com site has been updated with the forthcoming table of contents. The changes [...] Related posts:

De-chunkifying Sequences in Clojure At the first CAP CLUG meetup I gave a presentation...
Writing the Next Clojure Book Now available for purchase! Some of you may already know...
macro"> macro"> macro'>Understanding the Clojure -> macro On page 109 of Paul Graham’s ANSI Common Lisp he...

Related posts brought to you by Yet Another Related Posts Plugin.

2010-03-10T14:09:59+00:00

I was reminiscing about my youth and loaded up an old game I wrote in high school... Pizza Delivery! Totally a blast to play you drive around to flashing locations on the minimap to deliver pizzas and have to refuel occasionally. Going off the road makes your car blow up :)

download pizza.zip

Today in the Intertweets (March 9th Ed)

2010-03-10T07:16:28+00:00

RubyLeaning.com: Clojure 101 — A New Course (here, via @devfunnel) — Do you want to learn Clojure in 4 weeks, and for free? Just for this time, this online course is free for all. Slated to start somewhere in April, and taught by @citizen428 and @ghoseb.
- Already 100 signups for the #clojure course at RubyLearning on the first day, seems like we have hit a nerve there. (via @citizen428)
- Stuart Halloway talks to RubyLearning’s #Clojure Course Participants (here, via @IndianGuru)
Visualizing Maps Using #Incanter by @nakkaya (here, via @liebke) — Using the Incanter-processing library, the author plots geographical data on a map of Turkey.
Tomorrow at 1930 we start monthly #Amsterdam #Clojure meetup. Proudly hosted by @sourcesense (via @neotyk)
Design By Contract with Clojure (here, via @emcconne_reads) — Using the :pre and :post facilities of Clojure 1.1
Context searching with clojure-opennlp (here, via @thnetos) — A follow-up article to this one in which the author introduces clojure-opennlp. In this new article, the author show how to perform more meaningful text searches by finding relevant words in the context of the searched text.
ClojureでMagicPacket送信スクリプト作ってみました (here, via @___otabat___) — Roughly translates to: “tried to write a script to send a magic packet in Clojure”. A magic packet sent to a computer will wake it up if it is asleep and the network card supports “Wake on LAN”.
#clojure memoize and the Rule of Three (here, via @kotarak) — About how to implement a version of memoize that supports different caching strategies, other than keeping all the values around forever (the behavior of cloure.core’s memoize impl). It is also a good article about structuring your code so it can be easily reused.

The Rule of Three

2010-03-09T22:12:00+00:00

As Joshua Bloch says in his talk „API Design and why it matters“:

Write multiple plug-ins before release!

If you write one, it probably won't support another.

If you write two, it will support more with difficulty.

If you write three, it will work fine.

In a recent post of mine on the Clojure group I encountered exactly this rule. So let's explore it using the example of clojure.core/memoize.

Visualizing Maps Using Incanter

2010-03-09T12:30:13+00:00

When I first saw Mathematica's WorldPlot function I was impressed, its a nice way to visualize various forms of geographical data, for some time I thought this should be a very labor intensive task, who would go around labeling each pixel. Couple of days ago I somehow ended up reading the Processing article on Wikipedia, it contains an example which shows a map of the results of the 2008 USA presidential election, turns out using Scalable Vector Graphics implementing WorldPlot functionality is extremely easy.

We will be plotting how population moved between different regions in Turkey (in to and out of a region). I am using data provided by The Turkish Statistical Institute, you can grab the map I used from Wikipedia here.

 ;; Data for 2009
 (def pop-taken [{:id 1 :name "Marmara" :population 582771}
                 {:id 2 :name "Iç Anadolu" :population 297919}
                 {:id 3 :name "Ege" :population 164896}
                 {:id 4 :name "Akdeniz" :population 188441}
                 {:id 5 :name "Karadeniz" :population 256654}
                 {:id 6 :name "Güneydoğu Anadolu" :population 171910}
                 {:id 7 :name "Doğu Anadolu" :population 214082}])

 (def pop-given [{:id 1 :name "Marmara" :population 677395}
                 {:id 2 :name "Iç Anadolu" :population 310293}
                 {:id 3 :name "Ege" :population 181459}
                 {:id 4 :name "Akdeniz" :population 193231}
                 {:id 5 :name "Karadeniz" :population 247397}
                 {:id 6 :name "Güneydoğu Anadolu" :population 118611}
                 {:id 7 :name "Doğu Anadolu" :population 148287}])

Turkey is divided in to seven geographical regions, pop-taken represents how many people moved in to that particular region and pop-given represents how many people moved out of that region during 2009.

 (defn region-color [val min max]
   (lerp-color (color 0xffd120) (color 0x920903) (norm val min max)))

In order to paint the map like a heat map, we need to assign colors using the amount of people moved in or out of a region, given a min, max and a value in between norm will normalize a value to exist between 0 and 1, lerp-color on the other hand will calculate a color between the given range using the normalized value. So our map will go from yellow to dark red depending on the people moved.

 (defn map-region-color [regions]
   (let [min (apply min (map #(:population %) regions))
         max (apply max (map #(:population %) regions))]
     (map #(vector (:id %) (region-color (:population %) min max)) regions)))

Now all we need to do is calculate min and max values in the data set, iterate over the data set and return a sequence of [id color] pairs.

 (defn sktch [regions]
   (sketch
    (setup [])
    (draw 
     []
     (let [tr-map (load-shape this "MapTurkishProvincesNumbers.svg")]
       (.shape this tr-map 0 0)
       (doseq [region (map-region-color regions)]
         (let [[id color] region
               child (.getChild tr-map (str id))]
           (.disableStyle child)
           (.fill this color)
           (.noStroke this)
           (.shape this child 0 0)
           no-loop))))))

Using incanter-processing library, we can load and access parts of the SVG map. Processing sketches are made up of the functions setup and draw, in setup as its name suggests you setup your stuff frame rate, stroke properties etc. Draw will be called once or multiple times depending on your frame rate, we load the map as a shape then paint it on the canvas, then we iterate over the data set, using getChild method of the PShape class we can access parts of the image, the map we are using has 7 children named 1 through 7 corresponding to the geographical regions of the country, we get the child then paint it using the color we calculated on to the canvas. One thing to note, sketch macro just returns a PApplet so for any function not implemented in incanter, you can access them just like any other Java function. Now lets see the results,

(view (sktch pop-given) :size [1052 744])

(view (sktch pop-taken) :size [1052 744])

Today in the Intertweets (March 8th Ed)

2010-03-09T07:36:30+00:00

Handy #Clojure snippet for apply-ing Java methods (here, via @citizen428) — A rather old snippet from Rich Hickey himself that makes it easy to apply Java methods to objects.
Whew, new blog post about my library: Natural Language Processing in #Clojure with clojure-opennlp (here, via @thnetos) — Grab a text, break it into sentences, parse the words and tag them. It’s that easy! Work in progress…
@hugoduncan ’s criterium is a pretty sweet start toward a fantastic benchmarking suite for clojure (here, via @bradfordcross) — “Criterium measures the computation time of an expression. It is designed to address some of the pitfalls of benchmarking, and benchmarking on the JVM in particular.”
Wow! Latest Linux Journal has #clojure on cover & Rich Hickey interview! (here, via @sfraser)
Clojure featured in Issue 192 of Linux Journal
Analyzing Word Frequencies with #Clojure, #Enlive and #Incanter by Ethan Fast (here, via @liebke) — Uses enlive to scrape a web page, count the words and show their frequencies graphically with Incanter.
Discussing IoC in #clojure for our #Java friends (here, via @fulldisclojure) — This episode of Full Disclojure’s video series is very focused on Java developers who are interested in Clojure. It shows how easily one can do in Clojure what is known as Inversion of Control in the Java’s world(and other imperative languages too).
another html library emerges for #clojure benchmarks; that makes 3 (here, via @wmacgyver) — Hiccup is the HTML generation library that James Reeves is extracting from Compojure. In this code snippet it shows that this library is very fast, close in speed to simple string concatenation.

Episode 14 - Inversion of Control

2010-03-09T03:48:43+00:00

In this episode I review how to relate Java's inversion of control pattern to first class functions.

Cast: Sean Devlin

Leiningen (unknown severity): AOT compilations in `lein jar' ignored by jar users

2010-03-09T00:00:00+00:00

Leiningen (unknown severity): AOT compilations in `lein jar' ignored by jar users

This weekend in the Intertweets (March 7th Ed)

2010-03-08T07:48:52+00:00

My #clojure talk at #cojug on 3/9 has been posted (here, via @wmacgyver) — If you live near Central Ohio, don’t miss this Clojure talk at the Central Ohio Java Users Group by Mac Liaw. That’s on 3/9/10.
Clojure Pre- and post-conditions: a quest for a nicer syntax (here, via @tm_interesting) — An alternative syntax for pre- and post-conditions that is much easier to the eye and cover some of the most common use cases.
Interview with Rich Hickey, creator of Clojure (here, via @tm_interesting) — Good tidbits about the future parallel features.
New ssh tool for #clojure (here, via @jclouds) — It’s a wrapper for jsch.
gen-class little brother: proxy (here, via @kotarak) — Proxies in Clojure allows you to create new java classes on the fly in order to interoperate with the Java environment, but with some limitations. This article explores the many uses and features of ‘proxy’.
A library to make Jython interop in Clojure more elegant. I created a Google Code repository for it. (here, via @rplevy) — Google Code? What’s wrong with GitHub? ;)
How to make a quick performance ping to 5 storage clouds in #clojure (here, via @jclouds) — A nice and simple example of how to use JClouds with Clojure.
What I really want it to use rspec to test my clojure libraries (here, via @kyleburton) — What is that CljHelper that he is using?…. oh, wait!
- jrclj (1.0.0): Helper library for working with Clojure from JRuby. (here, via @rubygems) — … so it is a JRuby gem…
- Call Clojure from JRuby easier than ever! by @kyleburton (here, via @jonnytran) — … and here is the source code. Case closed!

proxy – gen-class little brother

2010-03-06T20:35:00+00:00

Clojure provides two basic ways to interface with the host platform. The more comprehending is gen-class which I touched in a previous post. It's little brother is proxy. Although less powerful it is more dynamic than gen-class. Let's see how it works…

Pre- and post-conditions: a quest for a nicer syntax

2010-03-05T18:15:09+00:00

When pre- and post-conditions appeared in Clojure (since version 1.1 they are “official”), I though that was a pretty neat feature and that I ought to use it. I added conditions to a couple of functions and was satisfied. But rather soon I noticed that I used conditions very sparingly, only where I expected wrong data to be fed into a function. Considering that bugs also happen where we don’t expect them, and that conditions are great documentation as well as error-checking code, I should use them more. So why don’t I?

Something I didn’t like about pre- and post-conditions is the rather heavy syntax. For short functions, the conditions take up more space than the code itself. Moreover, parsing them visually takes some effort as well, as much as reading the code itself. Wouldn’t it be nice if preconditions could be written somehow right in the argument list, and postconditions at the level of the function definition?

Well, this is Lisp, so if you don’t like some syntax, you just roll your own. I didn’t come up with a better general syntax though, but I think that what I describe below is much nicer and suitable for 90% of pre- and post-conditions used in practice. The main limitation is that each condition can depend on only one argument, or on the return value. For the other cases, there is still Clojure’s general syntax, which is perfectly compatible with my extension. For those who want to play with this themselves, here is the code.

As a first example, here’s a pretty stupid algorithm to calculate integer powers of a number:

(defn (number?) power
  [(number?) x
   (integer?) (pos?) n]
  (apply * (repeat n x)))

The preconditions check that the first argument is a number and that the second one is a positive integer. The postcondition checks that the result is a a number – the utility of this test is a bit dubious, but it serves as an illustration. Note that you can have multiple conditions per argument, and also multiple postconditions. The full form representing the condition is constructed by inserting the argument to be tested in the second position of the supplied list. The above function definition actually expands to

(defn power
  ([x n]
   {:pre [(number? x) (integer? n) (pos? n)], :post [(number? %)]}
   (apply * (repeat n x))))

One precondition in the above example is actually too strict. The argument n needn’t be positive, just non-negative. There is not simple test function for “non-negative” in Clojure, but with the above rule we can write this as:

(defn (number?) power
  [(number?) x
   (integer?) (>= 0) n]
  (apply * (repeat n x)))

Another possibility is to use the -> macro:

(defn (number?) power
  [(number?) x
   (integer?) (-> neg? not) n]
  (apply * (repeat n x)))

Preconditions can be combined with destructuring. Here is a variant of Clojure’s function second that actually verifies that its argument has at least two element:

(defn my-second
  [[f & (seq) r]]
  (first r))

There is however one limitation: I couldn’t find a way to use my new syntax with map destructuring. So for now at least it works only with vector destructuring.

Comments on this syntax are welcome. Do you like it? Can you come up with something better? Or do you think that Clojure’s standard syntax is just fine?

WebDAV + SSL on Debian

2010-03-05T10:30:09+00:00

I was looking for a way to easily share documents between machines, since WebDAV shares can be accessed by Windows, Linux or Mac machines out of the box, I choose WebDAV over SSL. I don't use SSL for anything so WebDAV is served from DocumentRoot. I've been using it for a few days, so far it beats carrying USB sticks around.

Enable relevant Apache modules,

a2enmod ssl
a2enmod dav_fs
a2enmod dav

Create SSL certificate,

 mkdir /etc/apache2/ssl
 openssl req $@ -new -x509 -days 365 -nodes -out /etc/apache2/ssl/apache.pem \
     -keyout /etc/apache2/ssl/apache.pem
 chmod 600 /etc/apache2/ssl/apache.pem

Create your WebDAV directory and create a password file,

mkdir /path/to/webdav/
chown www-data /path/to/webdav/
htpasswd -c /path/to/passwd.dav user

Edit and add the following snippet to the configuration for the host you want to enable WebDAV,

 <VirtualHost *:443>
         ServerAdmin user@host.com
         DocumentRoot /path/to/webdav

         SSLEngine on
         SSLCertificateFile /etc/apache2/ssl/apache.pem

         <Directory /path/to/webdav/>
            DAV On
            AuthType Basic
            AuthName "webdav"
            AuthUserFile /path/to/passwd.dav
            Require valid-user
        </Directory>

         ErrorLog  /path/to/webdav/error.log
         CustomLog /path/to/webdav/access.log combined
 </VirtualHost>

Reload Apache configuration,

/etc/init.d/apache2 reload

Today in the Intertweets (March 4th Ed)

2010-03-05T08:02:25+00:00

Page layout can be performed at the decorator level with Enlive and Compojure (here, via @cgrand) — This is now you can decorate Enlive templates in Compojure.
Interesting talk on hardware for 1000+ core Java w/ Hardware Transactional Memory (here, via @stuartsierra) — This is not a new video, but still very relevant. Cliff Click, from Azul Systems discusses the challenges that they face at building the kind of very large Java systems that they build. encounter. Pretty hardcore :). You can also check out this infamous back-and-forth between Cliff Click and Rich Hickey.
conjure 0.4, a rails like framework for #clojure has just been released (here, via @wmacgyver) — The list of features of this 0.4 version is impressive!
- Hello World Tutorial – Conjure – GitHub (here, via @ajlopez)
If you’re in the Pittsburgh area on March 10th, then consider stopping by the Pittsburg Clojure Users Group. (here, via @fogus)
Clojure wont win the battle but may win the war (here, via @kicauan) — This is in fact the infamous talk “What killed Smalltalk could kill Ruby as well”, but as the article says, this could apply to Lisp/Haskell. Interesting talk, you don’t have to agree with it though.

Today in the Intertweets (March 3rd Ed)

2010-03-04T07:24:30+00:00

#Incanter blog post roundup (here, via @liebke) — A list of recent Incanter-related blog posts
Working with a java.util.HashMap in an idomatic Clojure fashion (here, via @sldfj) — A stack-overflow thread about the best (most idiomatic) way to operate on a HashMap (java) in Clojure.
I have a #clojure benchmarking library, criterium (here, via @hugoduncan)
Great Bay Area Hadoop meetup this month – Mapreduce Online + Flightcaster/Clojure (here, via @rjurney) — That’s on March 24th @ 6pm at Yahoo’s Sunnyvale’s offices.
If you’re in Seattle and interested in Clojure, the Seajure meeting is tomorrow night (here, via @technomancy) — That’s at 7pm, in Seattle, WA

liebke

2010-03-04T02:06:15+00:00

There have been several cool blog posts over the last few weeks featuring Incanter that I would like to highlight here.

The most recent post by Nurullah Akkaya demonstrates how to use Google analytics with Incanter. I’m looking forward to using this code myself.
In another excellent demonstration of log analysis, Michael Kohl shows how to Plot Amazon EC2 instance usage with Clojure and Incanter.
Jake McCrary demonstrates Plotting time series data with Incanter, using the new time-series chart that he contributed to the Incanter project.
The Commonwealth Computer Research Blog has had a series of posts featuring Incanter,
- The most recent of which is a fantastic demonstration of Incanter and generalized linear models,
- The previous posts include Monte Carlo Pi calculation,
- and Functional programming and root finding
And finally, Stuart Halloway (author of Programming Clojure) posted a screencast on Circumspec, Inferior-Lisp, Clojure, and Slime where he briefly talks about Incanter. He also previews a cool code browser that he will be using in the upcoming Pragmatic Clojure Studio that I’m excited to be attending.

Move EC2 AMIs between regions

2010-03-03T15:24:05+00:00

Today I spent some time figuring out how to move EBS-backed AMIs between different EC2 regions. To save you the effort, here’s a quick summary:

create a volume from the snapshot backing the AMI

attach this volume to a running instance and mount it

create a new volume in the target region

attach to a running instance, format and mount

in the source region, connect to your instance

rsync (over ssh, so no messing with security groups) the mounted volume to the target volume

once the rsync is done, create a snapshot in the target region

Not rocket science, but definitely not as good as just being able to share your own AMIs between regions…

Update: Robert J. Berger wrote this detailed step by step guide based on my short summary, you should go check it out!

Today in the Intertweets (March 2nd Ed)

2010-03-03T07:38:43+00:00

Fragment selectors (a new Enlive feature) are useful for webscraping too (here, via @cgrand) — Fragments are a way of creating sub-templates in Enlive. This example shows how to use them also for web scraping.
tcrayford’s clojure-refactoring at master (here, via @sldfjd) — Library for Emacs that provides some automated refactorings.
Clojure just passed CL to become the #19 on the github high score chart (here, via @IORayne)

Clojure is num.19 at GitHub

Google analytics with #Clojure and #Incanter (here, via @liebke) — Another illustrative article on how to create instant applications with Clojure from nakkaya.com: Pull data from Google Analytics and then process it and show it in nice graphs with Incanter.
Next week (3/11) is “Happy Tasty Clojure” from Kurt Christensen (here, via @jasonbock) — If you are near the Twin Cities by 3/11, there is a talk on Clojure hosted by the Twin Cities Languages User Group

Analytics with Incanter

2010-03-02T16:00:14+00:00

These past few days I've been playing with Incanter, which is a Clojure-based, R-like platform for statistical computing and graphics. This post covers the basic steps of using Clojure to access your Google Analytics Data with the Google Analytics Data Export API and visualize, filter the data returned using Incanter.

Google provides a Java library to simplify use of any Google Data API with Java, to access Analytics you need to grab the following list of Jars from gdata-java-client and google-collections.

gdata-client-1.0.jar
gdata-client-meta-1.0.jar
gdata-core-1.0.jar
gdata-analytics-2.1.jar
gdata-analytics-meta-2.1.jar
google-collect-1.0.jar

After running "lein deps" add them to the lib/ subdirectory.

 (defn service [username pass]
   (doto (AnalyticsService. "Clojure_Incanter_Sample")
     (.setUserCredentials username pass)))

In order to retrieve data we need a service object which handles all interaction between our application and Analytics Data Export API.

 (defn account-feed [service & args]
   (let [url (URL. (str "https://www.google.com/analytics/"
                        "feeds/accounts/default?max-results=50"))
         feed (.getFeed service url (get-class "AccountFeed"))
         accs (reduce #(assoc %1 
                         (-> %2 .getTitle .getPlainText)
                         (-> %2 .getTableId .getValue)) 
                      {} (.getEntries feed))]
     (if (nil? args) accs (accs (first args)))))

To retrieve data for a profile, we need its table id. Asking service for an account feed, returns a list of entries containing title, table id and profile id but we are only interested in title and table id.

 (defn data-feed [service & args]
   (let [args (apply hash-map args)
         feed (.getFeed service (.getUrl (query args)) (get-class "DataFeed"))
         cols (map #(str "ga" %) (concat (:dimensions args) (:metrics args)))]
     (map (fn [e]
            (map #(.stringValueOf e %) cols))
          (.getEntries feed))))

As with the account feed, first thing we need to do is build a feed request URL, query function handles that nothing fancy, it just calls a bunch of setters for dimensions, metrics etc. Querying analytics service with a data feed URL returns a list of entries, data-feed maps over them and returns a sequence containing dimensions and metrics we requested.

Now that we have some data to play with, we can start off by doing fairly standard things, like which pages got the most visits for the past month,

   (def analytics (service "username" "password"))
   (def acc-nakkaya (account-feed analytics "nakkaya.com"))

   (def pageview (data-feed analytics 
                            :date ["2010-01-26" "2010-02-25"]
                            :dimensions [:pageTitle :pagePath]
                            :metrics [:pageviews]
                            :sort [:pageviews]
                            :num-result 10
                            :id acc-nakkaya))

This is where incanter makes things fun, as long as you have a sequence of rows, in this case what data-feed returns you can call view to visualize the data,

(view pageview)

or we can filter the data leaving only portions of it which we are interested, such as pages with views more than 200 and lower than 800,

   (with-data (col-names (map (fn [[x y z]] [x y (BigInteger. z)]) pageview)
                         [:title :path :views])
     (view ($where {:views {:$gt 200 :$lt 800}})))

Alternatively you can filter the data in Clojure, requesting top 10 keywords people used to find your website and filtering the ones that contain "clojure" or "java" in them,

   (def keywords (data-feed analytics 
                            :date ["2010-01-26" "2010-02-25"]
                            :dimensions [:keyword]
                            :metrics [:visits]
                            :sort [:visits]
                            :num-result 10
                            :id acc-nakkaya))

    (let [words ["clojure" "java"]] 
      (reduce (fn[h v]
                (if (some true? (map #(.contains (first v) %) words))
                  (conj h v) h)) [] keywords))

 analytics.core=> [("clojure xml" "62") ("clojure turtle graphics" "31")
                   ("clojure opencv" "26") ("detect faces from webcam+java" "26")]

Besides visualizing stuff using tables, we can plot graphs containing the information we are intrested,

   (def browsers (data-feed analytics 
                            :date ["2010-01-26"]
                            :dimensions [:browser]
                            :metrics [:visits]
                            :sort [:visits]
                            :num-result 10
                            :id acc-nakkaya))

   (view (bar-chart (take 4 (map first browsers))
                    (take 4 (map #(BigInteger. (last %)) browsers))
                    :title "Browser/Visits"
                    :x-label "Browsers"
                    :y-label "Visits"))

   (def view-date (data-feed analytics 
                             :date ["2009-11-26" "2010-02-25"]
                             :dimensions [:date]
                             :metrics [:visitors]
                             :sort [:visitors]
                             :num-result 10
                             :id acc-nakkaya))

   (view (line-chart (map first view-date)
                     (map #(BigInteger. (last %)) view-date)
                     :title "Visits"
                     :x-label "Date"
                     :y-label "Visits"))

analytics.clj

Today in the Intertweets (March 1st Ed)

2010-03-02T07:58:04+00:00

Published (finally) a Clojure app to get twitts from your account and push them to MySQL database (here, via @b0j3) — An interesting (brave) way of this author to learn clojure, by building a twitter app and using a database. Whatever happened to SICP? The interesting thing for me is that the author is using yet another library to access twitter? How many are there?
Again, another #clojure #wallpaper done in GIMP (no PS installed :/) (here, via @dsp_) — Don’t click that link yet… read below first.
- Turns out that #clojure logo shouldn’t be used in derivate works, deleting the wallpapers from twitpoic. (via @dsp_) — Awwwwww!
Look at my Software Transactional Memory implementation for Cocoa/Obj-C. Based on Clojure’s. Still experimental (here, via @aramallo) — Rich Hickey is a trend-setter.
Webjureなんてフレームワークもあるのか (here, via @bojovs) — Webjure is yet another Clojure-based simple web framework.
#circumspec #inferior-lisp #clojure and #slime (video) (here, via @stuarthalloway) — This video is awesome in many ways. Shows a mini-browser that lets you use a web-browser to navigate your code dynamically. Shows a cool emacs configuration. Finally, it shows how Circumspec looks today. And it looks awesome :). Watch. It.
Conjure 0.4がリリースされたらしい (here, via @bojovs) — Conjure (a Rails-like web framework for Clojure) 0.4 has been released.
Would be really interested in seeing feedback on this musing about 43556b; padding: 0px; margin: 0px;" title="#clojure" rel="nofollow" href="http://twitter.com/search?q=%23clojure">#clojureand dependency injection (here, via @puredanger) — How to do dependency injection in Clojure (or is it needed at all?)
Are you ready for a Bay Area Clojure Bootcamp? Cast your vote now! (here, via @disclojure) — Amit Rathore, author of “Clojure in Action” and founder of the BA-CUG is asking “How much would you pay to attend a day long Clojure boot-camp?”. Go ahead and vote!
Vote up, if you want to see how to use tools like #ant #clojure and #hadoop in the cloud (here, via @jclouds) — Another pool: if you leave in or around San Francisco, you like Clojure and know that there are more clouds than the ones in the sky, you might want to vote for JClouds to present at the SFJUG (and then attend, of course!)
Episode 13 is up – inferring information (here, via @fulldisclojure) — This episode from @fulldisclojure is about writing an inference engine in clojure. Pretty intense.
Clojure + Emacs (here, via @mdelaurentis) — A compilation of ways to interact with the Clojure REPL from Emacs

Episode 13 - Inference

2010-03-02T02:21:27+00:00

In this episode I review how to write an inference engine in Clojure.

Show notes:
github.com/francoisdevlin/Full-Disclojure/blob/master/src/episode_013/episode_013.clj

Cast: Sean Devlin

Leiningen (unknown severity): "lein test" broken for Clojure 1.0.0 projects

2010-03-02T00:00:00+00:00

Leiningen (unknown severity): "lein test" broken for Clojure 1.0.0 projects

Enclojure (unknown severity): Problem in the classpath string

2010-03-02T00:00:00+00:00

Enclojure (unknown severity): Problem in the classpath string

Leiningen (unknown severity): enhancement - Test classification

2010-03-02T00:00:00+00:00

Leiningen (unknown severity): enhancement - Test classification

Anonymous function literals and syntax-quote don’t work well together

2010-03-01T10:43:05+00:00

Clojure has a couple of macro-like features built right into the reader, providing shorthand notation for commonly needed constructs. However, unlike real macros, which are built on a sound conceptual framework that guarantees arbitrary composability (macro calls can contain macros and expand into other macros), the different macro-like features in the reader don’t always play well together.

Consider the following piece of code:

(defmacro foo [x]
  `(map #(identity %) [~x]))

At first sight, nothing looks wrong with this, other than it doesn’t do anything useful. But any use of this macro causes an error message:

(foo [:a :b])
java.lang.Exception: Can't use qualified name as parameter: user/p1__3328

What’s going on here? The error message hints at a problem with a function argument. The only function being defined is here #(identity %), which uses a shorthand notation for function literals expanded by the reader. Let’s see what the macro call expands to:

(macroexpand-1 '(foo [:a :b]))

yields

(clojure.core/map (fn* [user/p1__3328] (clojure.core/identity user/p1__3328)) [[:a :b]])

So here’s the problem: #(identity %) is expanded by the reader into (fn* [p1__3328] (identity p1__3328)). This is a perfectly valid function literal, but the other reader feature used here, syntax-quote, doesn’t know about function literals. It takes the expanded function literal as an arbitrary form and does namespace resolution on all symbols. This leads to a namespace-qualified symbol for a function parameter, which is not legal Clojure syntax.

Moral: use reader features sparingly, ideally one at a time. Except for syntax-quote, they only save you a couple of keystrokes, a convenience that you may end up paying a high price for in terms of debugging time.

This weekend in the Intertweets (Feb 28th Ed)

2010-03-01T08:20:32+00:00

Very simple #clojure wallpaper (here, via @dsp_) — A nice wallpaper for clojurians
- Another one, this time in 1280×800 (here, via @dsp_)
clojure-maven-plugin now uses (run-tests) and not (run-all-tests), and defaults to including test deps for run/repl/nailgun/swank. (via @talios)
Clojure serialization benchmark testing printer, JSON, and binary (here, via @mmcgrana) — Mark is running some benchmark on the different forms of serialization of clojure data structures. His own clj-json wins hands down. The micro-benchmark framework that he uses seems pretty generic too.
I made an attempt at uniform JVM benchmarking, please help make it better (here, via @laujensen) — Speaking of micro-benchmarks, Lau has written an article trying to settle on a set of guidelines for running micro-benchmarks for the JVM.
Wrote my experience with compojure’s session down (here, via @dsp_) — How to create a simple web session with loging/logout using Compojure’s own session capabilities.
A simple web application in Clojure using ring and enlive (here, via @lshift) — Ring and Enlive together create a very barebones but powerful web framework. This article shows how to write a web app with them both.
See #clujure syntax brains curled up into a tube (here, via @ignart) — (don’t blame me; this is an automated translation from Russian by Google Translate) This is a fairly long and complete article introducing clojure to Russian readers by @alexott_en.

Getting benchmarking right

2010-02-28T15:30:56+00:00

Several times on this blog, I’ve dealt with issues where some kind of benchmarking was required. The method, implementation, environment all play a role and has subsequently been the object of much discussion. In this post let’s see if we can agree on some way of benchmarking.

Benchmarking is 4D

In my oppinion there are 4 dimensions to benchmarking which we need to deal with separately. In industry benchmarking can be crual to architecting your solution correctly and/or dealing with performance issues. Industry benchmarking needs to come as close to the real scenario as possible. If you’re working on a massive scale, benchmarking on a laptop is a no go, if your setup is heavy on I/O you need to clone your production environment in order to produce reliable results, etc etc.

The second type of benchmarking is what is typical for bloggers and I’m no exception: Microbenchmarking. Microbenchmarking is where we take certain routines out and benchmark them individually, my recent Fibonnaci posts are examples of this. Like the commentators pointed out, there needs to be an emphasis on equality where possible. A Ruby solution won’t look like a Clojure solution, but differences like returning or printing the result should be eliminated and defining the specific area to benchmark is very important: Are we timing the function or the inner-loop?

This second form of benchmarking is in divided into 2 distinct categories as well (bringing us up to the 4D benchmarking), because we have many languages which put the JVM to good effect with all the added benefits, but there are also languages that dont. Benchmarking on the JVM requires a good knowledge of both the JVM and HotSpot, whereas benchmarking outside of the JVM requires a good knowledge of whichever compiler you are working with. In that regard I couldn’t do Haskell any favors, for not having a clue about how the compiled code runs, start up times etc?

JVM Benchmarking

Being able to benchmark on the JVM is important and requires some background. I recommend reading this post in particular.

The unix time function is ruled out for the obvious reasons that 1) We dont want to be affected by the startup time of the JVM and 2) We dont need it. Especially when we’re looking at single algorithm/loop etc, what we really want to know is how well that given body of code performs. Its interaction with the rest of our system can be disregarded for the sake of comparison and since we’re often dealing with very small numbers, subtle differences quickly become not so subtle. Before we can get into the actual benchmarking we need to boot the JVM.

What we dont care about

For the sake of blogging/microbenchmarking we’re typically not doing any major optimizations, because usually idioms are being compared. There are exceptions, but we can deal with them once we get there. The JVM is a fantastic eco-system which allows for rigorious introspection — all of which we can disgard for our simple exercises.

Things we can disregard:

JVM Parameters:

–XX:+PrintCompilation — This gives us a heads-up whenever a method is compiled.

–verbose:GC — Lets us keep tabs on when the GC is running

–XX:-PrintGC — Print messages at GC

–XX:-PrintGCDetails — More verbose GC messages

–XX:-TraceClassLoading — Trace the loading of classes

–agentlib:hprof[=options] — Heap/CPU — read this: options

All of these (and dim sum) can be very helpful in tracing performance bottlenecks, but for microbenchmarking they seem overkill. Depending on how much allocation we’re doing we might trigger some heavy GC, but in that case we should account be stripping high/low values from our timings.

The things we cannot disregard:

JVM Paramters:

–Xms128M — The minimum amount of memory the JVM allocation

–Xmx512M — The maximum amount

–server/-client — Depending on which we choose for a given task, you actually get different compilers

We should strive to keep the values identical because depending on the task they can make all the difference. The minimum amount is allocation at the JVM boot, meaning no time will be spent growing the memory space while the program is running if you don’t exceed this. Thought I haven’t checked Jarkko Oranen suggested that the preformance degradation which comes by approaching the Xmx value is due to the GC working double time to free up memory, therefore its important to set the max to an appropriate value — though it varies from test to test we should strive to keep these values identical.

Methodology

I don’t think there’s a set standard for how we benchmark when doing blog comparisons, so allow me to introduce and outline for how we can handle comparisons done on this blog:

Do multiple passes: The code being benched should be executed repeatedly a given number of times, at default for smaller algorithms could be 20 passes.

Because we eventually hit GCs or other burps of the system, multiple passes ensure that we get the overall picture.

Garbage collect: Dont heap it up between runs, but clean after every pass.

This is the simplest way in which you can tame the GC and help get uniform results.

Filter highest and lowest values: Take the highest and lowest value and remove them from the timings.

When timing on the JVM some of the bumps are very significant — You can have a series of runs going at 5ms and then suddenly a pass that takes 40ms — Since its more a reflection of the disturbance than the actual algorithm, it can safely be stripped.

Warm up the JVM: To trigger all the optimizations etc, the main loop should be repeated a number of times

Since we’re blogging, I recommend just going for either 1 pass of the algorithm, or repeated passes of the algorithm until we’ve have been crunching for 1 minute. So if you algorithm takes 30 seconds for 1 pass, it will run twice, if it takes 2 minutes per pass, it will run once.

Implementation

To make this fair I’ll provide a benchmark macro in Clojure and if you are also the happy user of some other JVM language which you want included in future benchmarks on this site, please provide me with a similar function/macro, which I will then both post here and use in future comparisons.

(defmacro microbench
  " Evaluates the expression n number of times, returning the average
    time spent in computation, removing highest and lowest values.

    If the body of expr returns nil, only the timing is returned otherwise
    the result is printed - does not affect timing.

    Before timings begin, a warmup is performed lasting either 1 minute or
    1 full computational cycle, depending on which comes first."
  [n expr] {:pre [(> n 2)]}
  `(let [warm-up#  (let [start# (System/currentTimeMillis)]
                     (println "Warming up!")
                     (while (< (System/currentTimeMillis) (+ start# (* 60 1000)))
                            (with-out-str ~expr)
                            (System/gc))
                     (println "Benchmarking..."))
         timings#  (doall
                    (for [pass# (range ~n)]
                      (let [start#    (System/nanoTime)
                            retr#     ~expr
                            timing#   (/ (double (- (System/nanoTime) start#))
                                         1000000.0)]
                        (when retr# (println retr#))
                        (System/gc)
                        timing#)))
         runtime#  (reduce + timings#)
         highest#  (apply max timings#)
         lowest#   (apply min timings#)]
     (println "Total runtime: " runtime#)
     (println "Highest time : " highest#)
     (println "Lowest time  : " lowest#)
     (println "Average      : " (/ (- runtime# (+ highest# lowest#))
                                   (- (count timings#) 2)))
     timings#))

The code is effectively divded into 3 sections:

Warm-up: A while loop runs for at least 1 minute, trapping all output so we dont see any printing from whatever we’re benchmarking.

Timings: The actually expression is repeatedly run and the milliseconds spent in each pass is returned

Stats: Highest/Lowest values are filtered, results are printed

Using this to test our Fibonacci code from my previous post, then becomes:

user> (microbench 20
              (let [limit (.pow (BigInteger/TEN) 999)]
                   (loop [a 0 b 1 i 1]
                      (if (< b limit)
                          (recur b (+ a b) (inc i))
                          nil))))
Warming up!
Benchmarking...
Total runtime:  110.08787899999999
Highest time :  7.98691
Lowest time  :  5.135988
Average      :  5.386943388888889

Notice the final ‘nil’ in the code. If I had returned ‘i’ instead it would have printed the result of each pass before printing the final stats — It would not affect the timing. Or if we used the more idiomatic (?) version:

user> (microbench 20
                  (let [limit (.pow (BigInteger/TEN) 999)]
                       (count (take-while #(< % limit) fib-seq))))
Warming up!
Benchmarking...
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
4782
Total runtime:  74.73764200000001
Highest time :  5.031017
Lowest time  :  3.168419
Average      :  3.696567

Of course in that case its cheating, because the fib-seq is calculated once and then kept in memory.

Conclusion

Benchmarks can be fun, but I think its important to 1) Agree on some methodology and 2) Not go nuts over a few milliseconds here and there. Certain algorithms perform better than others, and certain algorithms can be more or less idiomatically expressed in various languages. Ultimately when looking at the results we have to apply some common sense as we’re not always doing 1:1 comparisons. Using these benchmarks to definitely positively declare one language superior to the other should neither be our goal nor is it possible.

If you’ve taken a look at Alex Osbornes attempt at the WideFinder 2 challenge, you saw how he plowed through 45 Gigabytes of text in a blazing 8m 4s blowing both Scala and Java out of the water. Does that then conclusively state that Clojure is faster than both those languages? Of course not, Clojure compiles to bytecode, just like Scala and Java so the exact same speed could be obtained by both of them. The point is, that the concise and elegant Clojure code can be as powerful than other languages without the same level of incidental complexity being put on the user, so it makes sense to focus on the quality of the code while keeping half an eye on the performance. If all that mattered was speed, we’d all still be writing ASM.

This is my proposal for a uniform way to benchmark in the future — let me know what you think, I’m only happy to accept changes, improvements, implementations in other languages etc. And JVM outsiders shouldn’t feel left out.

Tokyo-Cabinet-Clj (unknown severity): documentation?

2010-02-28T00:00:00+00:00

Tokyo-Cabinet-Clj (unknown severity): documentation?

Today in the Intertweets (Feb 25th Ed)

2010-02-26T07:45:49+00:00

This ought to be good, Cap-Clug meeting on 3/17 with @richhickey and @stuarthalloway (here, via @mattpodwysocki) — If you are going to be anywhere between 0 and 1000 miles from Reston (VA) on March 17th, you should definitely plan to attend.
My #clojure concurrency talk from Øredev is online (here, via @stuarthalloway) — This talk is an introduction to Clojure and its concurrency features. If you can’t go to the meet-up above, you can always watch Stuart Halloway’s presentation :)
rosejn’s midi-clj, a high level midi library to play notes, external midi devices in clojure (here, via @ajlopez) — this will *definitely* be tested this weekend on my Virus Classic!
rosejn’s osc-clj, an Open Sound Control library for Clojure (here, via @ajlopez)
Talk on “Emerging Languages” in Palo Alto, March 10th (here, via @hamiltonulmer) — Amit Rathore will be introducing Clojure in that talk. Clojure, Go, Scala and Ruby will be presented.
#clojure + #chef + #cloud = pallet; great stuff @hugoduncan! (here, via @jclouds) — Pallet is used to start provision compute nodes using crane, jclouds and chef. It is designed for use from the REPL, along with Clojure scripts.
A short preview of ClojureQL (here, via @kotarak) — ClojureQL is undergoing a lot of changes in the front-end. Here is a sneak preview of how the new front-end will look like.
A Resettable Memoize in Clojure (here, via @pjlegato) — Since you cannot reset a memoized function with the ‘memoize’ function in Clojure’s core API (yet), you can always create your own resetteable memoize. Here’s how.
Wrote a blog on using the with-command-line option parser in Clojure (here, via @puredanger) — If you need to parse command-line options, know that ‘there is a clojure lib for that’… with almost no documentation. Well, “there is a blog post for that too”!

Penumbra (unknown severity): Cannot run examples

2010-02-26T00:00:00+00:00

Penumbra (unknown severity): Cannot run examples

Plot EC2 instance usage with Clojure and Incanter

2010-02-25T23:46:29+00:00

Have you ever wondered which instance types you use most when dealing with Amazon’s EC2? I have, so I came up with a short Clojure/Incanter program to plot a nice chart like this:

(full size image)

Here’s the code, if you have any feedback please leave a comment, this was the first time I used Incanter.

A preview of ClojureQL

2010-02-25T22:31:00+00:00

As you might already know, I'm rewriting ClojureQL. This takes a lot of time at the moment, so I don't find much time to blog. But to not stall the blog completely and to shorten the time a little until the first version of CQL is released I decided to provide a little example on the look'n'feel of the new frontend. So here we go…

New issue of Russian Journal of Functional Programming

2010-02-25T17:07:01+00:00

Today the new issue of Russian journal of functional programming was released. In this issue my article about Clojure was published. This article should be first big article about Clojure in Russian. Article describes Clojure from scratch and should be useful for Lisp & Java programmers. It describes Clojure syntax, basic constructs, Java interop & concurrent programming.
P.S. If you interested, you can read article via Google Translate...

Comprehensive list of cloud-related links

2010-02-25T10:34:55+00:00

Lately a lot of my working time is spent on cloud technologies, most notably the Amazon Web Services. In recent month I compiled a rather comprehensive list of cloud-related links which I now decided to share with you all.

Cloud Links

It’s just a markdown file copied into a Gist, so if you want to add any links just fork away!

Today in the Intertweets (Feb 24th Ed)

2010-02-25T08:06:16+00:00

Groovy is the #21 most popular language on GitHub. Clojure is the #20 most popular language on GitHub. Glad I switched. (via @foogoof)
ClojureでCompojureを使わずにサーブレットを書く方法を書きました (here, via @tnoborio) — For those who don’t read Japanese, this roughly translates to: “How to write a servlet in Clojure without Compojure” (Thanks @chosuke for the help with the translation!)
New chapter four – Polymorphism with Multimethods – added to the Clojure in Action MEAP – get it here (via @amitrathore)
Chinarro The Robot eats XMPP in #Clojure (here, via @ajlopez) — This robots connects to a MUC jabber room, and a #hashtagged message is posted, the robot stores it for you.
Montréal #clojure luncheon on March 10th. Come join us! New users welcome – I’m one of them (here, via @jperras)
Dean Wampler on Programming Languages – Scala, Clojure, functional paradigm (here, via @vkelman) — Dean Wampler is co-author of Programming Scala and owner/principal of Aspect Research Associates. Touches on Clojure tangentially, but interesting nonetheless.

Writing Leiningen Plugins 101

2010-02-25T04:00:18+00:00

I'm trying to switch, building my projects using Ant to leiningen. Almost all of them requires customs tasks such as building native executables, move files around etc. Which requires I have to come up with a lein plugin for each ant task, unfortunately not much documentation exists about writing lein plugins, this post collects bits and pieces of information I gathered over the web.

To begin with lein tasks are functions named "your-task" defined in the namespace "leiningen.your-task". They take a project argument containing information defined in defproject and command-line arguments. For simple tasks or quickly testing something, you can simply define them in project.clj after the defproject definition,

 (ns leiningen.foo)
 (defn foo [project & args] (println "Hello Foo!!"))

Now lein should have a new task named foo, running it should print "Hello Foo!!". Of course for longer tasks, you are not going to want it cluttering your project.clj, they can be placed under leiningen/ folder not src/leiningen/. Since tasks are just functions, making a task depend on another task is as easy as calling depencies on top of the function.

 ;;leiningen/bar.clj
 (ns leiningen.bar)

 (defn bar [projects & args] 
   (apply leiningen.foo/foo projects args)
   (println "Hello Bar!!"))

Now running bar task should give you, "Hello Foo!!" and "Hello Bar!!". For sharing plugins across projects create a separate lein project for the plugin, after creating a Jar with "lein jar" you have two options, you can either push it to clojars and add your plugin as a dev-dependency for your project or just move the jar to the lib folder of the project, either way lein will pick it up.

Haskell,Ruby,Scala,Clojure — Tweaked!

2010-02-24T20:17:11+00:00

Last night I did very small superficial comparison, of 3 ways of getting the first fibonnaci number consisting of 1000 digits. This attracted a lot of attention from Rubists, Haskaloonies and Clojurians alike, here I’ll share their contributions.

Preface

Microbenchmarking sucks, especially on the JVM. HotSpot is doing such a great job of optimizing code, that sometimes you end up getting an incorrect idea of how your code performs. Nevertheless I got, many tips for improving performance, so it seems the Internet Performance Rule of Thumb is: Always benchmark the Fibs!

Haskell

Team Haskell weren’t pleased completing the job in 7 msecs, so they suggested that I recompile the program using –O2. I used GHC v. 6.10:

limit = 10^999
fibs = 0 : 1 : zipWith (+) fibs (tail fibs)
main = print . length . takeWhile (< limit) $ fibs

(thanks to: “laulau”)

Real time, average of 20: 7 msecs.
Nothing gained, but 7 msecs is blazing.

Ruby

Ruby had big room for improvement as the very clever Matrix approach didn’t yield very satisfying performance. Here’s one commentators approach

limit = 10**999
def fib2(a,b)
 return b,b+a
end
a,b = fib2(0,1)
while b < limit
 a,b = fib2(a,b)
end
puts b

(thanks to: Aaron)

Average on 20 runs: 35 msecs.

Huge improvement and honor is restored to the Ruby camp!

Scala

Camp Scala was good enough to contribute a solution as well:

object Main {
  def main(args: Array[String]) {
    def f(a: BigInt, b: BigInt) = (b, b + a)
    val max = BigInt(10).pow(999)
    var (a, b) = f(0, 1)
    while(b < max) {
      val temp = f(a, b)
      a = temp._1
      b = temp._2
    }
    println("\nb = " + b)
  }
}

(thanks to: Rahul G ‚*updated*)

Average: 18 ms.

Clojure

Clojures loops are recursive and side-effect free, but even still it looks a little clunky:

(time (let [limit (.pow (BigInteger/TEN) 999)]
        (loop [a 0 b 1 i 1]
          (if (< b limit)
            (recur b (+ a b) (inc i))
            i))))

(thanks to: Duc)

Nevertheless, it averages at: 5 msecs. Hooray! It’s a new record!

Conclusion

If nothing else I think its fun to see these challenges approached from different languages, which is also what I like about Project Euler. And I’m certain that this will not be the last time I bring Haskell along for the ride. Last nights post were meant as small reflections on 3 very different languages, so I was a little surprised of the attention that it got — But its positive, I think its great when we all can learn a little from one another.

I’m open for ideas on another arena for comparisons. We could do something crazy like solve Global Warming.. oh wait, I already did that. How about Euler 200 then?

Today in the Intertweets (Feb 23rd Ed)

2010-02-24T07:07:55+00:00

Generic equation solver & macros (video here, via @fulldisclojure) — A new episode of the weekly series by Sean Devlin that expands on macro creation (he did an episode on simpler macros a couple of weeks ago). He’s got some nice macro-fu going on there!
Analyzing JVM performance is hard …but Clojure hash-maps really are fast (here, via @chrishouser) — That’s from Clojure’s email list; someone posts a micro-benchmark comparing the performance hash-maps in Clojure and in Haskell. The conversation that follows shows once again that micro-benchmarks are almost always flawed and that, oh well, it is really hard to make fair comparisons.
Clojure job market flys past Java’s and Python’s!!!1!!1! (here, via @cemerick) — In RELATIVE terms, that is!
I find this chart of Scala/Clojure/Erlang/Haskell/F#/Lisp jobs interesting (here, via @puredanger) — And Clojure is the one that has grown the fastest in the last 6 mo., again in relative terms. In absolute terms, it now seems to be on par with F# and getting close to Haskell and Erlang.
A simple web interface to mongodb written in clojure (here, via @sethtrain)
Graph Reasoner for Clojure (here, via @ml_bot) — Graph reasoning is a way to do inference over a relationship graph.
Generating deftype forms in macros (here, via @jneira) — When trying to write macros that generate deftypes, you will find that the standard behavior of syntax-quote might not work too well because it tries to resolve unqualified symbols to their namespace-qualified equivalents. Int this article, an alternative for syntax-quote is proposed for things like deftype.
Clojure, Haskell & Ruby Vs Euler 25 (here, via @fulldisclojure) — Speaking of micr0-benchmarks, here is a new article from Lau Jensen comparing Clojure, Scala and Haskell. Needless to day this micro-benchmark raises some discussion. His results? Clojure has the most compact implementation and Haskell is absolutely fast.

mwmeyer

2010-02-23T19:59:14+00:00

In order to keep better track of what iPhone users are missing out on, I’ve set up a page listing Android applications – or at least categories of them – that Apple doesn’t allow on the iPhone. You can find it under Pages on the right hand side of any page on the blog, listed as Apps you won’t find in the app store.

Clojure, Haskell & Ruby Vs Euler 25

2010-02-23T18:53:47+00:00

Haskell is a mature, statically typed, functional language which was recently compared to Ruby in an attempt to solve Euler #25. In this post I’ll share the code, the benchmark and add a Clojure version for those interested.

Preface

Normally it’s bad to start with the disclaimer, but for the sake of all you angry young men with ill tempers I’ll say it anyway: Similar to the Clojure Vs Ruby & Scala post this is in no way an exhaustive comparison between the languages, its intended at a superficial glance at a simple challenge, namely Euler #25, all in the name of good clean fun. Haskell is by far too vast, too impressive and too powerful to be fully dealt with in a simple exercise like this, and the Ruby code is still running :)

Euler #25

Create a Fibonnaci sequence and walk through it, until you find the first number which is 1000 digits. The Fibonnacci sequence is exceedingly simple. Its seeded with 0 1 and all consecutive values n, are generated by calculating n-1 + n-2.

0,1,1,2,3,5,8,13,21,34,55,89,144,233,377,610,987,1597,2584,4181.….

Solutions

All 3 versions are compact and powerful — I’m assuming they’re idiomatic, but if I’m wrong let me know. Like I said — If you can produce a better version, please submit it instead of taking this as an invitation to a flamewar :)

Ruby

Ruby goes like this:

require 'matrix'

limit = 10**999
FIBONACCI_MATRIX = Matrix[[1,1],[1,0]]
def fibonacci(n)
  (FIBONACCI_MATRIX**(n-1)) [0,0]
end
i = 1
i+=1 while fibonacci(i) < limit
puts i

The real trick here is building the Fibonnacci seq (fib-seq), which is done by applying the Matrix form, which is a clever way to leverage some differential equations in order to produce the sequence. According to the author this runs faster than a regular memoized variant.

Haskell

Haskell code looks very different from both Ruby and Clojure, but for a simple task like this is reads almost like plain English:

limit = 10^999
fibonacci_numbers = 0:1:(zipWith (+)
                        fibonacci_numbers (tail fibonacci_numbers))

index = length w where w = takeWhile (< limit) fibonacci_numbers

main = do
  putStrLn(show(index))

First the limit is defined as the first number having 1000 digits, and then a sequence seeded with 0 and 1. When you have a sequence which requires a constant look-behind of n-values, it can be represented as a recursive set of vectors. I call that a rule of thumb, Christophe says its a theorem. One thing which is really great about the Haskell code, is the automatic currying which enables the author to do the elegant w = takeWhile (< limit). Although this example is not flaunting it, Haskell is very different from Clojure in that its statically typed and there’s been many discussions back and forth on which is to prefer, dynamic or static typing. The truth is, no one answer fits all but there’s a simple test:

If the arrow and the explanation actually helped you, then static typing is for you.

Clojure

To produce the fib-seq, we can mimic Haskell almost point for point, but where Haskell has direct support for sequences in the syntax, I need to call a few functions:

(def fib-seq (lazy-cat [0 1] (map + fib-seq (rest fib-seq))))

(let [limit (.pow (BigInteger/TEN) 999)]
           (count (take-while #(< % limit) fib-seq)))

The noticeable exterior differences is the call to lazy-cat and the anonymous function, which all readers of this blog should be able to read without any explanation. Under the hood, Haskell is fully lazy where Clojure is now Chunky-lazy. Chunked-seqs mean that I work with the sequences as if they were fully lazy, but under the hood Clojure is realizing the sequence, one chunk at a time. If I take 50 values, 82 may in fact be calculated — this is for improved performance without wrecking the use of infinite sequences. According to one of the authors of this book (which looks extremely promising btw, so go look), Rich Hickey has provided an example which eliminates the need for chunked seqs: here. That is idiomatic Clojure running faster than our old seq implementation written primarily in Java.

Results

So the usual indicators for expressiveness and performance are LOC and runtime speed, so here we go:

Language	LOC	Runtime
Clojure	3	0.021s
Haskell	6	0.007s
Ruby	9	7.1s

Last time I benchmarked Ruby it also came in… a little late, and some commentators suggested trying out other Ruby compilers — The truth is however, that small benchmarks like this really show very little and if you are working in a performance critical zone, you’ll want to test something which mimics the actual project and the actual environment. Secondarily, neither Haskell nor Clojure are optimized for performance which accounts for some of the extra Ruby code.

The lesson we can take away, is that Haskell and Clojure are blazing, and Haskell is blazing to the point where I’m not finding words good enough to describe it.

Generating deftype forms in macros

2010-02-23T14:36:39+00:00

One of the common uses of macros in Clojure, as in other Lisp dialects, is to abstract away boilerplate code. Instead of writing very similar lengthy forms several times, one defines a macro that specializes a template for each particular use, and then uses that macro a few times. The template is usually written using syntax-quote: the template form is preceded by a backquote, and inside the template a tilde marks expressions that are replaced by their values.

Syntax-quote has one more effect: it resolves all symbols in the current namespace (the one in which the macro is defined, not the one where it is used) and replaces the unqualified symbol by its namespace-qualified equivalent. For most symbols in most forms, this is the right thing to do in order to make the macro work in any namespace, as well as to avoid unwanted variable capture. More specifically, it is the right thing to do for symbols that are defined by the macro, and for symbols that will ultimately be evaluated (names referring to vars, in particular function names). It is not the right thing to do for symbols bound locally inside the form (function parameter names, symbols bound in a let form). And it is also not the right thing to do for symbols that just stand for themselves and are used in some special way by the form that the macro expands to.

The latter situation is particularly frequent in macros that generate deftype forms. Consider for example the following deftype form, which is a simplified version of the type definition used in my multiarray design study:

(deftype multiarray

  [descriptor
   data-array]
  :as this

  Object
    (equals [o] ...)
    (hashCode [] ...)

   clojure.lang.Counted
     (count [] ...)

   clojure.lang.Indexed
     (nth [i] ..)

   clojure.lang.Sequential

   clojure.lang.Seqable
     (seq [] ...))

Of all the symbols shown in the above example, the only one for which namespace-resolution is appropriate is multiarray, the name of the type being defined. All the other symbols name fields of the type, Java interfaces, or methods. They must remain unqualified. In real-life deftypes, there are of course symbols that could or should be namespace-qualified, in particular most of the symbols used inside the method definitions, which are just like function definitions. However, method definitions are often short, and rarely subject to variable capture, meaning that not namespace-resolving those symbols is rarely a problem.

In a syntax-quote template, there are two ways to deal with symbols for which the default (namespace-resolution) is not appropriate:

Prefixing with ~' (tilde + quote). This is a special case of an expression inside a template, whose value is the quoted symbol. A tilde-quoted symbol is taken over into the instantiated template without namespace-resolution.
Postfixing with # (hash sign). Such symbols are replaced with system-generated symbols that are guaranteed to be different from any other symbol in existence. This is another technique to avoid variable capture.

For generating a deftype form from a syntax-quote template, the only solution is thus to prefix all the symbols shown in the example above with tilde-quote. I tried: it works, but it’s a mess. It’s not very readable, and the inevitable mistakes lead to unpleasant error messages.

Well, this is Lisp, and in Lisp you are always free to make your own tools if you are not happy with the ones provided by the system. What I want here is a template expansion system that doesn’t do namespace resolution on symbols. However, I didn’t need a full-blown equivalent to syntax-quote templates either, given that I would use those deftype templates only for one application. So I came up with the following definitions, which for me are the right compromise between simplicity and useability:

(defn instantiate-template
  [substitution-map form]
  (clojure.walk/prewalk
   (fn [x] (if (and (sequential? x) (= (first x) 'clojure.core/unquote))
	     (substitution-map (second x))
	     x))
   form))

(defmacro template
  [substitutions form]
  (let [substitution-map (into {} (map (fn [[a b]]
					 [(list 'quote a) b])
				       (partition 2 substitutions)))]
    `(instantiate-template ~substitution-map (quote ~form))))

Compared to syntax-quote, this has two restrictions: it has no splicing, and it admits only symbols after a tilde, not arbitrary expressions. The template macro takes a let-like vector as its first argument. This vector contains the symbol-value pairs for substitution inside the template. The second argument is the template form, which presumably contains tilde-prefixed symbols for substitution. Note that the Clojure reader translates ~x to <code (clojure.core/unquote x)

, which is what the above code searches for.

Here is an example for using such templates:

(defmacro foo [typename fieldname]
  (template [type  typename
	     field fieldname]
    (deftype ~type
      [~field])))

(foo bar boo)

(bar 42)

This prints #:bar{:boo 2}, illustrating that the macros does what it is expected to do. Of course this is not the perfect example for the utility of my little template instantiation system, as it could just as well be written using syntax-quote!

Episode 12 - Equations

2010-02-23T07:28:57+00:00

In this episode I demonstrate how to use macros & Newton's method to numerically solve any equation for any variable.

The code is available here
github.com/francoisdevlin/Full-Disclojure/blob/master/src/episode_012/episode_012.clj

Cast: Sean Devlin

Today in the Intertweets (Feb 22nd Ed)

2010-02-23T07:13:01+00:00

My new clojure project is now up http://github.com/tashafa/nozzle Let me know what you think (via @ashafa) — Nozzle is a Clojure library for streaming statuses from the “Twitter Streaming API.”
pushed full rewrite of cascading-clojure to master this morning. co-authors @mmcgrana @nathanmarz (here, via @bradfordcross)
New chapter 4 for Clojure in Action: Polymorphism with Multimethods (here, via @ManningBooks)
Third draft of new #Clojure testing framework (here, via @stuartsierra) — This time with documentation, so you don’t have to read all the code to figure out how it works ;)
clojureql redesigned (here, via @wmacgyver) — ClojureQL, a clojure library for interacting with SQL databases is up for a re-design of it’s query DSL. This article explains what the changes are. The end result seems much cleaner.

mwmeyer

2010-02-23T01:05:59+00:00

A recent interview with yags* consisted of solving problems and writing demo code for them, which is pretty standard for a gs. One of the problems captured my attention, because it looked like a perfect problem to explore the strengths and weaknesses of clojure.

The problem is simple enough to state: Given a set of lists of words, remove every phrase group that occurs in all the lists. A phrase group is any sequence of three or more words in a row. Two things make this interesting for clojure: 1) it’s a purely functional problem: the output result depends only on the input values, and 2) it involves manipulating relatively deep structures – at least in the solution I came up with. That is something working with immutable data structures typically makes painful.

Since I’m still exploring clojure, I chose to do this in Python in the interview. The code here – and in the repository at bitbucket – uses the same algorithm, expressed with the high-level data structures of clojure instead of Python.

First, a couple of short helper functions:

(defn mfilter
  "Return a hash-map built by removing entries for which (pred (key entry))
  returns false from mapin."
  [pred mapin]
  (apply hash-map (apply concat (filter #(pred (key %)) mapin))))

(defn enumerate
  "Return pairs of an index into sequence, and the value at that index"
  [sequence]
  (map vector (iterate inc 0) sequence))

As the document string for mfilter says, it returns a copy of a hash-map built from a map by removing entries for which the predicate applied to the key is false. Likewise, enumerate counts the elements in a sequence, starting at 0, and returns pairs of them and the counter.

Note that we only have to worry about phrases of length three. A phrase of length 4 will show up as two phrases of length three with an overlap, and one of length five as three such phrases, all overlapping. So we can ignore phrases longer than three words. And of course, phrases shorter than three words aren’t phrase groups, so we ignore them as well. So the list of phrases in an input list is a list triples that has two fewer elements than the input list, as the last two elements don’t start phrases.

The data structure used for my solution is a dictionary, index by phrase groups – meaning triples of words. Each entry is also a dictionary, indexed by an input lists index in the list of input lists. The entries in those dictionaries is a list of places that that phrase group starts in the input list. I’m going to call this the phrase dictionary.

That leads us to the first building block function, which accepts a phrase dictionary and a phrase, along with the index of a list and the index of that phrase in the list, and returns the update phrase dictionary:

(defn add-phrase-to-phrase-dict [so-far phrase list-index phrase-index]
  (if (or (so-far phrase) (= list-index 0))
    (update-in so-far [phrase list-index] conj phrase-index)
    so-far))

This uses the clojure function update-in, which is something I don’t ever remember seeing in a lisp before (though cl’s setf might be close). It finds an element in a nested structure – like the phrase dictionary – using it’s first argument, which is a sequence of indices into the structure. That’s [phrase list-index] , so this finds the list of places where phrase appears in the input list list-index. Further, if the elements aren’t there, it creates the intermediate map needed, and return nils if the last lookup fails. The resulting value gets passed to the second argument, along with any remaining arguments, and the result of that call is used in this position in the new version of the structure that is returned. While this might sound expensive, since everything in the structure is immutable, the two versions can actually share everything but the values along the path to the updated value.

The conditional application of update-in is an optimization. If the phrase we’re updating isn’t in the dictionary, we want two different behaviors: if this is the first input list, we want to add that phrase to the dictionary. Otherwise, we can ignore the phrase, because it isn’t in at least one input list – the first one. Adding it won’t change the final result, but will generate more work. So we check for it there, and then only add it if this is the first input list. The alternative case returns the input phrase dictionary unmodified.

We now need to invoke this function on every phrase in every input list. Let’s start with a function to invoke it on every phrase in a single input list:

(defn add-list-to-phrase-dict [phrase-dict phrase list-index phrase-index]
  (reduce (fn [phrase-dict [phrase-index phrase]]
	    (add-phrase-to-phrase-dict phrase-dict phrase list-index phrase-index))
	  phrase-dict
	  (enumerate (map vector list (rest list) (rest (rest list))))))

As input, we get the phrase dictionary we’re going to update, the index of the input list in the list of lists, and the input list itself. Wanting to update a value based on processing values in a list is what the lisp function reduce is for. It takes a function, an initial value, and a list, then calls that function with the initial value and each element in the list. So our function – introduced by (fn – takes two arguments, a phrase dict and a pair of phrase-index and phrase, and returns the result of calling add-phrase-to-phrase-dict with them and the input list values. The initial value is the input phrase dictionary. We generate the phrase list by mapping vector over the input list, and the results of removing the first and then second element from it, giving three-element vectors of three consecutive words – which are our phrases. Again, the primitive does the right thing for us, and stops producing maps when the last list runs out, so the last two words in the initial input list don’t start phrases. We pass that list to enumerate to get the index and phrase pairs we need.

Given that, we basically repeat the same construct to deal with all the input lists:

(defn build-phrase-dict [lists]
  (let [phrase-dict (reduce (fn [phrase-dict [list-index list]]
			      (add-list-to-phrase-dict phrase-dict
						       list-index list))
			    {}
			    (enumerate lists))
	list-count (count lists)]
    (mfilter #(= (count (val %)) list-count) phrase-dict)))

The input is a sequence of lists. That’s the sequence we reduce over here, once again passing it to enumerate to generate our indices. This time, the initial value is an empty map. The most important change is that, instead of returning the result of reduce directly, we use the mfilter function defined earlier to remove any elements that don’t have as many entries as we had input lists. That’s the criteria for inclusion – that the phrase be in every list. If it isn’t in some list, then it’s dictionary entry won’t have an entry for that list, and hence the count won’t match the length of the list of lists.

That’s the first half the problem – building up the phrase dictionary. We now need to use that to remove the phrases from the input lists. Oddly enough, the phrases themselves don’t matter – we just care about their positions. So we write a function that takes the list of phrase positions, and removes them from a list:

(defn remove-phrases-from-list [phrase-starts list]
  (mfilter (fn [[x _]] (not-any? #(and (>= x %) (< x (+ % 3))) phrase-starts))
	   (apply sorted-map (mapcat vector (iterate inc 0) list))))

Here's the second mfilter – this time, it’s remove words from the list. We start by creating a sorted map indexed by word position of the words in the list. Then mfilter uses not-any? to check each word’s position against the phrase start positions, specifically that the position is not greater than the start position and less than three plus the start posistion. If a position falls into that range for any phrase in the phrase-starts list, it’s removed.

So all that’s left is printing the result:

(defn remove-shared-phrases [lists]
  (let [phrases (apply merge-with concat (vals (build-phrase-dict lists)))]
    (doseq [[idx list] (enumerate lists)]
      (println (vals (remove-phrases-from-list (phrases idx) list))))))

This actually combines everything so far: building the phrase dictionary from the list, extracting the list of phrase positions from that, using merge-with to concat the lists in each of the element dictionaries into a single list for each file, then using doseq to call println on the result of removing the phrases in each list from the list. At this point, we see why the output map was a sorted-map: that forces the results of each list to print in the proper order.

All in all, this has been an illuminating exercise. Clojure’s functions for dealing with deep structures and non-list structures work very well with them, providing what is still a very Lisp-like environment, even though we’re not working with lists. In particular, the handling of what in python were special cases is right by default, which feels very much like classic lisp behavior. I’d say Clojure held up very well here.

*) That would be Yet Another Google Spinoff.

Redis-Clojure (unknown severity): authentication doesn't work

2010-02-23T00:00:00+00:00

Redis-Clojure (unknown severity): authentication doesn't work

Futures and promises in Clojure (asynchronicity and concurrency, part 2)

2010-02-22T22:42:33+00:00

A few weeks ago I posted about my initial impressions of CommonJS’s approach to concurrency, in particular the use of promises. Today I add my look at promises and futures in Clojure.

I’m aware that I’m kind of writing these blog posts backwards. I’m learning more as I go, which means before I’m done I will no doubt regret things I have written in these opening posts. Oh well.

Clojure offers more than one option for achieving concurrency. Whereas CommonJS embraces asynchronous processes, Clojure offers the programmer a choice of synchronized or asynchronized tools for concurrency. And whereas JavaScript doesn’t offer you the opportunity to spawn your own threads, Clojure most definitely offers you threads. Clojure 1.1 has added abstractions for promises and futures, and Clojure’s multithreaded environment colors their usage.

In Clojure, as in CommonJS, a promise is also a stand-in for an asynchronously-computed value. As Rich Hickey explains in his change notes for the 1.1 release:

A promise is a synchronization construct that can be used to deliver a value from one thread to another. Until the value has been delivered, any attempt to dereference the promise will block.

A future, however, represents a unit of computation that can be executed on another thread.

Futures represent asynchronous computations. They are a way to get code to run in another thread, and obtain the result.

Sean Devlin, the creator of the excellent screencasts at Full Disclojure, has a screencast covering promises and futures. He offers some examples:

(def a-promise (promise)) (deliver a-promise :fred)

(def f (future (some-sexp))) (deref f) ; blocks the thread that derefs f until value is available

There’s a lot of commentary out there, especially on this thread from the Clojure Google group in which different use cases for the abstractions are offered. Here are some highlights:

Laurent Petit offers this explanation of the virtues of futures and promises:

It’s an easy way to make a computation done in thread A (and using a pre-declared promise) block until thread B has delivered the promise (given it its value).

Sean Devlin adds that they are useful in dealing with mechanical objects. Instead of a while-loop to wait for an external condition to be satisfied, you can block on the value of a promise while a separate process devoted to detecting this desired condition delivers a value to the promise when and if the desired condition is met.

Finally, the venerable Chouser offers another example of blocking: setting a promise and then making an RPC call that delivers its result to the promise when it is complete.

Some general background: Dataflow concurrency

Rich Hickey, in his Clojure 1.1 release notes, and Laurent Petit, on the Clojure mailing list, point out that promises and futures support dataflow programming. Dataflow programming is laid out in some detail in a 900-page textbook. There are also less complete explanations of it online.

The wikipedia entry for dataflow concurrency is predictably terse and dense.

A dataflow network is a network of concurrently executing processes or automata that can communicate by sending data over channels (see message passing.)

I found the description of dataflow concurrency in Groovy by the folks at Codehaus very illuminating. Basically, they define three tasks, each of which determines a value for one of three DataFlowVariables x, y, and z.

These run in parallel, and, it’s implied, the first task is blocked until a value for z is available. Once that’s accomplished, the println line executes.

In Clojure, then, would x, y, and z be promises, and the tasks be closures sent to agents?

Yep.

So this is a nice way to parallelize computation. One thread can hold the final piece of the computation and wait for others to deliver values to the promise(s) it needs. These other threads can similarly wait on other ones. This seems to replace the use of await for agents, at least in some cases.

Strangely, I get this example better than the ones that just sleep threads and print “here I am” messages. Have I become math-biased? If so, I blame SICP.

Next up: promises and futures in wordy languages

I started out talking about promises in JavaScript. Next time, I hope to say a few things about the uses of futures in C# and Java to accomplish concurrent tasks. These concepts are clearly important for anyone who wants to understand concurrency beyond specific implementations and recipes in one language or another.

Redesigning ClojureQL's Frontend

2010-02-22T17:02:56+00:00

I've not updated this blog in a while as I've been pretty busy for thepast few weeks. I've got many projects on the table and one of them isstarting to get really interesting. I'm obviously talking about thesubject of this post,ClojureQL. I've mainly beenworking on the backend since the end of last year. In the meantime,Meikel has started areworkof the frontend to provide a cleaner API, free from the magicalartifacts introduced by using macros like in the current (0.9.7)version. This rework have been triggered by apostcarefully explaining the issue, courtesy of Zef.

I won't go further than a linkfest so here are the important links forthose interested in the future of ClojureQL:

We're also ready to receive your suggestions on the brand newclojureql group. Please,visit the Wiki pages and tell us about what you think.

This weekend in the Intertweets (Feb 21st Ed)

2010-02-22T08:32:04+00:00

Overtone is a #Clojure based musical generation and manipulation system for live-coding and more (here, via @8quashes) — Wraps Super Collider
Looking forward to the #clojure namespace management overhaul (here, via @RickMoynihan) — There are a few proposals out there to tidy up a little bit the namespace management. :use is considered harmful as it imports by default all the symbols in the imported package. Here are discussed some of the proposed solutions.
Software Transactional Memory in JRuby using Cloby (here, via @sai_venkat) — This is about using Cloby (by @headius, of JRuby fame) that allows using Clojure’s STM in Ruby.
Byte-spec: a declarative DSL for reading and writing binary file formats in #Clojure (here, via @8quashes) — Load and parse binary files the functional way.
2nd draft of new #Clojure test framework (here, via @stuartsierra) — Stuart Sierra is working on a new testing framework for clojure. This new version is targeted at Clojure 1.2 as it uses some if its new features.
Writeup about my #clojure RPG so far (with video) (here, via @BrianCarper) — Great post about Brian’s experiences in writing this RPG game in Clojure. It covers concurrency, OpenGL and making things move smoothly. If there is anything that summarizes this post for me is the following: “Clojure is a mostly functional language, in the sense of strongly discouraging unnecessary use of mutable state, and this program is no different”
Applescript with Clojure (here, via @bubbl) — Title says it all. Now you can control your iTunes from Clojure (that is, if you own a Mac)
Some reflections on my first few weeks of heavy Clojure usage (here, via @puredanger) — Looks like we have another convert :) Main point? Can go from thoughts to code without having to create boilerplate code.
creating a simple game in #clojure (here, via @ztellman) — Yet another game, this time using Penumbra to create the OpenGL code. The article is from the author of Penumbra himself. This article describes the design of this Asteroids game in Clojure. A very nice read!
Notes on upcoming #compojure 0.4 (here, via @wmacgyver) — Summary: More reliance on Ring for all things HTTP (some features have been crossported from Compojure to Ring), some of the libraries (HTML, routing) have been pulled from Compojure into their own libraries, and a change of syntax in how Servlet data is obtained.
join me… in the *future* (starting to build the pieces for a Clojure enterprise CMS) (here, via @acfoltzer) — A wrapper to JCR-283: Java Content Repository.

liebke

2010-02-22T00:55:14+00:00

It’s been a long time coming, but Incanter is available in a public code repository (http://repo.incanter.org) once again.

The version available in the Clojars repository has grown increasingly out of date, but couldn’t be updated due to Incanter’s new modular build structure; a structure that lets developers include only the subset of Incanter functionality that they need.

For instance, if you only need the functionality found in incanter.core and incanter.stats, then include the incanter-core dependency in your project.clj file. If you need charts but, not Processing visualizations, include incanter-charts. If you want access to the incanter.chrono library, but nothing else, use the incanter-chrono dependency.

Instructions for building Leiningen-based projects using the repository are available on the Incanter repo’s homepage, http://repo.incanter.org.

Plotting time series data with Incanter

2010-02-22T00:33:30+00:00

Recently I found myself wanting to plot some time series data and wanted to do this in Clojure. Unfortunately Incanter, a good statistical and graphics library for Clojure, did not provide a way to plot data where the x-axis is a time value. A quick fork on github and a pull request later and now Incanter does. Since I added this functionality I thought I would write up a short example of using it.

The example time series data I’m using I took from Yahoo’s finance section. At the bottom of that page there is a link to download the table as a csv file. That is the data used in this example.

I’m using the read-dataset function provided by Incanter. This procedure reads a delimited file (or URL) and returns an Incanter dataset.

(def yhoo (read-dataset "table.csv" :header true))

Yahoo stores the date in a yyyy-mm-dd format. I need to convert that to milliseconds from the epoch so it can be used in time-series-plot as the x-axis data. To do this I wrote a function which takes the string representation of the date, splits in on “-”, then use the joda-date and to-ms functions from incanter.chrono to get the number of milliseconds from the epoch.

(defn to-milliseconds-from-epoch [x]
   (to-ms
    (apply joda-date (map #(Integer/parseInt %)
                          (.split x "-")))))

Now that we have a function which takes the string representation and get the milliseconds it is time to get the data I want from the dataset. The below code selects the :Close and :Date column while mapping the :Date column to a millisecond from epoch representation of date.

(def mod-data
     (col-names
      (conj-cols
       ($ :Close yhoo)
       ($map to-milliseconds-from-epoch :Date yhoo))
     [:Close :Date]))

The next step is to use the time-series-plot function to actually create the plot. Because the data we have is in a dataset, we can pass in the column names as the x and y parameters and provide the data set as the value to the :data key in the optional parameters.

(def chart (time-series-plot :Date :Close
                             :x-label "Date"
                             :y-label "Closing Price"
                             :title "Closing price over time for Yahoo"
                             :data mod-data))

Then we use the Incanter function view to actually see the chart.

(view chart)

Major changes in ClojureX

2010-02-21T16:21:00+00:00

When ClojureX was started it tried to be everything for everybody. Not only did it checkout and compile the git HEADs of clojure, clojure-contrib and jline, it also (optionally) downloaded packages for Clojure support in TextMate or Emacs, configured and installed them. This has two obvious drawbacks. First off, people end up downloading stuff they don’t need. With current disk sizes and connection speeds that’s probably not much of a problem, but some users may still find annoying. More importantly though it was hard to keep up with the upstream changes, e.g. clojure-contrib’s recent switch to Maven.

So now instead of trying to provide a one-size-fits-all solution I’m taking a totally different approach: from now on ClojureX will only contain the most recent stable releases of clojure, clojure-contrib and jline, as well as a very full-featured clj script:

I know that this potentially makes ClojureX less useful to some people, but I do believe that it’s one of the fastest and most painless ways to set up a Clojure development environment.

Making an RPG in Clojure (part one of many?)

2010-02-21T00:27:40+00:00

What do you get when you combine old-school Final Fantasy-style RPGs with Clojure? Fun times for all. Well, for me at least.

I'm working on a sort of RPG engine in Clojure so I can make my own RPG. Click the thumbnail for a very preliminary video demo (6.5 MB) showing the engine in action.

All I do in the video is walk around, and eventually start adding random NPCs to the map to test collision detection. Not all that exciting, but I'm proud nonetheless.

To forestall questions, yes I'll eventually post the source code, but no, not yet. It barely works. Just a proof of concept so far.

Right now I can walk around a map while NPCs also randomly walk around the map, not much more. So there isn't much to talk about. But not bad for 4 days and 600 lines of code (one tenth of which is ASCII art... more on that later). Keep in mind that I have no idea what I'm doing.

Collision detection works so people don't walk through walls or each other, and after endless tweaking I got all the animations to be very smooth, even when I add a few dozen NPCs to the map (as I do in the video). The video is a bit jerky but it looks better in person. All of the admittedly poor artwork is also created by myself, thanks to the GIMP and some hastily-read tutorials on pixel art.

It all runs on plain old Swing in Clojure. Here's some of what went right and what went wrong so far.

Background

I've played a lot of RPGs, but I've never programmed a game more complex than Pong. I knew what double-buffering is, and that's as far as my knowledge of game programming went when I started.

My first idea was to use a plain old Swing JFrame. I'd make a bunch of sprites saved as PNG files, read them all in and draw them on the JFrame, as many times per second as I could manage. Then there's some global state to keep track of where everything is. Simple enough.

(PS I have no idea what I'm doing.)

Failures

My first version used Clojure agents (i.e. threads) for everything. The game logic was a thread, the renderer ran in a thread, every NPC was its own thread. The world itself was a single ref that all of these agents banged on. So the NPCs would tell the ref "I want to move down one square", another thread might say "Brian just pushed 'left' on the keyboard, so start scrolling the map". The world-ref would kindly oblige while preventing two NPCs from standing on each other or letting the PC walk through the walls.

Clojure is awesome in letting you do this in a completely safe and coordinated way. Everything worked well. But even for the crappy 2D graphics I'm using, all those threads caused way too much lag. I could get a good framerate if everyone was standing still, but if I was walking while the NPCs were walking, I'd get lots of lag. It was even worse on a slower computer.

My best guess is that the reason for the lag was the constant restarting of canceled transactions due to multiple threads trying to edit the world ref 50+ times per second. My sucky 2-core CPU couldn't keep up. It isn't surprising that this failed, in hindsight.

OpenGL?

Next, I decided to try out OpenGL. There are multiple options for OpenGL in Java. One is JOGL and another is lwjgl.

I program in Linux, and my video card is ancient. I can barely get OpenGL to work in the best of times. Installing JOGL was a slight chore (Gentoo doesn't even include it in its repo). JOGL is not just a JAR you throw onto CLASSPATH, you need some native extensions, obviously. I got it running somehow, but I wouldn't want to explain to someone else how I did it.

I did get jwjgl to work too, eventually, which was nice. There is a really nice Java 2D game framework called Slick which uses lwjgl. Some good games were created using this, for example Stickvania, which recently hit Reddit recently. I got Slick up and running in short order.

Unfortunately Slick doesn't play nicely with a Clojure REPL. I could build and start a game, but once the game is stopped, it never runs properly again without restarting the REPL. Slick caches images to try to be speedy, and it seems like the cache is either corrupted or destroyed when you close down your game, for one thing. This is not conducive to Clojure REPL-style development, and I didn't want to spend a lot of time fixing it.

Another issue is that I'd really like this game to be cross-platform and available to non-hackers, and the thought of trying to tell the average gamer how to install JOGL or lwjgl was daunting, given the bullcrap I had to go through. Not sure if I can just throw JOGL into a JAR and distribute it, maybe I can, but I didn't want to bother reading about it. Swing on the other hand runs everywhere with no effort.

My main problem is that I know even less about OpenGL programming than I do about Swing, and don't have a month to learn. Back to the drawing board.

Success

It turns out I don't need OpenGL anyways. All I need is program more intelligently. (Did I mention I have no idea what I'm doing?) Instead of dozens of threads, my current (working) version has one thread. It updates the game logic, then renders the game, then waits 10 milliseconds or so, then repeats this (forever).

With the single-threaded version, the logic is actually more complex than the multi-threaded version. Agents and refs were really nice and braindead-easy to work with. But such is life.

Once I learned about keeping track of things in realtime by counting milliseconds instead of counting frames or using timeouts to do logic/render updates, things worked better. (Did I mention I have no idea what I'm doing?) The game loop looks like this now:

(defn game-loop [#^Canvas canvas]
  (loop [last-time (get-time)]
    (let [curr-time (get-time)
          delta (- curr-time last-time)]
      (do-logic delta)
      (do-render canvas)
      (when @RUNNING
        (Thread/sleep 10)
        (recur curr-time)))))

The code to actually start the game, to give you an idea:

(defn start-world [world]
  (let [#^JFrame frame (doto (JFrame. "Game")
                         (.addWindowListener (proxy [WindowAdapter] []
                                               (windowClosing [e] (stop)))))
        #^JPanel panel (doto (.getContentPane frame)
                         (.setPreferredSize (Dimension. REAL-WIDTH REAL-HEIGHT))
                         (.setLayout nil))
        #^Canvas canvas (Canvas.)]
    (doto canvas
      (.setBounds 0 0 REAL-WIDTH REAL-HEIGHT)
      (.setIgnoreRepaint true)
      (.addKeyListener (proxy [KeyAdapter] []
                         (keyPressed [e] (handle-keypress e))))
      (.addMouseListener (proxy [MouseAdapter] []
                           (mouseClicked [e] (handle-mouse e))))
      )
    (.add panel canvas)
    (doto frame
      (.pack)
      (.setResizable false)
      (.setVisible true))
    (.createBufferStrategy canvas 2)
    (dosync (ref-set RUNNING true)
            (ref-set PAINTER (agent canvas))
            (ref-set WORLD world))
    (send-off @PAINTER game-loop)))

That's about it for the Swing side of things, aside from scribbling on the Canvas in the render function.

The agent I wrap around the Canvas controls the thread that runs the game loop. The agent helpfully keeps track of any exceptions that happen during the loop, and I can view those exceptions via agent-error, which is handy for debugging.

Note how little code it is to set up a keyboard event handler, thanks to proxy:

    (doto canvas
      ...
      (.addKeyListener (proxy [KeyAdapter] []
                         (keyPressed [e] (handle-keypress e))))

A couple lines of Clojure for what would be a lot of senseless boilerplate in Java. Notice how you the keyboard handler calls a normal Clojure function handle-keypress. Clojure / Java interop really is seamless.

Maps

My maps are made using ASCII art. (Did I mention I have no id-... never mind.) Here's the code for the test map, for example. Map here is a deftype (available in bleeding-edge Clojure), which takes a map of tiles, a "pad" tile, the map, and then a mask showing walls / tiles where the player shouldn't be allowed to walk. My Map type acts like a Clojure hash-map most of the time, but it also lets me name the fields that all maps should share, and has a proper "type", among other things.

(deftype Map [tileset pad tiles walls]
  clojure.lang.IPersistentMap)

(def test-map (cache-map
               (Map {\  (tile "ground")
                     \/ (tile "ground-shadow-botright")
                     \< (tile "ground-shadow-left")
                     \d (tile "dirt")
                     \| (tile "wall_vertical")
                     \- (tile "wall_horizontal")
                     \1 (tile "wall_topleft")
                     \2 (tile "wall_topright")
                     \3 (tile "wall_bottomleft")
                     \4 (tile "wall_bottomright")
                     \u (tile "below-wall")
                     \v (tile "below-wall-shadow")
                     \w (tile "wood-floor")
                     \c (tile "cobble")
                     \b (tile "bush")}

                    (tile "ground")

                    ["                 1----2                            "
                     "                 |vuuu|                            "
                     "1----------------4<bbb3--------------------------2 "
                     "|vuuuuuuuuuuuuuuuu/   uuuuuuuuuuuuuuuuuuuuuuuuuuu|<"
                     "|<1------------2               cccccccc d        |<"
                     "|<|vuuuuuuuuuuu|<       d      c      c          |<"
                     "|<|<           |< cccccccccccccc    d ccccccc d  |<"
                     "|<3------ -----4< c                         ccccc|<"
                     "|<uuuuuuu/uuuuuu/ c  1-------2    1------------2 |<"
                     "|<1---2 bcb       c  |vuuuuuu|<   |vuuuuuuuuuuu|<|<"
                     "|<|vuu|< c  d     c  |< 1-- -4<   |<           |<|<"
                     "|<3- -4<bcb       c  |< |vucuu/   3------ -----4<|<"
                     "|<uu/uu/ c        c  3--4< c      uuuuuuu/uuuuuu/|<"
                     "|<  c  dbcb       c  uuuu/ cccc         bcb      |<"
                     "|<  cccccccccccccccccccccccc  cccccccc   c   d   |<"
                     "|<  d        d                     d ccccc       |<"
                     "|<                    1--------------------------4<"
                     "3---------------------4uuuuuuuuuuuuuuuuuuuuuuuuuuu/"
                     "uuuuuuuuuuuuuuuuuuuuuuu/                           "]

                    ["                 xxxxxx                            "
                     "                 x    x                            "
                     "xxxxxxxxxxxxxxxxxx xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx "
                     "x                                                x "
                     "x xxxxxxxxxxxxxx                                 x "
                     "x x            x                                 x "
                     "x x            x                                 x "
                     "x xxxxxxx xxxxxx                                 x "
                     "x                    xxxxxxxxx    xxxxxxxxxxxxxx x "
                     "x xxxxx x x          x       x    x            x x "
                     "x x   x              x  xxx xx    x            x x "
                     "x xx xx x x          x  x         xxxxxxx xxxxxx x "
                     "x                    xxxx                        x "
                     "x       x x                             x x      x "
                     "x                                                x "
                     "x                                                x "
                     "x                     xxxxxxxxxxxxxxxxxxxxxxxxxxxx "
                     "xxxxxxxxxxxxxxxxxxxxxxx                            "]
                    )))

This lets me make all of my tiles be simple PNG files with sane names. I could dork around with sprite sheets, but why bother? Emacs or Vim column-editing and overwrite modes make it easy enough to make a map this way. Swing thankfully handles transparency for me if I use PNGs, so it's a no-brainer to make multiple map layers later, which I'll need later. The code for reading in PNGs is brain-dead simple thanks to Java's ImageIO:

(defn tile [name]
  (ImageIO/read (File. (str "img/" name ".png"))))

The cache-map function (below) iterates over the ASCII art (via seqs on the Strings), using each character as a key into the map of real images, and builds a BufferedImage out of it. This is cached, since the renderer needs to re-draw the background every frame. In this function I'm also drawing a "padding" layer to sit under the background, so that I see endless fields of grass when I walk close to the edge of the map, instead of garbled, smeared-out graphical artifacts.

(defn cache-map [amap]
  (let [height (count (:tiles amap))
        width (count (first (:tiles amap)))

        #^BufferedImage img (BufferedImage. (tile-to-real width) (tile-to-real height) BufferedImage/TYPE_INT_ARGB)
        #^Graphics2D g (.createGraphics img)

        #^BufferedImage pad (BufferedImage. (tile-to-real width) (tile-to-real height) BufferedImage/TYPE_INT_ARGB)
        #^Graphics2D padg (.createGraphics pad)]
    (doseq [[y row] (cseq/indexed (:tiles amap))
            [x tile] (cseq/indexed row)]
      (.drawImage padg (:pad amap) (tile-to-real x) (tile-to-real y) REAL-TILE-SIZE REAL-TILE-SIZE nil)
      (if-let [#^Image img ((:tileset amap) tile)]
        (.drawImage g img (tile-to-real x) (tile-to-real y) REAL-TILE-SIZE REAL-TILE-SIZE nil)
        (throw (Exception. (str "Missing tile " tile ".")))))
    (assoc amap
      :map-image img
      :pad-image pad
      :max-map-x width
      :max-map-y height)))

This code is nasty, mostly due to converting between tile-based coordinates and pixel-based coordinates. This code needs to be cleaned up a bit. But that's about the most complex code you'll find in my program so far.

Who needs mutable state?

Clojure is a mostly functional language, in the sense of strongly discouraging unnecessary use of mutable state, and this program is no different. I'm sometimes amazed how far I can get before I need mutable state at all. The vast majority of my functions take a world value (a plain old hash-map) as an argument, and return a new world value after making changes to it. The current state of the world is whatever value is currently in the global WORLD ref.

The render loop grabs a snapshot of the world from the ref on each iteration, and then draws it. Thanks to Clojure refs, the snapshot of the world is guaranteed to be consistent (e.g. no NPC objects in the middle of mutating themselves) and persistent (the world value sticks around as long as the renderer needs it, even if the WORLD ref is changing in another thread). Once it's been drawn, the renderer throws the world snapshot away and it's garbage-collected later.

This all happens around 50-100 times per second in my game, and there's no noticeable lag. So that's a good thing.

Conclusion

That's about it. I'll post the full source code once it doesn't suck as much.

In my opinion, exploring a new area of study is some of the best fun a person can have. It's a flood of information and a surprise every 10 seconds.

Making a game has been like that. There's a huge wealth of knowledge about this kind of programming that I never knew existed. Everything I read on this topic is new to me and fascinating. The saddest(?) part of this whole thing is that I'm going to have more fun programming this game than I usually have playing games I buy in the store.

Penumbra (unknown severity): targeting clojure.contrib-1.2.0

2010-02-21T00:00:00+00:00

Penumbra (unknown severity): targeting clojure.contrib-1.2.0