Manipulate source code like the dom

I've always believed that most programming problems are expression problems. Being able to say what it is that we want usually gets us 95% of the way to solving the problem. The rest usually takes care of itself. Libraries and tools should help us express ourselves through higher paradigms of thinking, where the most powerful features are declarative paradigms that allow us to just say what we want and for the library to figure out how to get us there.

Source code manipulation has always been a great source of difficulty. Most source code manipulation programs are really just built upon regular expressions with some parsing and then lots of string manipulation. Lisp code, having a more regular shape tends to be easier to manipulate, but I haven't really seen any nice tools for directly dealing with source code.

Having said that, I'm extremely excited to show off a new library for source code manipulation based upon principles that lisp code is in essence a huge tree. The library is called jai and is inspired by css/xpath/jquery. I've been working on and off on this concept for about a year but it came together in the past month, having had some time off to polish off the fine grain control and the placement of the zipper at the exact location that I want it to be. The traversal code alone took 3 tries to get right. Tree-walking is super hard and I now have a real appreciation for rewrite-clj, without which, this library would have been impossible.

As lisp code follows a tree-like structure, it is very useful to have a simple language to be able to query as well as update elements of that tree. The best tool for source code manipulation is rewrite-clj. However, it is hard to reason about the higher level concepts of code when using just a zipper for traversal. jai is essentially a query/manipulation tool inspired by jquery and css selectors that make for easy dom manipulation and query. Instead of writing the following code with rewrite-clj:

(use ' :as z)

(if (and (-> zloc z/prev z/prev z/sexpr (= "defn"))
         (-> zloc z/prev z/sexpr vector?)
    (do-something zloc)

jai allows the same logic to be written in a much more expressive manner:

(use 'jai.query)

($ zloc [(defn ^:% vector? | _)] do-something)

More examples can be seen in the documentation

There have been many forerunners for of thinking about source code as data; tangible data that we can control, reason about and manipulate as we see fit. The first for me was codeq, the second being rewrite-clj. The fact that we can take source code and change it structurally with so much ease is the reason why I find the lisp paradigm so incredible. There is no way to do this in any other language bar html/xml (hence the css/xpath inspired syntax).

Another paradigm that I've been playing with is that the query should take the same shape as the actual thing that we are querying for. This is nothing revolutionary; in fact, it's kind of common-sensical. We are seeing this with mongodb, graphql but to be honest, we should be using it everywhere. We saw a huge uptake in mongodb because people saw how easy it was to create applications quickly due to the fact that there was no mental overhead of using sql. So with jai, I wanted it to feel as intuitive as possible. It makes heavy use of core.match to do some cool pattern matching behind the scenes. In the past, the trade-off between speed and expressiveness meant that being declarative can lead to tremendous losses in performance but now, it matters less and less. Of course there may be exceptions to this rule but in general we as programmers/toolmakers should act as enablers, not gatekeepers. Programming should be easy, intuitive and above all else, fun.

So please have a play =)


Ambly Using JavaScriptCore C API

The Ambly 0.4.0 release involves a revision to use the low-level JavaScriptCore C API. Ambly was previously using the new Objective-C API wrapper that Apple had introduced with iOS 7.

This change broadens the usability of Ambly, making it easier to compose with projects that don't use the Objective-C API (React Native, for example). But more importantly: It fundamentally makes it possible to use Ambly in projects that don't have JavaScriptCore built with JSC_OBJC_API_ENABLED. One example is Ejecta; it is now possible to use Ambly to live-code Ejecta using ClojureScript.

To make this change essentially involved rewriting portions of the Objective-C side of the Ambly REPL to use C, using JSGlobalContextRef and related low-level API calls instead of the higher-level JSContext affordances. This really amounted to re-implementing working code to simply use analogous constructs, but all of this is internal and transparent to users.

The only visible Ambly API changes are that JSGlobalContextRef must be passed in in places where JSContext was previously. Apple makes this nearly trivial with a couple of bridging methods:

+[JSContext contextWithJSGlobalContextRef:]
-[JSContext JSGlobalContextRef]

Additionally, one of the JSContext-specific ABYContextManager APIs has been deprecated as it is no longer relevant.

From my perspective, the biggest change for this release is that a lot of the code became a little more… let's say, cumbersome, owing to the verbosity of the C style and the need to do manual memory management. But in my opinion, this is definitely worth it.

Also, there is a possibility that Facebook may use JavaScriptCore as the JavaScript engine for React Native on Android. If that's the case, it may turn out that coming to grips with the JavaScriptCore C API may pay off later when updating Ambly to work with Android!


Clojure in Action - Book Review

I just finished reading this book a couple of minutes here's my small review...

The book is kinda big...with 434 pages...and it's the first book and my first approach to Clojure.

I have to say...I'm not a big fan of Java...I don't even actually like it...but's something else -:) With its Lisp like syntax and it's functional's a delightful programming language...

The book itself it's a great introduction to have us ready for something Clojure is still a young language...some of the examples don't work mainly because some keywords or libraries had become obsolete...gladly most of the examples work just out of the box...

There are also many examples that help to fully understand what Clojure is all about...

If you haven't heard about closures, recursion, higher-order functions or currying...the this book is for you...

While I'm planning to read more Clojure books, I can say that this book is the perfect way to get started...


Development Culture.


How Not to Panic While Writing a Clojure Book

I made it to that magical moment when the Clojure book I had been working on so long was published and I could actually hold it in my hand.">

It was an immense project and I am very happy that it is finally done. Since then, I met some people that are interested in writing books as well. They asked if I had any insights or tips having gone through the process as a first time author. I have collected them in this post in hopes that they will be helpful to those going through the process themselves.

The very first thing to do is to get an outline for your book.

Start with an Outline

Ideas are soft and squishy. They drift into different shapes like clouds, and can melt away just as quickly. One of the hardest things to do was trying to arrange all those ideas in my head into an initial form that would serve as the structure for the entire book. I would pick up a pen and paper, start feeling overwhelmed, and suddenly remember I had something else very pressing to do. I successfully avoided starting a book for quite a while, until one day I cornered myself. I decided that I write my book outline on a long plane flight. With salted peanuts as fuel, and nowhere to escape, I finally wrote an outline. It wasn’t perfect but it was a start and looking back and it was not too far off. Here it is in all of its original roughness.

``` Title: Living Clojure

From beginning steps to thriving in a functional world

(Each Chapter will follow quotes from Alice in Wonderland and very use ideas from some examples)

Book 1 – Beginner steps

Chapter 1 – Are you comfortable? Talk about how OO is comfortable but there is another world out there and new way of thinking functionally.

        White Rabbit goes by

Chapter 2 – Forms & Functions – Falling down the rabbit hole Chapter 3 – Functional Transformations – Growing bigger and smaller – Key to thinking functionally is about transforming data from one shape to another shape.

        Map & Reduce

Chapter 4 – Embracing side effects – Clojure is impure functional language (The rabbit’s glove) – Cover do and io stuff. Also basic stuff about

        STM atoms and agents/ Protocols

Chapter 5 – Libraries, Libraries – – how to use Leiningen

        build system. Where to find clojure libraries, how to use
        Serpents - camel-snake-kebab

Chapter 6 – core.asyc – Tea Party introduction to the core.async library Chapter 7 – Clojure web – Chesire cat – introduction to Ring, Cheshire library, ClojureScript and OM

Book 2 – From here to there – thriving in a functional world

Training plan for thriving in a functional world.

Chapter 8 – Join the community – Surround yourself with other Clojure enthusiats – Twitter clojure – Github account – Clojure mailing list – Prismatic clojure news – Meetup for local community group. If there is not one in your area. start one! – Attend a Clojure conj

Chatpter 9 – Practice and build up Like Couch to 5K 7 week training program to work up to practicing Clojure


Now that I had an outline. I just needed to figure out how long it would take me to write the book.

However Long You Think It Will Take – You Are Wrong

Having never written a book before, I had no idea how much work it would be. The closest thing I had to compare it to was writing a blog post. I figured writing a chapter would be roughly equivalent to writing a blog post. If I could go back in time, this is the moment where my future self would pour a glass of ice water on my past self. Writing a book is nothing like that. It is a lot of time and work. If I had to compare it now to writing blog posts, the process would be this.

- You write a blog post.
- You rewrite the blog post.
- You write a second blog post.
- You rewrite that blog post and the first one too.
- You write another blog post.
- You rewrite all three post .....

So, if you have to commit to deadlines, make sure you remember how hard it will be, and then double the number.

Speaking of deadlines, they suck, but you should have them.

Make Deadlines

Deadlines are not fun. In fact, deadlines might even be a source of potential panic. But for me, they were necessary evil. There were a few beautiful times when inspiration came knocking at my door and I couldn’t wait to start writing. But most of the time, inspiration was off somewhere else eating biscuits. The only thing that actually moved the book along was me knowing that I needed to get the next chunk done by a certain date.

I found the best thing to do was to set aside a small bit of time on a daily basis to write something.

Routine, Routine, Routine

A daily routine was the most crucial thing for me. Life is really busy with work and family. It is so easy to get overwhelmed with daily life. I decided that mornings would work best for me. So I would stumble in to my computer an hour before work, with a hot cup of tea in hand, and write something. Some days I actually did quite a bit. Other days, I would write one sentence and declare it done. But, I would always do something. Even though those small slices of time didn’t seem like a lot, they added up over the course of a week.

Another curious thing happens when you do something, even a little bit, day after day. You start to get better at it.

Writing is a Different Skill from Coding

I was used to writing code all day. I found that the code writing skills are not the same as writing about code. In fact, I found it really hard to do at the start. But, just like writing code, you get better with practice. And to get better at anything, feedback is really important.

Get and Trust Feedback

After each chapter, I would get feedback from my editor. She was awesome and provided ways for me to improve the experience for the reader. I lost track of how many times I rewrote that first chapter, but each time it would get a bit better and I would improve as well. After the book was about half done it was sent out to others for technical review. They provided feedback not only on the writing style but also the technical content, making sure that it all made sense.

The feedback loop is much slower for writing a book than writing code, but it is just as vital. The great people providing feedback are you closest partners in this. You need to trust them. Especially during the roughest time, the middle of the book.

The Middle Bit is the Hardest

I found the hardest time was about halfway through the book. The initial excitement of the new endeavor had long since worn off. It seemed like such a mountain of a task, with no end in sight. I questioned my decision to continue with it daily. My routine and deadlines kept me moving forward. But my circle of friends and family kept me sane. It was great to have an outlet, not only to vent my frustration with my slow progress, but to get kind encouragement to keep my spirits up.

During these dark days, I also ate cheese.

Celebrate Your Small Victories

At the end of every chapter or deadline I would fix myself a nice plate of cheese and crackers. You have to celebrate the small wins. Cheese is also very tasty.

When the book was finally done. I had a really tasty plate, complete with Stilton, Brie, and a dependable Cheddar. I was incredibly happy to be finished. But I knew that I definitely could have not done it without the help of others.

Thank Everyone that Supported You

Writing a book is a huge undertaking that is utterly impossible to do alone. I could have not done it without the help and support of my editor, reviewers, family, friends, as well as the entire Clojure Community. I am so thankful to all of you that helped my in this project. You are great.

So, should you go ahead and write that book?

Do It

Yes, you should write that book and share your knowledge. Don’t panic, remember to breathe, get some cheese and tea, and go for it! It will be awesome.


Clojure Weekly, May 21st, 2015

Welcome to another issue of Clojure Weekly! Here I collect a few links, normally 4/5 urls, pointing at articles, docs, screencasts, podcasts and anything else that attracts my attention in the clojure-sphere. I add a small comment so you can decide if you want to look at the whole thing or not. That’s it, enjoy!

Speaker info - JBCNConf

Just enough self promotion to say that yours truly is going to speak about the basics of functional programming at the Barcelona Java Conference at the end of June. The talk will have a strong lispy background with example in Clojure and of course, Java 8. If you happen to be there for Euroclojure (the two conferences overlap one day) it might be worth checking a few of the talks at JBCConf for the Saturday.

The significance of the meta-circular interpreter

Oh that’s a good explanation for meta-circular interpreters and just in the first few lines. Grokking the impact of that on languages is another discussion. But this (old) article by Reginald Braithwaite goes on explaining what impacts such a feature (popularised by Lisp) has on language implementations. Most notably, it tends to create languages that are easier to change, including the deepest core primitives. It also impacts on meta-programming and in general in performances dealing with nested data structures, such as trees. Of course, because languages written in themselves need to analyse and compile efficiently.

talk-transcripts/ at master · matthiasn/talk-transcripts One of those talk transcripts to keep handy in your bookmark list. This talk by Rich from StrangeLoop 2011 made clear the distinction between what is subjectively “easy” (something that sometimes requires hard work to become familiar) and what is objectively “simple”. It also popularized the word “complection” that nowadays is in widespread use. It all boils down to the content of your toolbox. If you fill it with classes, type-only polymorphism, instance variables and so on, you get a complected toolbox that might feel familiar but it will hit you in the back when the application complexity increases. A must-read (and must-watch) presentation.

The Animated Guide to Paredit I’m sharpening my structural editing skills these days and I spend some more time observing my typing to make it more productive. I’m using vim-sexp under VIM and what I’m lacking most of all is the example of how others make good use of structural editing. There are a couple of things I’m using constantly already but I suppose there is more. Paredit literature is perfect in this case, since the logic if the structural change is the same (I just need to change keystrokes). This animated guide is both cool and rich in good examples. I’m searching more of those to use in everyday life apart from the usual (slurping, barfing, matching parens and so on).

The Many Faces of DrRacket I think I’ve found the best lambda logo so far, the one from the venerable PLT Scheme (aka Racket) around 2003. It was probably designed by Matthew Flatt in person judging from the initials (and someone else). Anyway, the skull makes the lambda really hard to ignore. Lambda rules!

nathanmarz/specter An interesting little lib from Nathan Marz of Storm fame. Specter is a small “walking language”, a Clojure DSL to query or update data structures. To the top of something like get-in or update-in it adds many feature like ways to filter, chaining multiple operations or apply your own selectors. The source code is so terse and readable that you are struggling to see how all of that can be implemented. It even “precompiles” with speed gain from 3x to 10x.

Planet Lisp Ah the venerable Lisp. There is still a tons of work to do to get close to what was already achieved with Lisp over the course of so many decades. So as a clojurian you should try to keep an eye on what is happening there (at much slower pace nowadays) and have a look at the past. That is, open different Lisp implementations and play with them, use libraries and attend conferences. Planet Lisp collects those resources for you.

Avian Avian is a Java virtual machine alternative to the standard JVM. It is built to be as minimal as possible and to run well on small devices or constrained environment. There are also attempts to build Clojure on top of it, although I couldn’t find evidence that it worked. An interesting project nonetheless.


Conditional read not allowed - Clojure Reader Conditionals

A new feature in the upcoming Clojure 1.7 release is Reader Conditionals. Reader Conditionals let you write portable Clojure/ClojureScript code in the same file, and wrap platform specific code in an expression which will only be executed on the platform it’s meant for.

A simple example is

 (>! c msg)
 (catch #?(:clj Exception
           :cljs js/Object)
 .... ))

One important restriction on Reader Conditionals is that they can only be used in Clojure files with a new .cljc extension.

I was recently porting some code to use Reader Conditionals and got this error message when compiling the namespace with a Reader Conditional in the file:

CompilerException java.lang.RuntimeException: Conditional read not allowed, compiling: <filename.clj>

I was very puzzled by this. After thinking for a little bit I started deleting parts of the file, to see if there was something in it which was stopping Reader Conditional use. I got down to a bare namespace with just a single reader conditional in the whole file and it still wouldn’t compile. I looked at my project.clj file to see if there was a weird setting in there causing problems but that looked fine too.

Eventually I realised that this file didn’t have a .cljc extension. I’d converted some of the files when I started, but then started adding Reader Conditionals to one I hadn’t converted. After changing the extension from .clj to .cljc everything compiled fine and I was on my way again.

I opened CLJ-1734 to give more descriptive error messages for Reader Conditional failures. Feel free to vote for it if you think it would be a helpful feature.


CLJS - Read files line by line on NodeJS Part 2

Note: There are a number of things that may be unfamiliar to new cljs devs. Rather than making this a huge post, I’ll write a separate post for each concept. For example, if the lazy-cat used in this post is unclear, check out my explanation of lazy-seq and recursion.

In my last post, we had some code that could read files line by line. We then used each line, either by a callback or a channel. While the code worked, I didn’t want the asynchronous reads. Instead, I wanted to get as close to this as possible:

(with-open [rdr (open-file path)]
  (doseq [line (line-seq rdr)]
    (js/console.log line))) ;;do something with line

This has the following benefits:

  1. It’s exactly the same as clj
  2. I don’t have to think about async code that provides no benefits here
  3. It lazily reads in the file and handles cleanup
  4. It’s a sequence and I can program to that interface

So the first thing was to switch from createReadStream to readSync. readSync synchronously reads in a user specified number of bytes.

(def fs (js/require "fs")) ;;require nodejs lib

(defn- read-chunk [fd]
  (let [length 128
        b (js/Buffer. length) ;;Buffer is a global nodejs lib
        bytes-read (.readSync fs fd b 0 length nil)]
    (if (> bytes-read 0)
      (.toString b "utf8" 0 bytes-read))))

We can now read an arbitrary amount of data from the file, but we have to manage finding actual lines of text. Here’s the logic we need to implement:

  1. Get the next chunk of data
  2. If it contains a line of text, return that line of text and save the remaining text for the next call
  3. If it does not contain a line of text, get more data and try our line check again
  4. If we read the EOL, just return whatever data we have

Here’s what I ended up with:

(defn line-seq
    (line-seq fd nil))
  ([fd line]
   (if-let [chunk (read-chunk fd)]
     (if (re-find #"\n" (str line chunk))
       (let [lines (clojure.string/split (str line chunk) #"\n")]
         (if (= 1 (count lines))
           (lazy-cat lines (line-seq fd))
           (lazy-cat (butlast lines) (line-seq fd (last lines)))))
       (recur fd (str line chunk)))
     (if line
       (list line)

line-seq takes a file descriptor and lazily reads from the file, returning a lazy sequence of lines.

The last bit is our with-open macro. Note: cljs macros have to be written in clojure

(defmacro with-open [bindings & body]
  (assert (= 2 (count bindings)) "Incorrect with-open bindings")
  `(let ~bindings
       (do ~@body)
         (.closeSync cljs-made-easy.line-seq/fs ~(bindings 0))))))

Now, we’re where we want to be:

(with-open [fd (.openSync fs "cljs-love.txt")]
  (doseq [line (line-seq rdr)]
    (js/console.log line))) ;;do something with line

And since we have a seq, we can use all of our normal tools:

(with-open [fd (.openSync fs "cljs-love.txt")]
  ;;only return lines that contain "awesome"
  (doseq [line (filter #(re-find #"awesome" %) (line-seq rdr))]
    (js/console.log line))) ;;nothing but awesome

Boom. Pretty sweet.

Here’s a gist of the full code.


Boot 2.0.0 Released

Boot Logo

Hey everybody, we just released Boot 2.0.0!

Boot is a build tool for Clojure that we use extensively here at Adzerk for everything from command-line scripts to web applications. For a great overview of Boot and why it's cool, check out Daniel Higginbotham's blog post, Boot, the Fancy New Clojure Build Framework.

Since RC1 back in December we've released 13 more release candidates, and are grateful for the feedback and assistance on them all we received from the thriving community of Boot users and contributors.

We're really confident in the utility and stability of Boot, and we encourage you to give 2.0.0 a shot.

API Changes

We didn't anticipate renaming any of Boot's public functions post-RC1, but Dom Kiva-Meyer convinced us that the functions related to Filesets API really needed to be reconsidered. Thanks Dom, you were totally right!

So, with Dom's feedback, we came up with a better set of more consistently-named fileset functions and added them to boot.core. The old functions will remain forever — and tasks using them will continue to work — but they are marked :deprecated and their use will print a warning message.

Community Tasks

The number of Community Tasks has roughly doubled since RC1 and there are now over 30 community tasks listed. Go us!


The CLJSJS project has also really taken off since RC1. CLJSJS is an effort to package various JavaScript libraries for ClojureScript use, and as of today is home to 45 build.boot-based projects. Each project in CLJSJS uses the boot-cljsjs task to cut down on boilerplate and perform common tasks.


Boot 2.0.0 is only a reality because of the active participation of a huge number of people over the past couple years. Beyond the 28 contributors recorded by git, many more people have contributed bug reports, documentation, and blog posts. To all who have contributed to or otherwise helped with Boot: thank you very, very much!


Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.