Creative computing with Clojure

creative_clojure

Clojure is gaining traction and popularity as a programming language. Both enterprises and startups are adopting this functional language because of the simplicity, elegance, and power that it brings to their business. The language originated on the JVM, but has since spread to run on the CLR and Node.js, including web browsers and mobile devices. With this spread of practical innovation, there has been another delightful development: a groundswell of people making art with Clojure.

Getting creative with Clojure

Creative Computing combines the power and engineering of the computer with the artistic inspirations of humans. People are using Clojure as a tool to generate music, visual art, poetry, and even dance. This ability to harness technology for creative purposes is both exciting and important. For it not only touches the heart and inspires existing technologists, but it also transcends all barriers. Art is a gateway to bring new people, young and old, from all walks of life, to the field of programming.

Let’s explore some of the areas of Creative Computing with Clojure, and showcase some inspiring examples from a selection of artist/programmers. We’ll look at projects that touch on music, art, games, writing, and even robots.

The scope of artistic expression is grand, so we’ll only scratch the surface, but we’ll attempt to highlight libraries and tools in each area that will provide a jumping-off point for the inspired.

We’ll start our tour with the realm of music.

Live coding music

Creating music with Clojure is a joy. The Overtone library allows you to not only make music with Clojure. Underneath the covers, the library provides an api to the SuperCollider synthesis engine. It has a rich set of scales, chords, rhythm, as well as a metronome timing system. One of the coolest things about Overtone, is the ability to make music interactively. Using the REPL, (Read-Eval-Print-Loop), a programmer can build up and create music in real-time. A wonderful example of this is a talk by Chris Ford on Functional Composition where he shows off Overtone in action. He starts off with a few simple principles, then builds up music in the REPL bit by bit, culminating in Bach’s Goldberg Variations.

This interactive, and collaborative nature of using Overtone was taken to new levels with Sam Aaron and Jonathan Graham. They combined forces to create a fantastic live coding band called Meta-eX. When performing, the music surrounding the audience is driven by live code. The code is front and center, being projected on a screen. The audience watches as the code is created and modified, driving the beat and the music. You really have to see them live to appreciate the whole experience, but here is a small excerpt from one of their live coding sessions, entitled Machine Run.

For live performances, Meta-eX also combines a visual element as well. Along with displaying the code, they will also show images, colors, and shadows that dance along with the music. Overtone has integration for two visual art libraries, Quil and Shadertone, that we’ll explore next.

Visual art

One of the most well known Clojure art libraries is Quil. It uses Processing to create sophisticated visual structures and artwork like this piece by Danielle Kefford.

hedera_helix_by_quephird-d8chot2

Quil supports both Clojure and ClojureScript. It is also well documented and provides examples of how to do generative art.

Shardertone is another lovely art library. One of the things I love about it is the integration with Overtone. It was designed to mix the Musical Synthesis and OpenGL shaders. Some of the shader examples can be seen in the Shadertoy website. When combined with the music of Overtone in a live-coding environment, the results are mesmerizing as in this video example by the REPL Electric.

People have also been experimenting with Clojure game platforms for visual art. This stunning pieces of visual art by Joseph Parker uses the Arcadia library to integrate Clojure with the Unity 3D game engine.

003

Other people have even been experimenting with combining Overtone and games as art. Joseph Wilk has been harnessing the musical beat in Overtone to generate 3D shapes in minecraft. By embedding a Clojure REPL inside the Minecrafter server, sound creates blocks and structures in this amazing demo.

Finally, Clojure has enabled a cool art project whose aim is to allow anyone to be an artist. The Devart-Codefactory project is an online design tool that empowers people by giving them a tool to create a complex 3d form. There is even the chance of having their art fabricated in 3D and showcased in an art exhibition. The tool, written in Clojure, has many fascinating examples created by its users in the gallery.

codefactory

Creativity is not limited to the visual and musical arts. Let’s explore how Clojure is used to produce poetry.

Poetry in code

Can computers create poetry? With Clojure, the answer is a resounding yes. People have used Clojure to generate haikus. They’ve also used them to generate text using Markov Chains. After this method is trained on a set of text, it then randomly generates new text and phrases with combined inputs.

A fun example of this is a Markov Chain generator that’s trained on the poetry of Edward Lear’s Nonsense Books with a good dose of Functional Programming from Wikipedia thrown in. The program is used by a twitter bot named FunctionalELear, that generates art tweets based off the input texts. The artful combination of such different spheres makes for some amusing text snippets.

16983522639_59a0bd68d0_b

So far we have covered music, visual art, and poetry. Let’s finish with some interpretive dance.

Robot dance party

With Clojure you can control robots. You can control everyday, helpful robots, like the Roomba with the clj-roomba library. You can control fun loving toy robots like the Sphero with the ellipso library. You can even control flying AR Parrot Quadcopter drones using Clojure with either the clj-drone or turboshrimp libraries. Once you can control movement with Clojure, through the interactive REPL, all you need for dance is to add music.

This presentation from O’Reilly Solid Con of a Real Time Robot Dance Party demonstrates the combination of music and robots. Through the power of the Clojure REPL and Overtone, a dance tune from Daft Punk is generated. Then using the signals from the beat, robot dance moves are coordinated all real time.

Hopefully you’ve enjoyed this small selection of creative computing projects using Clojure. The next time artistic inspiration strikes, pick up your text editor and create with Clojure.


Editor’s note: if you’re an experienced programmer looking for a thorough but gentle introduction to Clojure, check out Carin Meier’s Living Clojure.

Permalink

Finding functions with the clojure cheatsheet

When moving between languages built-in functions always end up with different names. In ruby the reject method will remove functions from a list using a predicate. It has the same name in underscore. In haskell it is called deleteBy. Do you know what it is called in clojure?

Use the Clojure cheatsheet

Clojure Cheatsheet

The clojure cheatsheet is a great resource when problems like this occur. You know what you want to do, and what kind of data it interacts with, but just can’t quite name it.

The clojure cheatsheet is also great for function discovery. There have been times where I want to do something related to composition or argument threading, but -> just doesn’t work. Looking at the cheatsheet around -> provides the other threading macros. I’ve used as-> and cond-> to make significant cleanups in my code, but would have missed them without the reference.

Additionally, the online cheatsheet version is connected to clojuredocs. With this you can click through and see examples of how to use any of the functions. Do you know how to use for with :when, :let, or :while? There are examples that show the syntax and result.

There are other sites too

There are other cheatsheet and reference sites such as the clojuredocs quickref or grimoire. They provide similar information, but provide different views and interactions. Try them out, and see which one works best for your workflow.

Permalink

Full-Stack Senior Functional Web Engineer at Front Row Education (Full-time)

Position

Senior full-stack functional web engineer to join fast-growing education startup that changes how over a million young students learn math.

TL;DR - Why you should join Front Row

  • Our mission is important to us, and we want it to be important to you as well: hundreds of thousands of kids learn math using Front Row every month. Our early results show students improve twice as much while using Front Row than their peers who aren’t using the program.
  • You’ll be one of the first engineers on the team, which means you’ll have an immense impact on our company, product, and culture; you’ll have a ton of autonomy and responsibility; you’ll have equity to match the weight of this role. If you're looking for an opportunity to both grow and do meaningful work, surrounded and supported by like-minded professionals, this is THE place for you.
  • A lot of flexibility: while we all work towards the same goals, you’ll have a lot of autonomy in what you work on. You can work from home up to one day a week, and we have a very flexible unlimited vacation days policy
  • You’ll use the most effective tools in the industry: Haskell, Postgres, Ansible and more. Front Row is one of the very few organizations in the world that use Haskell in production for most of their systems and is an active member of the Haskell community, including the Commercial Haskell Special Interest Group.
  • In addition to doing good, we’re doing really well: in just over a year after launch, we are in more than 20% of all US elementary & middle schools.

The Business

Millions of teachers around the USA are struggling to help 30+ students in their class learn math because every student is in their own place. In a typical fourth grade classroom, there may be students learning to count, students learning to add, students learning to multiply, and students learning how exponents work - and one teacher somehow needs to address all these needs.

Front Row makes that impossible task possible, and as of today, more than a hundred thousand students use Front Row to receive personalized guidance in their learning. Thousands of teachers use Front Row every day to save hours of time and make sure their students are growing at the fastest rate achievable. Front Row active users have been growing over 25% a month for the past 6 months.

Front Row is successfully venture-funded and on the road to profitability.

The Role

As one of our very first engineers, you will be part of a team of developers who take pride in their craft and push each other to continuously get better. You will strive for pragmatism and 80/20 in your work. You will be using tools that make you most effective and give you the most leverage. By working really smart, you will produce more than the average developer ever will, but without the crazy hours.

We love generalists who can quickly bring themselves up to speed with any technology we’re using: you will have the chance to learn a lot, and fast too. You will receive continuous support and mentorship on your journey to achieving mastery. We do however expect you not to need to be hand-held and rely on others for your own growth. You will have full autonomy over your work.

You will work in an effective team that plans, executes and reflects together. Because we’re a small team, everything you create will go into production and be used by students, teachers, parents and school personnel. You will never do unimportant work: every contribution will make a clear and tangible impact on the company’s trajectory. Your personal success will be directly aligned with that of the company.

Most importantly, your work will have purpose: Front Row is a mission-driven company that takes pride in making a significant impact in the lives of hundreds of thousands of students.

Tools

  • Front Row is a polyglot combination of multiple web applications, mobile apps and asset generation tools.
  • Web front-ends are a custom version of Backbone.js + plugins.
  • The backend is a series of Haskell+Yesod-based applications talking to PostgreSQL and 3rd party services. Some Clojure here and there from older codebases on the way out.
  • All test, build and deployment automation relies on Ansible. AWS for hosting.
  • We have mobile apps for both iOS and Android
  • Work is continuously happening to simplify and minimize the codebases. We <3 getting rid of code :)

Must haves

  • You have experience doing full-stack web development.
  • You understand networking and have experience developing distributed systems
  • You have experience with RDBMS
  • You have functional programming experience.
  • Extreme hustle: you’ll be solving a lot of problems you haven’t faced before without the resources and the support of a giant organization. You must thrive on getting creative in order to get things done.

Very nice-to-haves

  • You have existing familiarity with a functional stack (Haskell / Clojure / Scala / OCaml etc)
  • You're comfortable with the Behavior-Driven Development style and Continuous Delivery
  • You have worked at a very small startup before: you thrive on having a lot of responsibility and little oversight
  • You have worked in small and effective Agile/XP teams before
  • You have delivered working software to large numbers of users before
  • You have done system and network administration and are comfortable working in the Linux environment
  • You have implemented deployment strategies for cloud infrastructure
  • You have experience scaling distributed systems and designing large scale web backends

Benefits

  • Competitive salary
  • Generous equity option grants
  • Medical, Dental, and Vision
  • Lunch is on us three times a week, and half-day event every month (trip to Sonoma, BBQ, etc)
  • Equipment budget
  • Flexible work schedule
  • Flexible, untracked vacation day policy
  • Working from downtown SF, super accessible location right by Powell station

Front Row - our mission

It's an unfortunate reality that students from less affluent families perform worse in school than students from wealthier families. Part of this reason has to do with home environment and absentee parents, but much of it has to do with inferior resources and less experienced teachers. The worst part of this problem is that if a student falls behind in any grade, they will forever be behind in every grade.

That's the core problem Front Row solves - it doesn't let students fall behind. And if they fall behind, Front Row helps catch them up really quickly because Front Row arms teachers with the resources and knowledge to develop their students individually. Now, the probability of falling behind in any given grade is irrelevant, because it will never compound. The student who would have been the most at risk will instead be up to speed, and therefore far more motivated.

Get information on how to apply for this position.

Permalink

SwiftKey Clarity Beta: Android keyboard written in Clojure

In case you haven't hear yet: a week ago SwiftKey released their new experimental keyboard called Clarity. Its killer feature is powerful multi-word auto-correction that does not just look words up in the dictionary, but analyzes what you have written and fixes entire phrases. Clarity is still in its beta phase, some features are missing (no swiping as of yet), but @adampclements and his team did and continue to do a great job, so I'm sure the lacking functionality will be implemented in the near future.

Under the hood Clarity uses most of the Clojure-Android toolchain: lein-droid, Neko, Skummet and obviously our special Android-compatible Clojure build. Over the course of developing Clarity Adam contributed to many Clojure-Android projects, and drove some of the important architectural decisions. Overall, Clarity is a massive application in terms of complexity and used libraries, so being written in Clojure it proves that CoA can be used for serious projects. I wish Adam and SwiftKey to hit that sweet 1.0 soon, and I drink to one more app in CoA ecosystem!

Permalink

Kata: String Calculator in Clojure

Last week we started working on the String Calculator kata at the Clojure Developers Barcelona meetup.

Finally, this week I found some time to finish it.

These are the tests using Midje:

The resulting code which is divided in several name spaces.

The string-calculator.core name space:

The string-calculator.numbers-parser which is where most of the logic lives:

The string-calculator.numbers-validation name space:

Finally, the string-calculator.numbers-filter name space:

I used a mix of TDD and REPL-driven development to code it.

To document the process I committed the code after every passing test and every refactoring.

This time I didn't commit the REPL history because I used Cursive and I didn't find how to save it. Apart from that, I really enjoyed the experience of using Cursive.

You can find the commits step by step hereand the code in this repository in GitHub.

Permalink

Fast and Scalable Clojure Ring Web Applications with Comsat

As many clojurists know already, Clojure’s minimalistic conciseness and lucidity can bring productivity and maintenance benefits to daily development tasks, but also aesthetic bliss and even joy to our jobs; thanks to Pulsar you can also enjoy fibers’ high performance and advanced concurrency abstractions on traditional Clojure applications.  Preview Text:  ...

Permalink

Sr. Software Engineer at McGraw-Hill Education (Full-time)

McGraw-Hill Education is a digital learning company that draws on its more than 100 years of educational expertise to offer solutions which improve learning outcomes around the world. McGraw-Hill Education is the adaptive education technology leader with the vision for creating a highly personalized learning experience that prepares students of all ages for the world that awaits. The Company has offices across North America, India, China, Europe, the Middle East and South America, and makes its learning solutions available in more than 65 languages. For additional information, visit www.mheducation.com.

This Senior Software Engineer position will be joining the new adaptive learning engineering team at McGraw-Hill Education's new and growing Research & Development center in Boston's Innovation District.

We make software that helps college students study smarter, earn better grades, and retain more knowledge.

The LearnSmart adaptive engine powers the products in our LearnSmart Advantage suite — LearnSmart, SmartBook, LearnSmart Achieve, LearnSmart Prep, and LearnSmart Labs. These products provide a personalized learning path that continuously adapts course content based on a student’s current knowledge and confidence level.

On our team, you'll get to: Move textbooks and learning into the digital era Create software used by millions of students Advance the state of the art in adaptive learning technology Make a real difference in education

Our team's products are built with Flow, a functional language in the ML family. Flow lets us write code once and deliver it to students on multiple platforms and device types. Other languages in our development ecosystem include especially JavaScript, but also C++, SWF (Flash), and Haxe.

If you're interested in functional languages like Scala, Swift, Erlang, Clojure, F#, Lisp, Haskell, and OCaml, then you'll enjoy learning Flow. We don't require that you have previous experience with functional programming, only enthusiasm for learning it. But if you have do some experience with functional languages, so much the better! (On-the-job experience is best, but coursework, personal projects, and open-source contributions count too.)

Required Experience:

Have a solid grasp of CS fundamentals (languages, algorithms, and data structures) Be comfortable moving between multiple programming languages Be comfortable with modern software practices: version control (Git), test-driven development, continuous integration, and Agile methodologies.

Get information on how to apply for this position.

Permalink

Pixie Pi

Pixie is a Clojure-inspired lisp that has a quick startup time and nice FFI support that allows it to make use of existing C libraries, making it a nice option for working with the Raspberry Pi.

Installing Pixie

Meeting Dependencies

From a base Raspbian install, Pixie’s dependencies (as of eb5886ff95) except for libuv-dev can be satisfied using aptitude

sudo apt-get install libffi-dev libedit-dev libboost-all-dev

libuv-dev (and libuv) are available from the Jesse packages. Use the armhf builds.

Use wget to grab the packages and install with

sudo dpkg -i libuv*armhf.deb

Building the Pixie VM

make build_no_jit

It’ll run for about an hour. Enjoy the excellent ascii art the RPython build process produces.

Cross compilation from a faster machine may be an option as well.

Interfacing with the Pi’s GPIO

Using Pixie’s FFI, I made a library called wiringPixie wrapping wiringPi. Follow the instructions on the README to install

https://github.com/stuarth/wiringPixie

and start hacking (or controlling LEDs :))

(require wiring-pixie.gpio :refer :all)

(def pin 7)

(wiringPiSetup)         ;; required first!
(pinMode pin OUTPUT)    ;; pin to OUTPUT
(digitalWrite pin HIGH) ;; set pin 7 to HIGH

Pixie Pi was originally published by Stuart Hinson at Stuart Hinson on May 05, 2015.

Permalink

Tropology performance: PostgreSQL vs Neo4j

Or, apples vs. papaya salad.

Short version: I’ve moved Tropology to PostgreSQL for performance reasons, and because after some evaluation, Neo4j wasn’t as good a fit as it seemed at first blush.

You may want to start by catching up with the other Tropology articles.

All done?

Neo4j’s pros and cons

The conceptual fit

Neo4j seemed like a very good fit at first. We’re handling connected data, in the form of which concepts are referenced from which articles. This could in theory let us do queries such as finding the various paths that lead from Lampshade Hanging to If it bleeds, it leads.

However, the model is too widely connected, with some major hub pages having thousands of outgoing and incoming links. Even after some judicious pruning, we ended up with 11,566,441 relationships and 222,384 pages.

This means that while Neo4j can help us find the shortest path between two concepts (which may be a direct link), finding a set of paths likely leads to a combinatorial explosion for even a set of paths that are just 3 nodes long.

This means such queries aren’t really useful for us… removing one of the great advantages of Neo4j.

Peculiarities

Neo4j isn’t without its share of idiosyncrasies.

Nodes have labels. The way I was originally modeling the data, a label was a conceptual group - for example, Film or Manga. Labels are not the same as a table, since a node can have multiple labels.

Neo4j nodes have properties, which don’t need to be present in all nodes. You can have indexes on properties, which is of course very useful for speeding up queries when, say, you need to find a specific node by its code. However, said index will not be used unless you also include the node’s label - if for any reason you don’t know a node’s label, you’ll trigger a full scan.

While this may seem like a small issue, we don’t really know which potential labels we have so we can declare the indices in advance - we get our categories from crawling the site. We could parametrize these, building queries on the fly and keeping track of which labels we have already created an index for, but we end up with a convoluted codebase that’s also much slower to import data into.

The faster way is actually keeping all nodes into a single, known label - Articles in this case - and using MERGE to create them all at once…

At which point we’re effectively doing the same thing as if we were keeping every node as a tuple inside a table in a traditional RDBMS.

Cypher

Neo4j’s Cypher language is very powerful, and it allows you to tersely express queries for node relationships. It’s not extensively documented, unfortunately, and while there is the Cypher reference card available there doesn’t seem to be a syntax specification that fully details how you can combine these into valid expressions (see the documentation for Postgres’ SELECT as a counter-example).

This may seem like a minor thing, but it ends up resulting in a fair amount of fumbling around, often until you manage to get advice from someone more experienced with the language. In my case, I was unaware of how could you do multiple UNWIND expressions, and was trying several syntax combinations until I got the right one from Michael Hunger.

And while I’m hesitant to put it this way… it would seem like Cypher needs better automated tests against regressions.

I was working on improving a somewhat naive query, which was taking about six seconds to return the relationships among nodes related to a large, well-connected article.

While I managed to get a succinct way in which to express this particular query, I wasn’t initially able to get the results I wanted. After testing across several versions, I realized it likely wasn’t a problem in my query: Cypher just isn’t behaving consistently from 2.1.2 to 2.2.1 to the version that powers Neo4j’s online console.

This lead me to search for other Cypher issue reports, and I found more than I was comfortable with. In the case of querying inconsistencies sometimes whomever files the issue adds an alternate, equivalent query that does work as expected, which could help you if you’ve tripped over that same problem, but the fact that I ran into a querying issue with a not particularly complex statement gave me pause.

On to Postgres

The migration

I could have continued using the slow version of the query above while either I found a workaround or Neo4j released a fix, but decided instead to take the time to run some tests on PostgreSQL 9.4. After all, if we’re already storing all nodes under a single label, that’s effectively the same as dumping them all into a table, so might as well try our hand at a regular data storage then.

Clojure shone again. Refactoring the db namespace internals went smoothly, and it’s amazing the difference that being able to do structured editing on your expressions makes. Not to mention, its focus on data handling and transformations works wonders. “Oh, you get slightly different result keywords from your new database? Just apply a map transformation!

I’d dabbled in Korma before, so I chose it for the queries. It’s still a great way to write SQL in Clojure, but caveat emptor: if you’re dealing with query composition or more elaborate joins, the documentation could still use some more examples.

The results

Unsurprisingly, Postgres is significantly lighter. Look at that CPU usage drop for effectively the same process:

PostgreSQL CPU Usage

Funny enough, the amount of bandwidth being used is much higher. Guess why?

Bandwidth Usage

We’re getting at least a 4x increase on the number of pages imported, at 1/2 the CPU cost. That’s an improvement of at least 8x for those keeping score at home. And that’s considering that we’re storing a significantly higher volume of data now, since this branch caches the page contents on Postgres.

For the record, this example is in no way skewed towards Postgres. That data is running on a vanilla, untuned Docker container, and some of the queries are still naive because on my first pass my priority was retaining the same internal API while just migrating the data store.

Also, that multi-second query that made me trip onto the inconsistent result behavior in Cypher? It returns in < 300ms here.

Conclusion

Neo4j is a great tool, but is not necessarily the best one even for data that can easily be represented as a graph. You’d probably get significantly better results than I did when the connections between your nodes aren’t as dense as the ones on my model, when you know your node labels/categories in advance, and when you get to assign costs or other properties to the relationships to help with filtering.

Its power comes at a high CPU cost, of course, and it seems like on the testing and documentation departments it’s still not up to the standards of more traditional database systems.

When dealing with a large amount of densely-connected data that you need to get consistent performance out of, the old workhorses still have the edge.

Permalink

Early May 2015 London Clojure Dojo at uSwitch

When:
Monday, May 11, 2015 from 7:00 PM to 10:00 PM (BST)

Where:
uSwitch (Notcutt House)
36 Southwark Bridge Road
SE1 9EU
United Kingdom

Hosted By:
London Clojurians

Register for this event now at :
http://www.eventbrite.com/e/early-may-2015-london-clojure-dojo-at-uswitch-tickets-16859214364?aff=rss

Event Details:

London Clojure Dojo at uSwitch

The goal of the session is to help people learn to start working with Clojure through practical exercises, but we hope that more experienced developers will also come along to help form a bit of a London Clojure community. The dojo is a great place for new and experienced clojure coders to learn more. If you want to know how to run your own dojo or get an idea of what dojos are like you can read more here.

What should I bring?

We hope to break up into groups for the dojo. So if you have a laptop with a working clojure environment please bring it along.

How do I get in to the building?

There is a buzzer at the main entrance on the street.  Press that to alert reception you have arrived.

Is there a mailing list I can find out more information?

Why yes. Yes there is.. and we’ll be discussing this and other meetups on the london-clojurians mailing list

What is Clojure? 

Clojure is a JVM language that has syntactically similarities to Lisp, full integration with Java and its libraries and focuses on providing a solution to the issue of single machine concurrency.

Its small core makes it surprisingly easy for Java developers to pick up and it provides a powerful set of concurrency strategies and data structures designed to make immutable data easy to work with. If you went to Rich Hickey’s LJC talk about creating Clojure you’ll already know this, if not it’s well worth watching the Rich Hickey “Clojure for Java Programmers” video or Stuart Halloway “Radical Simplicity” video .


Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.