Full Stack Developer - Product Focus - Clojure at Swing Education (Full-time)

Overview: Swing Education is the education technology space with an immediate focus on tackling the substitute teacher shortage problem.

Our online platform makes it easy for great schools and quality subs to find each other. We help schools access more subs and offload tedious work associated with managing a pool of substitute teachers (recruiting, screening, payroll, etc.). Our network provides subs with a wider range of work opportunities to gain experience and build their professional network.

This position reports to the CTO and is based in San Mateo, CA. Swing is a revenue generating, Series A, Y Combinator company (through ImagineK12).

We are looking to grow our engineering team with engineers that have broad interests and want a chance to participate in many aspects of the business and want to help contribute to the broader goal of helping schools operate more efficiently.

Responsibilities:

  • Work closely with our product team to develop features in a pragmatic way. This may involve suggesting tweaks, offering potential trade-offs, and breaking down a feature into smaller chunks to be developed and deployed in stages. We expect our engineers to participate in the product development process starting with a rather broad goal/directive with minimal specification and and implement both a functional user interface as well as the backend infrastructure required to support it.
  • Support both customers and other teams tooling and smaller computer programs to aid in automation and productization of internal processes and data needs. We strongly believe that programming in a vacuum is not nearly as fun nor as rewarding as becoming intimately involved with your users and teammates and being able to observe the impact that your work is having and participate in the entire product lifecycle.
  • Assist in implementing infrastructure and tooling to increase overall developer productivity. Examples would be utility libraries, helper macros, locating and contributing to relevant open source projects.
  • Support and maintain existing software via performance and quality improvements as we continue to grow to keep everything running smoothly.

Technology:

  • Our primary applications are interactive ClojureScript web applications built on Reagent (which is a ClojureScript wrapper around React)
  • Our backend API is written in Clojure and our primary data store is Datomic. All day-to-day development will be in Clojure.
  • Please note that previous Clojure experience is not a requirement but is a nice-to-have. We are confident in our ability to train as needed and that engineers will enjoy working in Clojure
  • Only list what’s absolutely necessary here. Try to keep it to five one-line bullets or less.

Qualifications:

  • Solid foundation in computer science fundamentals. While we do not strictly require a degree in Computer Science and day-to-day work does not require algorithmic expertise, there are certain classes of problems that do tend to crop up that you should be comfortable with and have experience recognizing. Some examples:- Recognize when you have an n-squared algorithm or worse.
  • Recognize when you are searching for a pattern that cannot be matched via regular expression or have a problem that is equivalent to the Halting problem.
  • Track record for becoming a domain expert over time in both technical and business areas
  • Track record of a pragmatic approach to software development

Bonus Points for: - Open source experience - React Native experience - Previous edtech, startup, or online marketplace experience - New user on-boarding and first touch tailoring experience

Perks: - Medical/dental benefits - Generous PTO - Paid holidays - Stock options - 401k match - Maternity/Paternity leave benefits - A stocked kitchen and unlimited caffeine - Mission-driven work - Fun, collaborative, balanced culture

Swing Education is an equal opportunity employer and strongly encourages applications from people of color, persons with disabilities, women, and LGBT applicants.

Get information on how to apply for this position.

Permalink

Full Stack Developer - Product Focus - Clojure

Full Stack Developer - Product Focus - Clojure

Swing Education | San Mateo, CA
Swing Education is focused on tackling the substitute teacher shortage problem.

Overview:

Swing Education is the education technology space with an immediate focus on tackling the substitute teacher shortage problem.

Our online platform makes it easy for great schools and quality subs to find each other. We help schools access more subs and offload tedious work associated with managing a pool of substitute teachers (recruiting, screening, payroll, etc.). Our network provides subs with a wider range of work opportunities to gain experience and build their professional network.

This position reports to the CTO and is based in San Mateo, CA. Swing is a revenue generating, Series A, Y Combinator company (through ImagineK12).

We are looking to grow our engineering team with engineers that have broad interests and want a chance to participate in many aspects of the business and want to help contribute to the broader goal of helping schools operate more efficiently.

Responsibilities:

  • Work closely with our product team to develop features in a pragmatic way. This may involve suggesting tweaks, offering potential trade-offs, and breaking down a feature into smaller chunks to be developed and deployed in stages. We expect our engineers to participate in the product development process starting with a rather broad goal/directive with minimal specification and and implement both a functional user interface as well as the backend infrastructure required to support it.
  • Support both customers and other teams tooling and smaller computer programs to aid in automation and productization of internal processes and data needs. We strongly believe that programming in a vacuum is not nearly as fun nor as rewarding as becoming intimately involved with your users and teammates and being able to observe the impact that your work is having and participate in the entire product lifecycle.
  • Assist in implementing infrastructure and tooling to increase overall developer productivity. Examples would be utility libraries, helper macros, locating and contributing to relevant open source projects.
  • Support and maintain existing software via performance and quality improvements as we continue to grow to keep everything running smoothly.

Technology:

  • Our primary applications are interactive ClojureScript web applications built on Reagent (which is a ClojureScript wrapper around React)
  • Our backend API is written in Clojure and our primary data store is Datomic. All day-to-day development will be in Clojure.
  • Please note that previous Clojure experience is not a requirement but is a nice-to-have. We are confident in our ability to train as needed and that engineers will enjoy working in Clojure
  • Only list what's absolutely necessary here. Try to keep it to five one-line bullets or less.

Qualifications:

  • Solid foundation in computer science fundamentals. While we do not strictly require a degree in Computer Science and day-to-day work does not require algorithmic expertise, there are certain classes of problems that do tend to crop up that you should be comfortable with and have experience recognizing. Some examples:
  • Recognize when you have an n-squared algorithm or worse.
  • Recognize when you are searching for a pattern that cannot be matched via regular expression or have a problem that is equivalent to the Halting problem.
  • Track record for becoming a domain expert over time in both technical and business areas
  • Track record of a pragmatic approach to software development

Bonus Points for:

  • Open source experience
  • React Native experience
  • Previous edtech, startup, or online marketplace experience
  • New user on-boarding and first touch tailoring experience

Perks:

  • Medical/dental benefits
  • Generous PTO
  • Paid holidays
  • Stock options
  • 401k match
  • Maternity/Paternity leave benefits
  • A stocked kitchen and unlimited caffeine
  • Mission-driven work
  • Fun, collaborative, balanced culture

Swing Education is an equal opportunity employer and strongly encourages applications from people of color, persons with disabilities, women, and LGBT applicants.

Permalink

Clojure Tip of the Day – Episode 4: Threading Macros: part 1 – thread-first, thread-last, thread-as

After a long break, I’ve finally recorded another episode of the Clojure Tip of The Day screencast. I’ve spent last month or so by revisiting my goals for the next year and I want to bring more consistency to my blog and other creative activities.

This episode is about thread-first (->), thread-last (->>), and thread-last (as->) macros. Again, you can find it on YouTube: https://www.youtube.com/watch?v=w0CxOHmny80

The next episode will be about some->, some->>, cond->, and cond->.

Permalink

Clojure in London: Entrepreneur First

Bringing outliers together.

ef vr demo day

Entrepreneur First is a London-based start-up accelerator which assists entrepreneurs to design and run their own start-ups. It differs from other accelerators such as Y Combinator and Wayra in that it seeks out individuals before they have a concept or existing company.

Conan Cook is the Technical Lead at EF and is building a team using Clojure and ClojureScript to develop the platform. I caught up with Conan to learn more.

Jon: How would you describe Entrepreneur First?

Conan: In short, we help ambitious people create companies. Our founders Alice, and Matt, were working at a management consultancy and they realised that the traditional career paths for ambitious people were for instance working for large institutions such as banks. This didn’t always utilise their potential and in general, large companies are not geared towards making entrepreneurs.

What we aim to do is produce an environment to help people get into entrepreneurship. One of the highest value things we do is to help in finding co-founders. Even if someone has a great idea, they are often not able to follow it through alone. By analysing the data each person gives us, we can create teams of people who otherwise would not have found each other.

Jon: So what sort of functionality does the platform provide?

Conan: We have dashboards and databases for our internal teams, and we have another dashboard which is a bit like a social network for when people are finding co-founders, forming teams and developing products. This allows us to link people, to show progress of teams and to identify what they most need.

Also Entrepreneur First invests in these companies, and our platform gives us data to make decisions. Crunching this data occupies the tech team.

Jon: How many entrepreneurs do you have in the platform?

Conan: 100 funded companies have been founded at Entrepreneur First. We aim to increase that number and have recently expanded to Berlin, alongside our existing locations in London and Singapore.

ef logo blue

Clojure?

Jon: Why Clojure?

Conan: As Rich Hickey said in his 10 Years of Clojure speech, most of the world’s computer systems are based around data processing. Entrepreneur First is a good example of a company which transforms and moves data into various forms. This makes Clojure ideal.

We are interested in a collection of data about the individual (what funding, what company, what investors etc.). Lots of data can be attached to one person, and it’s so easy in Clojure to pick the bits you want to change around. It’s very easy to build systems that represent the data as well as having exploratory tools.

There’s a reasonable quantity of exploratory work going on to come up with hypotheses, and we need to test these against the data. We are asking lots of new questions that we hadn’t thought of when the data was first stored. Having dynamic simple data structures makes it easy to adjust.

Tech Stack

Jon: What is your tech stack?

Conan: In terms of Clojure APIs: Ring, Liberator and Compojure. Liberator is a good tool; it’s overkill in some places, but I still think it’s a good tool.

We also use ClojureScript, and we find that being able to write UIs in the same language as the back-end is a huge advantage. Our data flows very neatly from the back-end to the UI.

Currently most of our data is in Postgres and we are in the process of moving to Datomic.

Jon: Why move from PostgreSQL to Datomic?

Conan: The problem with traditional SQL databases is that you have to say what you’re going to store: they don’t handle sparsity well.

As individuals move through our system, we accumulate more data about them and the data-model starts to diverge. In Datomic we just add the data we need. In a SQL world you would end up with a large number of migrations.

When you are modelling something fixed, without room for variation, Postgres is then a good fit.

Jon: Any other reasons to choose Datomic?

Well, it’s easy to use, and is enjoyable to work with from an engineering perspective.

We have also started using Clojure Spec, and the connection between the two - namespaced keywords running throughout the system - is so neat. I know what the shape of every piece of data is based on its keyword.

Jon: You are a fan of Spec?

Conan: The more I learn about it, the happier it makes me. There’s a learning curve, for example with generators, and some of it is a little bit strange. I recently wrote a blog post with a good example.

Having generators in place to generate arbitrary data with a single line of code is fantastic. I write the description of the function next to the function. fdef is really nice.

There are some very good talks out there on Spec.

ef door close

Hiring

Jon: How many devs are you?

Conan: Just me at the moment! But we are in the process of hiring a team of four. We hope to be more than that eventually, but we have to prove the value of the team first - of the new hires we make.

Jon: And how have you found the hiring process?

Conan: Hard as Clojure developers are in demand. There are more developers than jobs and many are going freelance.

Niche languages tend to attract people with experience, and there’s a reasonably good pool of Clojure developers.

We find the same diversity problem that you have in many places - we find it difficult to hire from a wide selection of backgrounds. But then you do find people in Clojure who haven’t taken the usual career path.

ef founders

Jon: Would you consider hiring remotely to attract more people?

Conan: Maybe in the future. When the team is small, we find it more valuable to have people co-located. The company is built on face to face interactions.

Jon: What level of developer are you looking for?

Conan: Junior to mid-level. We are prepared to teach people, and we can cast the net widely. For mid-level I find people myself using GitHub, LinkedIn and Twitter. We’ve used agencies such as Functional Works.

Training

Jon: How have you found the training process?

Conan: I have done it in past and it’s always been great. There’s a learning curve for FP, but if you haven’t done much OO, then it can be quicker to pick up. Clojure does have an issue with error messages, but then in the Clojure world there are very few surprises.

Jon: What do you think of the current state of Clojure in London?

Conan: I’m slightly dissapointed that more organisations aren’t using it. There have been a few additions - but it’s not five or ten times the number of companies that it was five years ago.

Clojure Exchange

That said, it’s positive that there are more people using it and more positions available.

I enjoyed ClojureX - I’ve always enjoyed it. There is a bigger variation of new people using it, and people pushing the boundaries.

Jon: Any closing words?

Conan: I’d really like to understand how to manage a very large Clojure base. I get the impression that the Clojure community hasn’t thought much about this; that there isn’t a history of building enormous systems, rather lots of smaller systems.

This is something I’ll be thinking about very carefully.

Jon: At JUXT we use Emacs/Vim on Arch with Dell XPSs, what do you use?

Conan: I use Windows, with the Windows subsystem for Linux - it has bindings for the kernel. I use this as my terminal to run tests etc and to do terminally things such as SSH. I run Intellj with Cursive natively in Windows.

Permalink

Ebnf

Long time, no see

Well, it's been longer than I would have liked since the last post. As usual in this industry, things always take longer than you expect. I'll not bore you with the details. So let's get on to the cool stuff.

EBNF

I've been writing about parsing for the last couple of posts. This always involves building up a grammar of some sort from smaller grammars and then passing the full grammar to a function to produce a parser. The point I've been trying to make, and will drive home in this post, is that the grammar itself is just a data structure. Deriving a parser from it is just one way to interpret it. Here are a couple of others.

But first, a digression.

It's convenient to have a way to write out a grammar in a concise, readable way that clearly specifies what strings are valid. The EBNF is a very useful tool to have. Click on the link to read up on it.

One thing you can do with a grammar in Toccata is generate the EBNF for it. This can be done with this library. You can take any grammar and pass it to the produce-ebnf function and it will spit one out. Using this library, the EBNF for the JSON grammar we looked at last time is

whitespace = { ' ' | '\t' | '\r' | '\n' | '\f' };
escaped chars = '\\', '\"' | '\\' | '/' | 'b' | 'f' | 'n' | 'r' | 't';
string = whitespace, '\"', { escaped chars | not '\"' }, '\"';
digit = '0' - '9';
integer = whitespace, ['-'], digit, { digit };
object = whitespace, '{',
         [{ string, whitespace, ':', whitespace, value, whitespace, ',', whitespace },
          string, whitespace, ':', whitespace, value],
         whitespace, '}';
array = whitespace, '[', [{ value, whitespace, ',', whitespace }, value], whitespace, ']';
value = string |
        integer |
        object |
        array |
        whitespace, 'true' |
        whitespace, 'false' |
        whitespace, 'null';

I'd originally intended to explain how the EBNF generator works, but as I reviewed it, I realized I would have to write about 5 posts and I have other things I want to get to. Maybe down the road, I'll come back to that. But the important thing I want to communicate is that a grammar in Toccata is just a data structure and they can be used in many different ways. For instance, here is the EBNF for the Toccata language itself. (I just realized I should put this in a documentation directory.) This document is not the specification of the parser, that's here. But it's generated automatically from the specification, so it's always accurate as of the time it was generated. How many other languages do you know that have an EBNF you can examine to see if an expression is valid? Most of the time, whatever the parser accepts is what's considered valid and there are almost always some weird corner cases.

A picture is worth ...

I forget when I discovered it, but this article was written in 2007 and deals with parsing, regular expressions and Finite State Automata. I can not recommend it, and the three that follow it, highly enough. For present purposes, I want to draw attention to the diagrams scattered throughout. These show the different states a parser can go through as it reads each character of the input string. These diagrams can be very illuminating in a way the textual representation of the EBNF isn't. This is similar to the Syntax Diagrams on the JSON.org home page. Except the Syntax Diagrams and EBNF show the structure of the grammar. The state machine graphs show the states of the parser. Wouldn't it be nice to be able to generate these graphs automatically?.

Using that library, here's the graph for the JSON grammar we've been looking at.

link

This did require some updates to the json.toc grammar. How it works is the 'grammar-graph' library converts the grammar to a Graphviz DOT file. This file is then fed into the dot command to produce an SVG or PNG image. Slick!

The Toccata code is located in the 'examples' directory alongside the 'json.toc' file:

Parse some sample JSON strings
Generate EBNF
Generate a state graph
(To produce an SVG file, "run examples/json-graph.toc | dot -Tsvg > json.svg")

So that's three ways we can interpret a grammar. The only limit to others is your imagination or need. For instance, it would be easy to generate a regex from a grammar.

An interesting thing happened on the way to the show

I have to say, the code to generate the graphs is not easy to grok. It's pretty deep and there are several requirements that all have to be satisfied simultaneously. A real nightmare. Which is why it's been a week (or more) since the last post. However, as I was writing that code and testing it against the JSON grammar, I noticed something.

link

Can you tell which state graph is incorrect? By looking at the state graph, I found a bug in my JSON grammar which I subsequently corrected. Having different ways to look at things is really useful.

Custom parsers

One final point. Being able to easily specify a grammar and generate a parser from it allows you to create custom parsers. Say you want to use JSON to serialize some data structures from your code. Instead of using a generic JSON parser library, you can write a custom parser that only accepts an object value and limits the key strings to small set of strings. This custom parser is probably going to be much faster than a generic parser that parses the JSON into a hash-map which you then have extract the fields from. And, depending on the parser generated from the custom grammar, could give more useful error messages when it fails.

We'll wrap it up there, but there's some exciting things just ahead. I'm writing these posts to show how Toccata can be used for practical purposes and we're going delve into another format next time.

Permalink

Transducing a Text File

This post highlights of one of the core ideas posted in this blogpost. If you’ve already read it and you’re intimately familiar with transducers, this post probably won’t have anything new for you. I’ve posted this to Stackoverflow before and saving this to my blog for archival purposes.

In the pre-transducer era, reading text files was often done like this:

1
2
3
4
5
6
(require '[clojure.java.io :as io])
(with-open [rdr (io/reader "/tmp/work.txt")]
  (->> (line-seq rdr)
       (mapcat #(str/split % #";"))
       (map count)
       (doall))

Given input work.txt:

I;am;a;string
Next;line;please

this would return (1 2 1 6 4 4 6). One caveat with this approach is you have to realize the result inside the with-open macro, else the file would already be closed.

What if we want to use transducers instead of lazy collection transformations? The ingredient you need is something that allows you to treat the lines as a reducible collection and which closes the reader when you’re done reducing:

1
2
3
4
5
6
7
8
9
10
11
12
13
(defn lines-reducible
  [^java.io.BufferedReader rdr]
  (reify clojure.lang.IReduceInit
    (reduce [this f init]
      (try
        (loop [state init]
          (if (reduced? state)
            @state
            (if-let [line (.readLine rdr)]
              (recur (f state line))
              state)))
        (finally
          (.close rdr))))))

Count the length of each ‘split’

1
2
3
4
5
6
7
8
9
(require '[clojure.string :as str])
(require '[clojure.java.io :as io])

(into []
      (comp
       (mapcat #(str/split % #";"))
       (map count))
      (lines-reducible (io/reader "/tmp/work.txt")))
;;=> [1 2 1 6 4 4 6]

Sum the length of all ‘splits’

1
2
3
4
5
6
7
(transduce
 (comp
  (mapcat #(str/split % #";"))
  (map count))
 +
 (lines-reducible (io/reader "/tmp/work.txt")))
;;=> 24

Sum the length of all words until we find a word that is longer than 5

1
2
3
4
5
6
7
8
9
10
11
12
(transduce
 (comp
  (mapcat #(str/split % #";"))
  (map count))
 (fn
   ([] 0)
   ([sum] sum)
   ([sum l]
    (if (> l 5)
      (reduced sum)
      (+ sum l))))
 (lines-reducible (io/reader "/tmp/work.txt")))

or with take-while:

1
2
3
4
5
6
7
(transduce
 (comp
  (mapcat #(str/split % #";"))
  (map count)
  (take-while #(> 5 %)))
 +
 (lines-reducible (io/reader "/tmp/work.txt")))

Read https://tech.grammarly.com/blog/building-etl-pipelines-with-clojure for more details.

Permalink

Datomic Cloud

Datomic on AWS: Easy, Integrated, and Powerful

We are excited to announce the release of Datomic Cloud, making Datomic more accessible than ever before:
Datomic Cloud is a new product intended for greenfield development on AWS. If you are not yet targeting the cloud, check out what customers are saying about the established line of Datomic On-Prem products (Datomic Pro and Enterprise).
Datomic Cloud is accessible through the latest release of the Datomic Client APITo learn more, you can:
We would love your feedback! Come and join us on the new developer forum.

Datomic Cloud on the AWS Marketplace

Permalink

Parsing text with clojure.spec

This week I’ve been working on JUXT’s tick library.

Tick is a time library, and has a few playful functions for working out dates like Easter Sunday. But as part of a major overhaul of the library, towards a serious production-ready library, we’ve decided it needs to read holiday dates from official information sources (not all holidays can be computed!) and for that, it needs to be able to read the iCalendar format.

iCalendar is specified in RFC 5545, and like many RFCs, grammar rules are defined in a syntax known as Augmented Backus-Naur Format (ABNF), which is a lot more intuitive than it sounds.

iCalendar files are made up of a set of 'content' lines defined by the following rule:

contentline = name *(";" param ) ":" value CRLF

This tells us that each line has to begin with a name, optionally followed by some parameters, then a colon followed by a value, ending with a line-ending.

Of course, we now need to know precisely what names, parameters and values are. These are also defined for us (but for the purposes of this article I’ve simplified somewhat):

name = iana-token
iana-token = 1*(ALPHA / DIGIT / "-")
param = param-name "=" param-value
param-name = iana-token
param-value = iana-token
value = iana-token

Parsing

Having split up our iCalendar file into content lines (and there are some folding rules in doing that which we’ll ignore), we now need to parse them.

I’d like our solution to have the following properties:

  1. Be easy to understand and update (my simplifications aren’t going to go the distance)

  2. Be reasonably performant

  3. Be done by the time I get to the JUXT office (I’m writing this on my morning bus journey)

We could start by trying Regular Expressions, which are well provided for in Clojure and Java. But regexes don’t cope well with any degree of complexity. Our 'zero-or-more parameters' rule is going to trip us up.

At this point, I’d usually roll up my sleeves and be reaching for tools like instaparse in Clojure (or Antlr in Java) and paging-in all that Computer Science 101 stuff about LALR(1) context-free grammars, blah blah blah. But I’m on the bus and have rather intermittent Internet connectivity. Is there a quicker/better way?

Turns out, tick already has a dependency on clojure.spec, so can I use that?

Let’s start creating a spec for iana-token.

(require '[clojure.spec :as s])

(s/def ::iana-token
  (s/+ (set "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890-"))

;; Let's try it
(s/conform ::iana-token (seq "DTSTART")) => [\D \T \S \T \A \R \T]
(s/conform ::iana-token (seq "&64*")) => :clojure.spec.alpha/invalid

Gosh, that wasn’t so hard.

Now let’s see if we can hand-write our rule for contentline:

(s/def ::content-rule
  (s/cat
    :name ::iana-token
    :params (s/*
              (s/cat
                :semi #{\;}
                :param-name ::iana-token
                :equals #{\=}
                :param-value ::iana-token))
    :colon #{\:}
    :value ::iana-token))

Notice how readable and intuitive that code is.

Let’s try that rule with some a real iCalendar contentline:

(s/conform
  ::content-rule
  (seq "DTSTART;TZID=US-EAST:20180116T140000"))
=>
{:name [\D \T \S \T \A \R \T],
 :params
 [{:semi \;,
   :param-name [\T \Z \I \D],
   :equals \=,
   :param-value [\U \S \- \E \A \S \T]}],
 :colon \:,
 :value [\2 \0 \1 \8 \0 \1 \1 \6 \T \1 \4 \0 \0 \0 \0]}

clojure.spec has broken the string up into just the pieces we need. For our parameters, it’s given us a collection, because there could be a few of them.

As a quick example, let’s extract the value of first parameter.

(apply str (get-in … [:params 0 :param-value]))
=> "US-EAST"

Voila!

Conclusion

The fact that clojure.spec—which was designed for a different purpose—seems so adept in the area of parsing is a surprise, but it’s a sign that there’s a lot of power in this little library. More evidence that where many languages veer off into the minutia, Clojure gets the job done.

The performance on my laptop isn’t bad (~500µs), and will parse a 5000-line iCalendar file in couple of seconds, which is within my requirements.

Looks like I’m done, and my bus has arrived!

(With thanks to @thegeez for this gist which inspired me to try clojure.spec to solve this problem.)

Permalink

Neanderthal and friends support CUDA 9, Java 9, and Clojure 1.9

Uncomplicate libraries have just got a nice update and are ready for the latest underlying platform releases: CUDA 9, Java 9, and Clojure 1.9.

CUDA

ClojureCUDA now supports the latest CUDA 9.1. That should eliminate the trouble of downgrading your CUDA installation to CUDA 8. If you install the latest CUDA toolkit, everything should work smoothly. If you have CUDA 9.0, it will work. I am not sure about CUDA 8 - even that should work, but I haven't bothered to try it, since by now CUDA 9 is what most people install by default.

Of course, CUDA 9 is also suported in Neanderthal.

Java 9

ClojureCUDA, ClojureCL, and Neanderthal use some JVM internals to manage memory efficiently, which can clash with Java 9 modules. Fortunately, this has a solution! Include "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" in your JVM options if you use these libraries with Java 9, and everything will work smoothly, with no code changes required!

Clojure 1.9

Clojure cares a lot about backward compatibility, so this was not an issue. All Uncomplicate libraries work with Clojure 1.9 without change.

There will be even more

I have some tutorial articles on GPU programming with Clojure in my warehouse. Now that I don't have to explain the readers that they have to keep CUDA at the previous version, it seems to be a good moment to begin releasing them :)

Permalink

Interactive GPU Programming - Part 1 - Hello CUDA

Who wouldn't like to speed up their programs by thousands of times? When we reach the limits of software optimizations, the only hope is faster hardware. Today, that means parallel programs that run on Graphical Processing Units (GPUs). Thanks to the recent highly publicized deep learning and artificial intelligence advances, everyone has heard about Nvidia's CUDA - an environment and set of C++ oriented tools that transform your GPU card into a sort of a desktop supercomputer. Yes, that is interesting, and we want to get on board! But we would also like an interactive programming experience. I like power and performance, but I also like to experiment and play with flexible code interactively.

Listening to hunting stories of Google engineers is the furthest most programmers who work with higher-level platforms such as Java, .Net, Ruby, Python, Javascript, etc. will go. Except for the Pythonistas. They will get a bit further despite not knowing how to write any GPU-aware code, since there are DL libraries such as TensorFlow, Caffe, or PyTorch, which run on GPUs under the hood. However, even that depends on the other people writing the code that solves your problem. Sometimes there is such code, but more often there isn't any, and you are the one to write it.

The whole GPU programming ecosystem is not very approachable. Not only that you have to learn about parallel hardware and algorithms, but you have to find your way through the complex maze of different drivers, operating systems, and modified C++ build chains that is in itself the endless jungle of nitty-gritty details and incompatibilities.

There is a saner way I intend to show you in this series of blog posts. This better way relies on the environment that is:

  1. platform-independent (you access it from Java Virtual Machine)
  2. easy to set up and start with (relative to the "official" C++ way at least)
  3. interactive (gives an instant feedback for each tiny piece of code you write)
  4. easy to experiment and play with

And the best of all: it offers the access to almost the full power of your hardware, without hiding the details that do matter.

On top of it, there is a bonus: these tools work on Intel and AMD hardware too, so you are not constrained with Nvidia's proprietary ecosystem!

What also demonstrate how easy and malleable this approach is that this whole page has been automatically generated from a live org-mode session (this is a kind of interactive notebook), connected to a live Clojure REPL (read-eval-print-loop) session. The immediate output from the code execution is shown in the text as-is.

This first post shows the introductory Hello World example, and gives a glimpse of a typical CUDA application. In the following articles - this is only the first part of a series - I will explore major CUDA and OpenCL topics in detail.

Set up the environment

Hardware

The one thing we can't continue without is the hardware: ideally, you'd have a GPU in your machine:

  • Nvidia's GPU (supports CUDA)
  • AMD's GPU (you'd have to use OpenCL instead)
  • Intel's CPU and GPU (also supports OpenCL)

We access these hardware devices through the appropriate device drivers. Our everyday CPU programs do not need any special support of this kind only because the whole operating system is based around the specific architecture of the CPU (usually x86/amd64) and is a sort of the device driver for the CPU. So, you'd install the appropriate device drivers from the manufacturer of your GPU. Note that the generic drivers that come with your operating system are usually only capable of basic 2D display, and do not support GPU computing. Install the "real" drivers from Nvidia or AMD.

Toolkits

On top of that we need the actual GPU computing tools:

  • Nvidia: install the CUDA toolkit
  • AMD or Intel: install the support for OpenCL for your operating system

Clojure and Java

I assume that you already have a recent Java Development Kit (JDK). I also assume you like or are willing to try the fantastic Clojure programming language that compiles directly to the Java bytecode. There are many books, newbie tutorials, and conference talks that can help you quickly learning enough Clojure to be dangerous. I thing the free book Clojure for the Brave and True is very approachable and fun. You'll need to know at least basic Clojure to run this code yourself. If you do not have time to learn it right now, though, you'll still be able to read and understand examples, since Clojure has trivially simple syntax.

So, you'll create a new Clojure project (or use an example) that includes the actual library for GPU computing that I am talking about:

You can use any Java IDE, or any text editor + a Java environment that you like that has Clojure support (most do). I recommend the fantastic (if a bit unusual in Java community) Emacs + CIDER, with a convenient easy setup via Prelude.

Ready?

It seems there are lots of steps when I describe it like this, but that's because I've tried to support all kinds of cases. Most computers already have the drivers, many have CUDA Toolkit, most already have Java Virtual Machine, and Clojure is technically just a Java library we include in our Java projects through Maven (or a nicer Clojure build "mavens" Leiningen or Boot). Many readers have already been ready :)

This article uses CUDA, while the next one will show the code for OpenCL (most of the narrative applies for both).

Handle the GPU device(s)

At the beginning, we'll require the namespaces that contain functions for GPU programming. Functions that work with CUDA are in the uncomplicate.clojurecuda.core namespace of the ClojureCUDA library, while a few interactive compilation functions that we'll need are in the uncomplicate.clojurecuda.nvrtc namespace.

(require '[uncomplicate.clojurecuda.core :refer :all]
         '[uncomplicate.clojurecuda.nvrtc :refer :all])

CUDA environment has to be initialized before use. After that, I query the environment to see how many Nvidia GPU cards I have:

(init)
(device-count)
1

Now that I know that there is a CUDA-capable GPU device in my computer, I can grab it by the handle:

(def my-nvidia-gpu (device 0))

I've stored the handle of my GTX 1080 in the my-nvidia-gpu global var. Storing data in global vars is fine and convenient for tutorials, but should not be done in the "real" programs. There is support in ClojureCUDA for nice functional approach to writing programs, though, so this technique is fine in this case.

What is the type of this object and how does it look like?

my-nvidia-gpu
#object[jcuda.driver.CUdevice 0x460cb871 "CUdevice[nativePointer=0x0]"]

It's an instance of jcuda.driver.CUdevice, which itself stores a reference to a CUdevice native pointer. This is not something you should concern yourself with most of the time, but there is a reason why I'm showing it here:

  • ClojureCUDA shields you from the low-level details and lets you concentrate on the things that does matter;
  • Sometimes what matters is low-level, and you should be able to control it for full effectiveness;
  • You should be able to use existing CUDA-based books, articles, and documentation to learn and properly use GPU programming.

Although ClojureCUDA is fairly pleasant and high-level, it is designed to directly correspond to familiar CUDA constructs. It helps when it can, and moves out of the way when necessary!

Congratulations, there is the hello world! Just kidding. Although this is the code that interacts with the driver, I hope you'd expected some "real" code that actually compute something on the GPU. You won't be disappointed. I would still like to point out that you have witnessed something cool, though: a real code, that lets your write CUDA-ish stuff interactively, without even knowing what a basic CUDA program looks like. The best of all, we are able to experiment and get the instant feedback, which is the key thing for learning how a CUDA program looks like and what it can do in the first place!

Working in the context

The first thing we have to do in all CUDA programs is to create the context through which we will be controlling the device(s). It is a step analogous to setting up a connection with a database in traditional programs.

CPU can execute multiple programs simultaneously, and GPU can as well. An analogous of a CPU proccess is a GPU context. It manages the life cycles of various CUDA objects, such as: memory, modules (program code), streams (a kind of threads), events, etc. It's a management unit that sets up an environment in which your device will execute your programs.

The default context setup can be easily created with ClojureCUDA:

(def ctx (context my-nvidia-gpu))
ctx
#object[jcuda.driver.CUcontext 0x32f800ef "CUcontext[nativePointer=0x7f40800902e0]"]

As with CUdevice, when you need a specific information about how to use contexts, there is a convenient fallback to the official literature; just look for CUcontext.

Manage the memory on the GPU device

To utilize the capacity of GPU computing cores, we need to feed them enough data at enough speed. GPU has its own fast memory and dedicated integrated memory controller(s) on board, but, on top of it, it does offer a direct way to control that memory.

An important thing to notice is that, unlike the CPU, which is an all-round skillful commando, the GPU is good at very narrow set of relatively simple tasks, and most of these tasks revolve around numerical operations. The memory management will usually deal with supplying huge arrays of primitive numbers to huge number of parallel workers, and taking care that each worker work on the appropriate part of a huge raw array.

When thinking about CUDA memory, think about huge raw byte arrays. Additional libraries such as Neanderthal would add more structure on top of this, but CUDA API typically manages raw bytes of memory. Here's how:

(current-context! ctx)
(def gpu-array (mem-alloc 1024))

This creates the handle to a chunk of 1024 bytes of global GPU memory called linear memory in CUDA terminology. It is nothing more than a 1 dimensional array of raw bytes in the main (global) memory on the GPU board, as opposed to graphics-oriented 2D, 3D, and texture memory.

gpu-array
#object[uncomplicate.clojurecuda.core.CULinearMemory 0x3b8c8905 "uncomplicate.clojurecuda.core.CULinearMemory@3b8c8905"]

Transferring the data from the main memory to the GPU memory

We have defined the memory on which to unleash our many GPU cores. But, before going further, we should think about how to get the data there in the first place, and how to return the result. I assume that the data is not on the hard disk or in the CSV file available over the internet, but that we have already parsed it and loaded it in a float array in our main Clojure program in the Java virtual machine:

(current-context! ctx)
(def main-array (float-array (range 256)))
(take 10 main-array)
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0

We have created a float array of 256 numbers. Each float takes 4 bytes, so its size of 1024 bytes matches the capacity of the GPU array that is the destination for this data.

In GPU computing terminology, the memory on the GPU is called device memory, while the main memory is called host memory.

Let's do the transfer!

(current-context! ctx)
(memcpy-host! main-array gpu-array)
#object[jcuda.driver.CUcontext 0x32f800ef "CUcontext[nativePointer=0x7f40800902e0]"]#object[uncomplicate.clojurecuda.core.CULinearMemory 0x3b8c8905 "uncomplicate.clojurecuda.core.CULinearMemory@3b8c8905"]

To convince you that the data have really been transferred to the GPU memory, I'll transfer it back into a new empty float-array:

(take 12 (memcpy-host! gpu-array (float-array 256)))
0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 11.0

Now you believe me the data is on the GPU!

Compute something already!

We have the data in memory, but we do not have the program to crunch it. There's a lot to talk about here, but since this is a hello world article after all, I'll be as brief as possible.

In GPU programming terminology, there are two kinds of code:

  • host code that runs on the CPU and do various management calls to the device driver (such as memcpy-host!)
  • kernels that run on the GPU cores

We write the host code in Clojure, while the kernels are written in CUDA C. Our hello world example will increment each element in the array, in parallel of course. The kernel looks like this:

extern "C"
__global__ void add(int n, float *a) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        a[i] = a[i] + 1.0f;
    }
};

Keyword __global__ indicates that this is a CUDA kernel. Otherwise, it is a plain C function, that has some parameters, and calls some operators on arrays. Another CUDA-specific things are blockIdx.x and similar fields. Remember that each parallel worker will execute this same code at the same time. Thus, each worker need to position itself in the whole squadron. CUDA environment will make sure that each unit ("worker") will get this data populated. In this hello world case, each worker will be able to compute its ID, and work only on one cell of the array. It will read the value of that cell, add one, and write it to the same location in the global GPU memory.

How will the GPU know which kernel is going to be executed on which array in which order? That's the job of the host code:

(def kernel-source
      "extern \"C\"
         __global__ void increment (int n, float *a) {
           int i = blockIdx.x * blockDim.x + threadIdx.x;
           if (i < n) {
             a[i] = a[i] + 1.0f;
        }
       };")

(current-context! ctx)
(def hello-program (compile! (program kernel-source)))
(def hello-module (module hello-program))
(def increment (function hello-module "increment"))
(launch! increment (grid-1d 256) (parameters 256 gpu-array))
(def result (memcpy-host! gpu-array (float-array 256)))
(take 12 result)
1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 11.0 12.0

Each element of our array has been incremented! Cheers!

Why not write kernels in Clojure, or another higher-level language? It is possible to implement, but in my opinion it does more harm than good. Kernels are tiny, not very complex compared to the typical C++ code, and they match the problem of floating point computations really well. On the other hand, the host code can be long and complex, and C++ is notoriously hard to build. Therefore, simplifying host code by wrapping it with Clojure makes sense, while kernels - not so much.

Keep the environment clean!

The key to fast code is managing the scarce resources on the GPU. That means manually managing the key resource: memory!

ClojureCUDA has macros that can do the bookkeeping for us, which I'll show you in the following articles, but for now, we'll do that in the plain CUDA style: by freeing them by hand. Various specific CUDA functions are avilable for this task, and we can uniformly access them with the release function:

(require '[uncomplicate.commons.core :refer :all])
(release gpu-array)
(release hello-module)
(release hello-program)
(release ctx)

What follows next

Now that we have broken the ice by creating and running a complete CUDA program from scratch interactively in the REPL, I'll take time to get into the details of how to handle memory, kernels, transfer, contexts, and all these specialized CUDA topics. We'll also see how to integrate the custom low-level CUDA code we write with some powerful Clojure number crunching libraries such as Neanderthal.

But first, we'll repeat this hello world in OpenCL, for those of us who have AMD or Intel hardware, or just prefer to use a more open and standards-based solution with Nvidia.

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.