Senior Developer

Senior Developer

Indaba Music | New York
remote
Create tools for musicians, producers, songwriters, remixers.
$100000 - $125000

About Us

Indaba Music is an online community of musicians, producers, songwriters, remixers, and other music creators. We provide opportunities to our community to create music everyday for some of the biggest artists and brands in the world.

New York, NY

What we use

Languages: Clojure/JS/Ruby

Other buzzwords: Redis/Elasticsearch/Kafka/Onyx/Ansible/Hashicorp/React

Types of projects we do

  • Built a community-based competition/contest platform that connects musicians with major artists and brands for the creation of original songs as well as remixes. indabamusic.com
  • Built the world's premiere royalty free sample library. converse.com/samplelibrary
  • Building unprecedented platform for creating, distributing and monetizing derivative works (e.g. sampled songs, remixes) by bringing content, composers, musicians, and producers together.

What we expect

  • 4+ years experience in shipping web apps
  • Proficient in JS
  • Interest in Functional Programming
  • Interest in SOA architectures
  • Interest in building and scaling web architectures

What we'd like

  • Experience with react frameworks (JS or CLJS)
  • Proficiency in SQL
  • Interest in real-time event streams
  • Clojurescript experience
  • iOS experience
  • Deep interest in music (creating, or consuming)

What should interest you

  • Building services in Clojure & possibly Clojurescript
  • React frameworks like redux/om-next/reagent
  • Companies with a family feel
  • Being self-motivated and enjoying exploring problem spaces and solutions
  • Simplifying system architectures
  • Task Automation
  • Data driven product decisions
  • Helping musicians to create music, develop their craft, and make money

Permalink

Clojure Gazette 183: The Magic of Abstraction

Clojure Gazette -- Issue 183 - July 25, 2016


The Magic of Abstraction
Read this email on the web
Clojure Gazette
Issue 183 - July 25, 2016

Hi Clojurists,

We can solve any problem by introducing an extra level of indirection.

-- Butler Lampson (often attributed to David Wheeler)

I've been working in JavaScript and PHP quite a bit recently. I'm not a fan of either language. And yet, each has everything I need to make a small space for myself in which I can feel comfortable. Of course, I'm talking about named, first-class functions as a means of abstraction. I create a new strata of functions which call built-in operations but solve some of the fundamental problems of those operations.

My strata is very reliable because I've designed it to solve exactly the problems I see. For instance, my functions can check for nulls. The layer above it now doesn't have to be concerned with null checks. It's also shorter, easier to read, and has fewer bugs. As I debug my program, I strengthen that layer. For instance, if I do find a null pointer exception, I add a new check to a function in that layer. Over time, it becomes a more robust level of indirection. And I can imagine a time where I have solved the null pointer problem in its entirety in my code.

This idea is remarkable. How can I believe that I have solved Tony Hoare's Billion Dollar Mistake? Is it tremendous hubris on my part? I don't think so. I think I'm just taking advantage of the fundamental theorem of software engineering. And I think it's the second most important idea to come out of computer science. (I've talked about what I think is the most important idea in computer science before.)

Let's take a look at some code from Clojure's core. If you write Clojure, you're familiar with the seq abstraction, which is a function that converts things into Seqs. It's instructive to look at the implementation of seq and seqFrom.

At first glance, these functions looks awful to me. It's just a big case statement that switches on the type of the argument. But it's a perfect example: it's an abstraction written in lower level constructs that builds the unifying concept of the language. It makes iteration reliable in Clojure where it is not in Java. It's ugly code that makes a beautiful system.

I'm not a civil or mechanical engineer. Nor am I an architect. And I would love to be shown I am wrong about this, because I don't have much proof of this. But with that preface in place, I believe that Computer Science produced the notion that a layer of abstraction can make a system more reliable. Engineering in general relies on building reliable things out of reliable parts. For instance, making a reliable building requires reliable bricks laid in the right way. To scale the building, the bricks and laying need to made to even greater tolerances.

Perhaps a better example is the pre-internet telephone network. At the time the ARPANET was being built, the telephone network was expanding and scaling by increasing the reliability of each part. To scale up the distances and reduce the number of dropped calls, they were investigating better materials to make each part transmit data with higher fidelity. It was working. But it cost more as the network scaled up and progress was slowing down.

The ARPANET was different. It assumed that the physical network was unreliable. With that premise, how would one ensure reliable communication? A few levels of indirection later (known as the TCP/IP stack) and you have a totally scalable and reliable network. Packets are lost, lines are cut, routers fail, hard drives crash, and yet you can still load the web page. The Internet Protocol is like an inflection point that transforms bad physical networks into good communication networks, eliminating whole classes of problems.

We use this idea every day when we program. Our modern computers have a machine code that they can run. Hand-written machine code does not scale. We are usually programming in a stack of abstraction layers above it that help us write more complex programs than would be possible in machine code. There's an optimism embedded in this idea: no matter how bad your stack is, you can abstract away the problems with suitably designed levels above it.

It is with this optimism that I approach work in WordPress (PHP) and React Native (JavaScript). I assume the worst. Nulls exist so check values before doing a method call. Asynchronous calls may never finish so add timeouts. Then embed the solutions in levels of abstraction.

Does this "process" of assuming the worst generalize? Erlang seems to think so. Assume messages aren't delivered reliably (even on a single machine). Assume messages don't follow the correct format. Assume that bad data will be passed around. Assume there will be bugs in your program. With all of these premises, how should one program? Build layers of abstractions, where the reliability comes at the top of the stack. Just as Lisp embodies the most important idea in Computer Science, Erlang embodies the second most important idea.

So this is my vote for the number two idea. It's the power of abstraction to turn crap into gold. It seems to me that we should be cataloguing known limitations of systems (the halting problem, the CAP theorem, etc.) and known layers of abstraction to mitigate them. These are the "real design patterns" that could help us build better systems. What are these limitations? Are people already doing this? What would the abstraction levels look like? And how can we use pessimistic assumptions about the lower-level stack to eliminate them at a higher level?

I'm looking forward to hearing your ideas.

Rock on!
Eric Normand <eric@lispcast.com>

PS Want to get this in your email? Subscribe!
PPS Want to advertise to smart and talented Clojure devs?

Permalink

Slides and Video of programming languages talk at TheLeadDev

Tweet Hello, Recording of my programming languages talk at The Lead Developer conference this year is available now. It was a well run, and intimate conference despite nearly 500 attendees this year. It’s quite different from many conferences I have attended the last few years as it focuses on soft skills and leadership rather than specific technologies or

Permalink

Reacting to changes with Firebase, and how to interop

Transcript of Firebase screencast.


CoderX here.

Firebase is a hosting service that provides an authenticated realtime database. The database acts like a JSON tree that pushes changes out to clients. Clients can write to and subscribe to parts of the object. This subscription model fits nicely with reactive style pages.

Reacting to changes with Firebase, and how to interop


In this screencast we will build a website that allows users to login, subscribe to public data, and write updates. We will
Write to and read from Firebase
Listen to change notifications
Add user authentication
Build a UI to make use of the data
Deploy the project
Examine the impact of optimized compilation
Option 1: Deploy with whitespace optimization
Options 2: Use or provide externs
Option 3: Use CLJSJS as a blueprint
Review JavaScript interop

Write to and read from Firebase


Let’s start a new project. Firebase provides a JavaScript API (https://firebase.google.com/docs/web/setup).
Let’s load it in our index.html page under resources.

script src="https://www.gstatic.com/firebasejs/3.1.0/firebase.js"

Sign up for Firebase using your Google account. Click on console then create a project (https://console.firebase.google.com/). We need to use these API keys in our code. Let’s start a firebase namespace, and put some initialization code in it.

(defn init []
(js/firebase.initializeApp
  #js {:apiKey "USE_YOUR_KEY"
       :authDomain "YOUR_PROJECT.firebaseapp.com"
       :databaseURL "https://YOUR_PROJECT.firebaseio.com"
       :storageBucket "YOUR_PROJECT.appspot.com"}))

js/ is a special form to allow you to access root objects in JavaScript. The firebase object is defined by the api we included earlier, and contains an initializeApp function.
#js specifies a JavaScript object literal.

Add a database to the project. Modify the database access rules to allow anonymous read on everything, and write to test.

{
 "rules": {
   ".read": true,
   "test": {
     ".write": true
   }
 }
}

In our code, we will write a value to a key.

(defn db-ref [path]
(.ref (js/firebase.database) (string/join "/" path)))

(defn save [path value]
(.set (db-ref path) value))

(save ["test"] "hello world"))

A ref represents a specific location in your database and can be used for reading or writing data. You can reference the root, or a child location in your database by passing a path. Writing is done with the set method. We can check that the value was saved in the database from the Firebase console. We can now read the value using once.

(.once
(db-ref path)
"value"
(fn received-db [snapshot]
  (prn "Got:" (.val snapshot))))

Great, we can read and write data.

Listen to change notifications


The neat thing about Firebase is that it will notify clients as the database changes. The Firebase real-time database is a cloud-hosted database. Data is synchronized to every connected client.

Listening can be done with the on method, which is just like once except that it continues to call back to us whenever the source data changes. In order to make use of the new data in our application we want to watch for changes. Ratoms are convenient ways for the UI to respond to change. We can reset a ratom when new data arrives. Here we meet a common interop issue.
Components get added and removed from the DOM. We don’t want to have multiple subscriptions pile up. So we need a way to disconnect a listener when the component is removed. This is a lifecycle concern; the component needs to clean up a resource. Reagent’s third form is the appropriate solution for lifecycle concerns. We create a class that has an unmount handler that detaches our listener.

(defn on [path f]
(let [ref (db-ref path)
      a (reagent/atom nil)]
  (.on ref "value" (fn [x]
                     (reset! a (.val x))))
  (reagent/create-class
    {:display-name "listener"
     :component-will-unmount
     (fn will-unmount-listener [this]
       (.off ref))
     :reagent-render
     (fn render-listener [args]
       (into [f a] args))})))

This is a component that takes a path and a component as an argument. The component passed should expect a ratom to observe as the first argument. Once we have defined on we can observe values very concisely.

[on ["test"] (fn [a] [:div @a])]

While such a component is mounted in the DOM it will respond to any changes to the specified path into the database by reacting to a ratom bound to the values received. When a user changes a value, the other user sees the new data. We didn’t need to write any explicit communication code. The changes flow down as they happen.

Pretty cool.

Add user authentication


In the Firebase console select authentication providers and enable Google. To check the auth status we hook up to onAuthStatusChanged.

(.onAuthStateChanged
(js/firebase.auth)
(fn auth-state-changed [user-obj]
  (reset! user {:photoURL (.-photoURL user-obj)
                :displayName (.-displayName user-obj)
                :uid (.-uid user-obj)}))
(fn auth-error [error]
  (js/console.log error)))

When the user is logged in, we’ll show their profile picture as a button to logout. Let’s make a login button that calls loginWithPopup.

(defn login-view []
[:div
 {:style {:float "right"}}
 (if-let [{:keys [photoURL displayName]} @firebase/user]
   [:span
    [:button.mdl-button.mdl-js-button.mdl-button--fab.mdl-button--colored
     {:on-click
      (fn logout-click [e]
        (firebase/logout))
      :title displayName
      :style {:background-image (str "url(" photoURL ")")
              :background-size "cover"
              :background-repeat "no-repeat"}}]]
   [:button.mdl-button.mdl-js-button.mdl-button--raised.mdl-button--colored
    {:on-click
     (fn login-click [e]
       (firebase/sign-in-with-popup))}
    "Login with Google"])])


Let’s try logging in… it works!

Now modify the database access rules to require user authentication.

{
 "rules": {
   ".read": true,
   "test": {
     ".write": true
   },
   "users": {
     "$uid": {
       ".write": "$uid === auth.uid"
     }
   }
 }
}

Writing to the users uid path now requires the writer to be authenticated.

We are limited to writing JSON values. But we can encode more complex data models. For this project we will be using Datascript as the model. Save the model using  pr-str and load it with edn/read-string.

(pr-str @conn)
(edn/read-string {:readers d/data-readers} in)

Build a UI to make use of the data


Users will add short titles of topics that interest them. I’ll render the titles as nodes in a graph.
The nodes can be linked together. Users can put their faces on nodes to vote for them.

Deploy the project


The Firebase command line tool deploys our application. It is a NodeJS package that you can install by typing `npm install firebase`. Execute `firebase init` in your project directory to create configuration files. Database rules are specified in the database.rules.json file. The firebase.json file specifies a folder of files to copy for hosting. https://firebase.google.com/docs/hosting/deploying

Make a build task that creates a minified build to a `public` folder that contains our HTML and compiled JavaScript. Check that everything is working with `firebase serve`, which serves the public folder locally. To push the folder to the host, type `firebase deploy`. I put the release build and deployment step in a script and readme so I don’t forget it.

Our site is online, open to the public, and supports real-time collaboration.

Examine the impact of optimized compilation


While developing it is convenient to use no optimizations because compilation is far faster. The main disadvantage of no optimizations is that the output goes into many separate files. When deploying, it is more convenient to have a single JavaScript output file.

Advance mode compilation does aggressive renaming, dead code removal, and global inlining, resulting in highly compressed JavaScript. This power comes with some restrictions. Advanced compilation mode makes assumptions about the code that it optimizes, and breaking these assumptions will result in errors. This is a problem for interop with JavaScript libraries. JavaScript libraries must be accompanied by extern definitions, or the aggressive renaming will cause interop calls to be undefined. https://developers.google.com/closure/compiler/docs/api-tutorial3

Option 1: Deploy with whitespace optimization


I encourage you to avoid worrying about the effects of advanced optimization on interop by not using it at all. Configure your deployment build to use optimizations whitespace instead of advanced. This will produce a single output file suitable for deployment. The only downside is that your js file will be larger than it could otherwise be (for this project the output is 1.7Mb). For most purposes this is not noticeable.

:compiler {:output-to "resources/public/js/compiled/firefire.js"
          :main firefire.core
          :optimizations :whitespace}

Options 2: Use or provide externs


JavaScript functions and variables get aggressively renamed, causing interop to fail.
Extern definitions prevent renaming, so that interop will work correctly. For advanced compilation we need both the JavaScript source file and an externs file. An externs file is a JavaScript file that declares all the public interfaces.

firebase.initializeApp = function(options, name) {};

Externs files can also contain documentation annotation. Many libraries get distributed with externs. If you use npm to retrieve a library, check for an externs directory. If externs are not provided, you may be able to generate them using a tool like Michael’s (http://jmmk.github.io/javascript-externs-generator/).

Pro tip, you can also use the original JavaScript as it’s own externs file. You can turn off the warnings it will produce with configuration specified in David’s blog post here:

Declare foreign-libs is done by specifying an edn file like so:

{:foreign-libs
 [{:file "react/react.js"
   :file-min "react/react.min.js"
   :provides ["com.facebook.React"]}]
:externs ["react/externs.js"]}


You can specify a preamble for your build instead, which will prepend a JavaScript file to your final compiled output. But as we just saw, specifying a foreign lib is not much more work. In either case the important part is the externs.

In practise I recommend avoiding this option entirely in favor of using CLJSJS.

Option 3: Use CLJSJS as a blueprint


CLJSJS is a standard way to package the foreign-libs definitions file with the externs and the JavaScript library so that you can use them as a dependency.

There is already a package defined for Firebase. To use it we just add it to our dependencies, and require it in our code. (Don’t forget to remove the include from your HTML file).

But what if we need a newer version? Let’s take a closer look at the definitions for the Firebase package. https://github.com/cljsjs/packages/blob/master/firebase/build.boot

The javascript and externs come from a NPM distribution, which is referenced by a version number. At this point it should be clear that all there is to a CLJSJS package is specifying the location of the JavaScript, the externs, and the namespace to be used when requiring.

To package a newer version we can checkout this code, bump the version, and install it locally with `boot package install target`.

If the library we want to use doesn’t have a CLJSJS package, we can copy the definitions from another package. For example if the library we want to use is distributed on NPM, we could copy the Firebase package as a starting point for how to wrap it. CLJSJS provides a well documented blueprint for how to wrap JavaScript libraries to be compatible with advanced compilation.

Review JavaScript interop


Interop with JavaScript from ClojureScript is similar to interop with Java from Clojure.
You call native methods using the dot or dot dot form.

(.method object arg1 arg2)
(. object method arg1 arg2)
(.. object method1 (method2 arg1 arg2))

JavaScript has a global root object. To access the global root in ClojureScript, use the special js prefix.

js/window
(.setEventListener js/window (fn [e] (prn “hi”)))
(js/window.setEventListener (fn ...)))

To access properties, ClojureScript differs from Clojure slightly in that you must use a dash to indicate property access. JavaScript does not distinguish properties from methods, so the dash is required to explicitly designate property access.

(.-property object)
(. object -property)
(.. object -property1 -property2)
(set! (.-property object) value)

Array access is done with aget and aset.

(aget array index)
(aget array index2 index2 index3)
(aset array index value)

You can chain access forms together.

(set! (.-foo (aget array index1)) value)

Object constructors are the same as Clojure. You can use constructors from the global JavaScript root.

(js/THREE.Vector3. 0 0 0)
(new js/THREE.Vector3 0 0 0)

Modules can be referenced and imported from the Closure Library.

(ns example
 (:import [goog.event KeyCode]))

JavaScript objects can be created with js-obj or by using the special #js reader literal. These forms are not recursive. When nesting objects specify the children as literals.

(js-obj "key" "value")
#js {:key "value"}

I recommend using ClojureScript data literals as much as possible, and only using JavaScript objects for interop with third party libraries.

There are transformation functions to convert from ClojureScript data structures to JavaScript objects. These are both recursive.

(clj->js)
(js->clj)

You can optionally keywordize-keys but keep in mind that not all strings can be keywords.
There is a penultimate escape hatch form js* that takes raw JavaScript.

(js* "some(javascript);")

JavaScript allows anything to be thrown, so the ClojureScript catch form has a special term default which will match anything. Use this to catch unexpected values being thrown.

(try
 (/ :1 2)
 (catch :default ex))

You can annotate your functions with export to prevent them being name-mangled during advanced compilation. They will then be easily consumable from JavaScript.

(defn ^:export f  []
 (+ 1 2))

Conclusion


Firebase is a convenient hosting service you can leverage from ClojureScript. The free starter plan provides a database, hosting, and authentication. ClojureScript allows you to work with JavaScript APIs with interop code. For development you can include a JavaScript file from your HTML page. For deployment I recommend using whitespace optimization to avoid the build complications that advanced compilation introduces. When your project is mature and would benefit from shrinking with advanced mode compilation, you will need externs defined. CLJSJS is the best way to specify externs, and provides wrappers for many libraries already.


Until next time,
Keep coding.

Permalink

How Cognician uses Onyx

Cognician, the ed-tech company I co-founded in 2010, uses the Clojure stack:

  • Clojure and ClojureScript.
  • Datomic, a distributed, ACID-transactional, relational, immutable database.
  • Onyx, a masterless, distributed computation framework.

All of the actions listed below use the same set of immutable data-structures – good old maps, vectors, and sets – and they’re all plain old unadorned data – no special types!

  • Client interactions in the browser.
  • Network ops between the browser and API end points (with great transfer and parse performance, thanks to Transit).
  • Handling web requests.
  • Manipulating and storing data in Datomic.
  • Declaring and integrating computations in Onyx.

We have been using Clojure, ClojureScript and Datomic for several years. The most recent addition is Onyx, which we put into production in September 2015. We had some bumps along the way, but now that we’ve learned the requisite skills to run this powerful system suitably well, I’m incredibly happy with it.

A need for scalable, continuous event-stream processing

Around September 2014 we were planning the next version of our platform, and had a need for a horizontally scalable computation system; one which wouldn’t need a total rewrite in a couple years as we scaled up. It could not be batch-based, as we wanted to process events coming in from our users as we received them on a continuous basis - a facility known as ‘event stream processing’.

The only real option we had back then as a Clojure team, was Twitter’s Storm (now an Apache project). Storm is great in many ways, but one of them is not ease of development and debugging. The deployment story was quite complex, considering that until that time, all we had to do was put .jar files on servers behind AWS’ ELB. And, it used an old version of Clojure, which made it impossible to take advantage of new features in the language.

Despite all that, we were going to go ahead with it and spike a solution, to see how things go. Happily, before we made any significant headway, I saw Michael Drogalis’s first Onyx talk, which he gave at StrangeLoop 2014 (video).

Once I saw this video, I realised why I didn’t want to use Storm: its API is a special set of macros which encapsulates all the ‘glue’. But we’re in Clojure, we have something better: plain old data. And that’s what Onyx uses, for the most part; your actual workhorse ‘do-the-things’ logic lives as pure functions, and just about everything else is just data!

We’ve already learned that this is the best possible way to build an API. We’d been living this dream with Ring and Datomic for a while, and seeing how Onyx used the same great approach made it a natural fit for our context.

We decided to go ahead with it, and over the course of 12 months, along with the rest of our new platform, we built an event-stream processing system with Onyx.

Before I can describe how we use Onyx, I’m going to go over some of its key concepts.

Onyx’s API is mostly data

The Onyx data API is broken into several cleverly designed pieces (see the documentation site or the cheat-sheet for the full picture):

  • Peer configuration, which defines the shape of the overall system; how many peers (individual worker bees) you’re going to run, ZooKeeper (for coordination) and Aeron (for networking) configuration, and many, many dials and knobs for managing overall performance.
  • The Catalog, which defines the tasks that can make up an Onyx job. Think of this as a set of puzzle pieces before they’ve been assembled into a cohesive picture. There are many knobs and dials for controlling performance at the task level here, too. There are three kinds of task:
  • Inputs: typically queue consumers, such as Kafka, RabbitMQ, or Datomic (yep!). Core.Async channels work great too. Extensible; could be whatever you like.
  • Good old functions: pointers to your workhorse logic.
  • Outputs: typically some sort of persistence - SQL, S3, Datomic, etc. And yes, Core.Async channels too. Also extensible.
  • The Workflow, which arranges tasks from the catalog into a graph of inputs ⟶ functions ⟶ outputs. In the puzzle analogy, the catalog contains puzzle pieces, and the workflow is the completed puzzle.

These first three sets of data are required - you can build a fully functional (albeit quite simple) computation with just these. However, if you want to handle complexity, you’re going to need some additional tools:

  • Flow Conditions, which provide a way to run segments through a predicate function before allowing them to move from task to task. This allows us to stop work early, and generally control the flow of segments to tasks. Flow conditions also provide a nice way to handle exceptions in tasks.
  • Lifecycles, a way to manage the per-task and per-batch state for tasks, and also cross-cutting concerns like metrics and logging. We use these to send metrics to DataDog’s statsd agent. Another great way to keep actual business logic simple. Kind of analogous to Ring middleware.

We use both of these to great effect. Since we built our system, even more goodies have become available, which we’ve not yet had a chance to take advantage of: Windows and Triggers, which Michael describes far better than I can here.

In summary, they allow you to gather up segments into buckets (Windows) and do something of consequence with them (Triggers). One use we have for this in the future is to “debounce” events for a single user; that is, prevent processing events for that user more often than, say, every 5 seconds. This is a great way to reduce overall workload when processing at scale.

How Cognician uses Onyx

Our Onyx system, named Highstorm (which I lifted from Brandon Sanderson’s seriously fantastic The Stormlight Archive fantasy novel series), has several duties:

  • Calculate and store statistics and aggregates, so that web pages and offline analysis can be faster. As all of our data lives in Datomic, these values are 100% inferred; all the raw source data is in the database before Highstorm begins its work.
  • Create direct relationship caches between indirectly related Datomic entities that often appear in queries together, also to simplify queries and improve their performance.
  • Send messages, such as email.

Aside from the last of these, you could think of Highstorm as a sort of scalable, eventually-consistent database function system. Of course, it’s different to actual Datomic database functions in several very important ways, and should not be mistaken for those! But, even so, it’s a handy analogy.

Before we dive into the details of how we did this, I think it’s important to understand how we were able to fit Onyx into the overall infrastructure.

How Onyx fits into Cognician’s infrastructure

Typically, when integrating a new system, you’d decouple it from the rest of your infrastructure with a queue, and so that’s what we did. But we didn’t use a separate queue technology like Apache Kafka or RabbitMQ.

This is thanks to a great feature of Datomic’s, which is that every connected peer is aware of changes to the database in real time. They have to be, thanks to the way query works - see the green ’Live Index’ pictured in this Datomic architecture diagram.

Highstorm’s primary input task simply watches the Datomic transaction log, and puts each transaction log entry into the system to be processed as they become available. Then, whatever changes to be stored ultimately reach Highstorm’s output task as transaction data, ready to be transacted.

Onyx has a Datomic plugin which provides both the input and output tasks that Highstorm uses.

From an infrastructure perspective, this is great. Here’s a ‘before’ picture:

Web + Datomic + DynamoDB

We have a pair of web servers connected to a highly-available pair of Datomic transactors, and to DynamoDB for storage.

And here’s what it looks like with Onyx:

Web + Datomic + DynamoDB + Onyx

Note: web server lines removed for clarity. Also, I used https://cloudcraft.co/ to draw these diagrams.

Onyx only connects to Datomic the same places that the web peers do in the original diagram. That is, they’re also just Datomic peers. There is no separate queue or coordination mechanism between the web and worker tiers.

This is a crucial point: we’re applying the same lesson to architecture that we learned at the programming layer: make data your interface. Our web tier communicates with our worker tier only through data in the database!

To dive into the details a bit, we’ll look at how we send email with Highstorm, as it’s a great example of this data-as-interface pattern.

Data as interface between systems

When our system has to send email, such as when providing a user with a reset-your-password token, all the web server does is write a new ‘pending message’ entity to Datomic:

(ns your.app.messaging
  (:require [datomic.api :as d]
            [ring.util.response :as response))

(defn save-pending-message! [db-uri template params]
  @(d/transact-async 
    (d/connect db-uri)
    [{:db/id (d/tempid :db.part/user)
      :msg/uuid (d/squuid)
      :msg/template template
      :msg/params (pr-str params)
      :msg/status :status/pending}]))

;;;

(ns your.app.web.reset-password
  (:require [your.app.messaging :refer
             [save-pending-message!]]))

(defn handle-reset-password [db-uri email]
  ;; ... validate email, generate token ...
  (save-pending-message! db-uri "reset-password"
                         {:email email
                          :token token})
  (response/response "OK")

If you’ve any experience with Datomic, this is all pretty standard. As we don’t expect to do anything with the result, we’re happy to use transact-async here, which will return immediately, freeing the web server up to carry processing the request. At this point, the web server is done with sending the email.

In Highstorm, we have a task which immediately follows our Datomic transaction log input task in the workflow. It inspects Datomic transactions to check for the presence of datoms (atomic facts in the database) that match any of the preconfigured patterns in our worker tasks. I’ll refer to it as prepare-tx below.

So, our send-message task uses this pattern:

[_ :msg/status :status/pending _ true]

As is typical in Clojure, we use _ to mean ‘this can be any value; I’m not going to use it’. So, this pattern declares that:

  • Any entity ID is valid.
  • The attribute must be :msg/status.
  • The value for that attribute must be :status/pending.
  • Any transaction ID is valid.
  • Datoms contain a boolean true to indicate assertions in the database, and false for retractions. This is great: it means we can respond to retractions of facts just as easily as we do for new facts! In this case, though, the operation must be ‘assertion’, not ‘retraction’.

Through the use of dynamically generated flow conditions (easy to do, thanks to them being plain old data), we only allow a segment produced by prepare-tx to reach send-message-task when its datom matches this pattern.

The send-message-task function appears in the catalog, and is also quite simple:

(defn send-message-task [{:keys [db-uri t datom config]
                          :as segment}]
  (let [ ;; get db using the basis t provided by input task
        db (d/as-of (d/db (d/connect db-uri)) t)

        ;; get message entity id from the datom
        msg-id (first datom)

        ;; use the provided db and message id
        ;; to compose the message
        composed-msg (compose-message config db msg-id)

        ;; actually send the message
        api-result (send-message! config composed-msg)

        ;; build a tx modelling the new information we have
        result-tx {:db/id msg-id
                   :msg/composed composed-msg
                   :msg/result (pr-str api-result)
                   :msg/status :status/sent}]

    ;; add the result transaction to the segment
    ;; for later persistence to Datomic
    (update-in segment [:tx] conj result-tx)))

Some points of interest:

  • The segment, which is just a Clojure map, has all the Highstorm system configuration, so no global state is necessary. It’s easy to test by mocking alternate config.
  • We use the t value from the transaction to create a database to query. We do this for all the processing we do for a given task, making our view on the database consistent throughout all the tasks in the workflow for that segment.
  • The send-message! function is side-effecting, because it does I/O. There’s just no avoiding it in this case. However, once we’ve done that, we go back to working with immutable data.
  • Since Highstorm’s only output is more transactions for Datomic to process, all our intermediate tasks produce transaction data. Key for this task is that we change the message status to :status/sent. We also like to store the composed message so that our operations staff can easily diagnose issues when things don’t go as expected.

Once this segment reaches the output task, its transaction data is committed to storage, and it’s available for query on the web tier again.

So, to summarise the workflow, all of the segments in Highstorm follow this process, regardless of what the tasks in the middle do.

  1. tx-log-input
  2. prepare-tx
  3. [task(s) matching datom pattern]
  4. commit-tx

This is quite awesome, I think!

That’s it for now

I could go on and on :-)

Given how core this system is to our platform, I have plenty of other topics that I can talk about on it, which I plan to do in a follow up post or two:

  • Using Stuart Sierra’s ‘reloaded workflow’.
  • Ease of development.
  • Handling error conditions.
  • Working with metrics.
  • Tuning Onyx performance, including some hard-won lessons on back-pressure and throughput.

A huge, huge thank you

I want to thank the two (yes, just two!) guys in the Onyx team – Michael Drogalis, who first conceived of Onyx, and Lucas Bradstreet, who joined the cause about a year later – for their incredible patience and support during our learning curve with Onyx.

I know that although we were able to contribute by sharing bug reports, or feedback on this API or the utility of that feature, we at Cognician by far got the better deal. Thank you, guys. You’re both menschen!

I wouldn’t hesitate for a second to recommend them to anyone who wants to use Onyx in anger - they’ve just recently started a consultancy to offer commercial support, training, development and so on; you can hit them up at http://www.onyxplatform.org/support. I can’t think of a better way to get started.

You can also simply pop in for a chat on their their Gitter room, or in the #onyx room on the Clojurians Slack (not a member?). If you do, I’ll see you there!

Thanks for reading!

Permalink

Numbers and Arithmetics with functions only: lambda calculus live tutorial.

The mathematical theory behind LISP is the λ-calculus (pronounced lambda-calculus).

The amazing thing about λ-calculus is that it is possible to represent numbers and the arithmetic operations (successor, addition and multiplication) as functions.

Once you have arithmetics, you have almost all the mathematics and the computer science.

In this article, we are going to show this representation using clojure.

By the way, this is the signification of the λ sign in the clojure logo:

Clojure

Numbers representation in λ-calculus

Like everything in λ-calculus, numbers are functions.

A number n is a function that receives as argument a 1-arity function f and returns a function that applies f n times.

Let’s define the first natural numbers, manually:

(defn zero [f]
  (fn [x] x))

(defn one [f]
  (fn [x]
    (f x)))

(defn two [f]
  (fn [x]
    (f (f x))))

(defn three [f]
  (fn [x]
    (f (f (f x)))))

Numbers in action

In order to see the numbers in action, we need to execute them on a function and to execute the resulting function on a value.

All the code snippets on this article are live and interactive: feel free to modify the code and it will evaluate instantaneously!

We could take the increment function inc, and execute the result on 42:

((zero inc) 42)
((one inc) 42)
((two inc) 42)
((three inc) 42)

But, in order to make it more visual, we are going to define a custom function:

(defn visual [x]
  (list 'f x))

Let’s see what happens when we pass it to a lambda number:

((two visual) 'x)

Nice! Now, we see visually the definition of the number two: Application of a function twice.

In order to make things more convenient, let’s define a helper function view:

(defn view [f]
  ((f visual) 'x))

And let’s view the lambda numbers we have defined so far:

(map view [zero one two three])

Generating any lambda number

Let’s write a function that receives a regular number n and returns the corresponding lambda-number, using clojure functions comp, repeat and apply:

(defn lambda-num [n]
  (fn [f]
      (fn [x]
            ((apply comp (repeat n f)) x))))

Let’s view the lambda-number 6:

(view (lambda-num 6))

And we can easily view a range of lambda-numbers:

(map view (map lambda-num (range 5)))

Equality

Two lambda-numbers are equal if they are the same functions from a mathematical perspective.

It is disappointing that a lambda-number is not equal to itself:

(= (lambda-num 6) (lambda-num 6))

The reason, is that from a clojure perspective they are two different functions with the same code.

We will use the function view for equality test:

(defn eq? [m n]
  (= (view m) (view n)))

It works as expected:

(eq? (lambda-num 6) (lambda-num 6))
(eq? (lambda-num 4) (lambda-num 6))

Type definition for lambda

Let’s define a clojure type to customize the viewability and equality of lambda numbers. Our type will have the following properties:

  1. It is callable as a function
  2. It is viewable with view
  3. It redefines equality with view

For that purpose, we need to implement three clojure protocols:

  1. IFn
  2. IPrintWithWriter
  3. IEquiv

Let’s define a type named Lambda:

(If you are not familiar with the details of clojure protocols, you can skip the following code snippet.)

(deftype Lambda [f]
  Object
  (toString [_] (str (view f)))

  IPrintWithWriter
  (-pr-writer [this writer _] (-write writer (str (view f))))

  IEquiv
  (-equiv [this other]
          (= (view this) (view other)))

  IFn
  (-invoke [this g]
           (f g)))

And a function that converts a regular number to a typed lambda number:

(defn lambda-typed-num [n]
  (Lambda. (lambda-num n)))

Now, lambda numbers are viewable:

(map lambda-typed-num (range 5))

And comparable:

(= (lambda-typed-num 6) (lambda-typed-num 6))
(= (lambda-typed-num 4) (lambda-typed-num 6))

Now that we have defined lambda numbers and that they are viewable and comparable, we can introduce the basic arithmetic operations: successor, addition and multiplication.

The successor operation

Here is how one implements the successor function in λ-calculus. Successor is a function that receives a number n and returns n+1.

(defn successor [m]
 (fn [f]
  (fn [x]
   (f ((m f) x)))))

The code basically means: apply the function f m times and then another time.

Now, lets’ wrap it with our Lambda type

(defn succ [m]
  (Lambda. (successor m)))

Let’s check it works fine:

(succ (lambda-typed-num 1))
(= (succ (lambda-typed-num 8)) (lambda-typed-num 9))

The addition operation

Here is how one implements the addition function in λ-calculus. Addition is a function that receives two numbers and returns the result of their addition:

(defn add [m n]
 (Lambda.
  (fn [f]
   (fn [x]
    ((n f) ((m f) x))))))

The code basically means: apply the function f m times and then n times.

Let’s check it works fine:

(add (lambda-typed-num 3) (lambda-typed-num 2))

Make sure that 3+2 equals 5:

(= (add (lambda-typed-num 3) (lambda-typed-num 2))
   (lambda-typed-num 5))

The multiplication operation

Here is how one implements the multiplication function in λ-calculus. Multiplication is a function that receives two numbers and returns the result of their multiplication:

(defn mult [m n]
  (Lambda.
      (fn [f]
            (m (n f)))))

The code basically means: compose the function f n times and then m times.

Let’s check it works fine:

(mult (lambda-typed-num 3) (lambda-typed-num 2))

Make sure that 3*2 equals 6:

(= (mult (lambda-typed-num 3) (lambda-typed-num 2))
   (lambda-typed-num 6))

Conclusion

It is really amazing to see concretely how powerful is the λ-calculus. With a very limited set of characters and concepts we can build the arithmetics.

Permalink

Web Development with Clojure, Second Edition

I'm glad to announce that the second edition of Web Development with Clojure is finally finished. The book took longer to put together than I anticipated, and I ended up missing the original release target by a few months.

However, all the delays resulted in a much better book in the end. Having a long beta period allowed me to collect a lot of feedback from the readers and address any concerns that came up. This process helped ensure that the material is clear and easy to follow, while keeping a good pace. I discussed the specifics of what the book covers an earlier post here.

It's my sincere hope that the book will provide the readers with a smooth path into the wonderful world of Clojure web development.

Permalink

Docker, Clojure, Microservices, Google (and Alfresco): Introducing ECM Google Services

ECM Google Services

There are Google Vision „modules“ available for Alfresco and Nuxeo and most likely a bunch of other systems as well. These modules execute Google API Java code in process. They need to ensure the API along with its tree of dependencies is right there. Dependency trees can be huge – especially as we are heading into machine learning and artificial intelligence domains. The Alfresco repository comes with almost 300 Java libraries. The Google Vision API also has a few dependencies. Some of those are already in the Alfresco repository. Care should be taken if you want to put it there.

There are various approaches how to introduce a module along with its dependencies into a server process (Alfresco or Nuxeo). Alfresco suggests to use its proprietary AMP format, Nuxeo uses an OSGI „light“ approach (without lifecycle and/or isolation if I remember correctly). I don’t like either of them. Introducing more dependencies messes things up even further. And has great potential for all sorts of nasty issues.

I wanted to try the microservice route. It surely complicates matters as it introduces another process and network communication. On the other hand, it also has great advantages:

  • It does not mess with ECM server dependencies
  • The microservice can potentially be used from all your systems
  • The process lifecycle is independant of the client (which happens to be the ECM server)
  • Development turnaround is super quick – at least it was with tesla

I grabbed code which was there and quickly bent it towards microservices. One might consider it a hack. You have been warned. :)

The Microservice

I am always looking for Clojure adventures. That’s why I picked Otto’s tesla-microservice as a basis for the microservice (The Otto folks have base implementations for Java/Spring Boot, Node.js and maybe even other stacks on github). Naturally, I decided to wrap the microservice into a docker container.

If you want to try this out, make sure you have a Google API json key. If you don’t have one already, get yourself one. Playing around is free. And 1000 API calls/month are also free – if I remember correctly.

To spin up the docker container, do the following:

$ mkdir conf
$ echo 'vision-credentials-json=file:///conf/google-api-key.json' > conf/ecm-google.properties
$ cp /tmp/your-key.json conf/google-api-key.json
$ docker run -v `pwd`/conf:/conf \
   -e CONFIG_FILE=/conf/ecm-google.properties --rm \
   -p 9090:9090 deas/contentreich-ecm-google-ms:0.1.0

Alternatively, if you want to run the jar directly, download the ECM Google Microservice archive, extract it and exeute

$ cd contentreich-ecm-google-ms/
$ cp /tmp/your-key.json conf/google-api-key.json
$ CONFIG_FILE=./conf/ecm-google.properties java \
   -jar contentreich-ecm-google-ms-0.1.0-standalone.jar

The properties file supports way more values (generic microservice stuff and Google things I have introduced).

Either way, the service should be up and a demo form is available at http://localhost:9090/vision-form. Submitting an image should display the data returned by Google:

google-vision-demo-page

The Alfresco „Connector“

My Alfresco pieces are derived from the work of Angel Borroy. Links to sources and binaries can be found below.

To deploy the Alfresco extension, copy the repo-/share jar file to {alfresco,share}/WEB-INF/lib or modules/{platform,share} and (re)start the server.

There is a doclib action „Google Vision“ which is available for images. Try it out. It should set tags on the image. It should also set landmark, logo and text values when received from Google. At least I hope I preserved this functionality. :)

paulchen-doclib

Putting this together was an awesome experience. I am fairly sure this will continue to evolve over time. And maybe it can also serve to bring ECM nerd communities together. ;)

Download ECM Google Microservice
Download Alfresco ECM Google Service

References

The post Docker, Clojure, Microservices, Google (and Alfresco): Introducing ECM Google Services appeared first on CONTENTREICH.

Permalink

Meet up with Karsten Schmidt

The Clojure language draws together a diverse set of talents that makes for a great community full of inspiring developers.

A good example is Karsten Schmidt. Karsten has been a big influence on me and others Clojure devs I know. He combines a passion for creative programming with extraordinary accomplishment, demonstrating that there are few limits to where you can go and what you can do as a programmer.

I first met Karsten in London when he was busy preparing for an art installation at the Barbican Gallery in 2014. I later visited the exhibition and saw his work first-hand.

3D printed object Karsten sitting on a branch

The installation centered on a 3D printer protected by a 3-story high enclosure. Pedestals surrounded the enclosure, each holding an iPad running a software application for creating 3-dimensional objects. The software was written in ClojureScript, with a beautiful user interface (which itself could be considered a work of art).

Visitors to the exhibition, many of whom were children, created 3D objects using the software and submitted them to the 3D printer. The best creations were showcased in a display cabinet against the wall. I took a picture of one of these (these top two images seem computer generated, but they are actual photos).

Karsten sitting on a branch

It's almost impossible to see from the photograph, but on the left-hand wall is a small door leading to a small dark room where Karsten spent many days setting up the exhibit, working conditions that were hardly better than Michelangelo's when he was working on the (2D) ceiling of the Sistine chapel.

Building blocks

In his 2001 essay, The Roots of Lisp, Paul Graham describes the 'discovery' of Lisp as 'a landmark in the history of computers'.

"John McCarthy … did for programming what Euclid did for geometry. He showed how, given a handful of simple operators and a notation for functions, you can build a whole programming language."

Alan Kay once described Lisp as the "Maxwell's equations of software".

As Clojure developers, we believe this approach to building complex systems is more relevant today than ever. You reduce things down to atomic building blocks, which you can then compose together to form incredibly complex structures, such as computer systems or biological organisms.

As Graham's point suggests, there is a deep symmetry between geometry and Lisp which Karsten demonstrates in his work, both visually and, by choosing to work in a Lisp, in the code itself.

Literate Programming

Karsten sitting on a branch

Karsten has a rare, if not unique, programming style. He writes all his libraries using Knuth's Literate Programming style, where the code is written in snippets interwoven with the documentation. From this documentation (written using Emacs's 'org-mode' format), Karsten generates both Clojure and ClojureScript sources which are then compiled. So not only is his source 'open', but accessible too.

The best example is his geom library, which starts by defining points and lines, gradually forming rectangles, bezier curves and quaternions, meshes, matrix transformations, and just keeps going. If Euclid was alive today, I bet he'd be a Lisp programmer (and maybe a ClojureScript one!)

As Paul Graham also remarks in the same essay:

"It seems to me that there have been two really clean, consistent models of programming so far: the C model and the Lisp model. These two seem points of high ground, with swampy lowlands between them."

There are many other 3D libraries (arguably derived from the 'C model' that can produce similar results to geom, but what makes geom stand out is its Lisp foundations which enable playful creativity, discovery and experimentation in a world of 3D. This is a reoccuring theme that we are exploring at XT16, more on that later!

Something we created on the course

Ever since I discovered geom and Karsten's other libraries I've wanted to spend some time learning the basics, and perhaps one day to experience the power of wielding geom on real-world projects. So earlier this year I booked myself on one of Karsten's unique courses. The particular course I chose was on WebGl in the browser with ClojureScript, since I'm interested in combining WebGL in my applications and see the browser as the catalyst to a new wave of consolidation between 3D-graphics and business user-interfaces.

The course was excellent and hugely fun, yet challenging too and I took away an enormous amount of practical knowledge. I could explain more about the course here, but Karsten's already published a great write up of it. I loved experimenting with the code and trying out things: "what happens if I change this value to X" and the browser reload kicking in and showing me. But watching Karsten's wizard-like prowess with these libraries was something to behold. (If you want details on Karsten's future courses, see here).

Meet Karsten at XT16

Jon Pither and I caught up with Karsten recently in his home borough of Highgate, North London to talk about Karsten's work, philosophy, current projects and his upcoming talk and live-coding at XT16. We entered the woods (what Karsten calls his 'office') and had some fun building a makeshift tripod for the camera. You can eaily tell that Karsten's appreciation of Nature influences and inspires his programming work.

We're very excited that Karsten is coming to our XT16 event in the Autumn and giving a talk with some live-coding. Due to the intimate nature of the conference there'll be plenty of opportunity to meet and socialise with Karsten (and everyone else!) both during and afterwards.

If you haven't got your ticket yet, then visit our (ClojureScript-powered!) XT16 ticketbooth. Sadly it doesn't yet use Karsten's geom library, but who knows, coming to the conference might be the first step in your own ClojureScript 3D adventures!

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.