So #ClojureX was excellent, now here’s my corrections – @skillsmatter @onyxplatform

My first #ClojureX done and I’ve already spent the morning thinking about what I could possibly talk about in 2017. Hands down one of the best developer conferences I’ve attended. It’s all about the community and the Clojure community gets that 100%, it showed over the two days.

Many many thanks to everyone at SkillsMatter for looking me, the weary traveler, the tea was helpful. Also great to meet some of the folk I regularly talk to on the Clojurians Slack channel.

15267854_10155678189670200_7186480394544666013_n

A couple of things following my talk now I’ve watched it back. If you want to watch then this is the link: https://skillsmatter.com/skillscasts/9153-introducing-streaming-processing-with-kafka-and-the-onyx-platform

Firstly, when you create the Onyx app with lein it does create the docker compose file but it only has the Zookeeper element, not the Kafka one – that has to be added in afterwards.

Secondly, the percentage scheduler adds up to 100, I said zero. Brain detached for a second, thought one thing and something else came out.

Apologies, I don’t like giving out wrong information.

Craig and Darcey would have been proud, kind of, perhaps.

Here’s to 2017.


Permalink

The REPL

The REPL

spec-tacular, macchiato, and deep learning
View this email in your browser

The REPL

Notes.

There were several big pieces of Clojure news this week. First, Datomic announced a change in license to allow unlimited processes and a new client API suited for microservices. I wrote a guide to the licensing changes. Second, Clojure Conj is on as I write this, and has been getting all the videos out in record time. I'll put some of my favourites in the next letter. Finally, Clojars has joined the Software Freedom Conservancy, and announced the Clojars Community Committee to support diversity and open source development in the Clojure and ClojureScript communities.

-main

Libraries & Books.

  • durable-ref: Durable reference types for Clojure, with some excellent docs.
  • alia, a Cassandra client has released 4.0.0-beta1

People are worried about Types. ?

Foundations.

Tools.

Recent Developments.

  • CLJ-1912: Optimised versions of the '<' and '>' functions for arities larger than 2.
  • CLJ-1952: include var->sym in clojure.core

Learning.

Misc.

Copyright © 2016 Daniel Compton, All rights reserved.


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by MailChimp

Permalink

DropWizard vs Ring

I have been blogging here a lot about the new kids on the JVM block, namely Clojure and Scala. I wanted to see how they compared against the more established Java frameworks. When version 1 of DropWizard was released, I decided that was the time to write another news feed micro-service implementation. This blog is about the news feed micro-service implementation using DropWizard.

conceptual level class diagram of dropwizard service

I wrote the feed, ran some load tests, then compared the results to the same load tests for the Clojure version. I documented my findings in the PDF at the bottom of this page. Here are some answers to questions about the document below.

Q: How did you come to the conclusion that Guice is more popular than Spring when it comes to Dependency Injection in DropWizard?
A: It’s been four months since version 1 of DropWizard has been released. The dropwizard-guice integration was available immediately. There still isn’t an integration with Spring yet.
Q: Why did you bother using Swagger?
A: I have been an advocate for MDSD for a long time now. Swagger is fairly popular and I wanted to see just how well the popularity was deserved by using it to generate all the RESTful parts of the service.
Q: Well? What do you think of Swagger?
A: I like the technology. Curiously enough, the web site doesn’t mention MDSD even once. The focus seems to be about onboarding start ups to get to market faster using their pre-existing templates. That’s cool but the real value comes when you adopt and adjust the templates as an expression of your own architecture. One size does not fit all.
Q: What is your “standard load test?”
A: Basically, it is a program that I wrote that simulates news feed load by creating participants, creating friend relationships between the participants, then posting news to participant outbound feeds.
Q: What is the difference between low load and high load?
A: Running the load test program with one thread vs three threads.
Q: You referenced JDBI in the architecture section and then you mentioned Java annotation in the differences section and then you listed Guice and Jersey. Didn’t you also mean to list JDBI in the annotation paragraph too?
A: No. I used the JDBI fluent api instead of the Java annotation approach.
Q: How do you run all these load tests?
A: I used aws for all my load test experiments.
Q: You used to use Solr for the search component. With this implementation of the news feed micro-service, you switched to Elastic Search. Does that mean you now prefer Elastic Search over Solr?
A: I had to switch over to Elastic Search for the DropWizard service due to a version conflict in the downstream dependencies with Jetty. I switched from Solr to Elastic Search for the other news feed apps in order to have a more valid basis for comparison during all the load testing.
Q: Why do you still stand up your own EC2 instances of Elastic Search and Redis when you can just use the AWS PaaS versions of those services?
A: I tried using Amazon Elasticsearch Service and Amazon ElastiCache but I couldn’t get it to work. I think that I had to set up IAM based access for those and I just didn’t want to. I didn’t try very hard because it is pretty easy and quick to stand up those services yourself.
Q: Any interest in switching to Memcached?
A: I already coded those classes and will most probably write a blog in the future comparing news feed performance when using Redis vs Memcached.

DropWizard vs Ring: The Java Framework Strikes Back

Permalink

LED is my new Hello World - Clojure Time

The more I learn Clojure...the more I like it...and I'm liking it so much that I couldn't help myself and start working on my beloved "LED_Numbers" application...

After making the Fibonnaci Generator app work...this one wasn't as hard as I expected...actually I think I'm slowly getting used to Clojure...which is always nice when learning a new language -;)

Here's the source code...

LED_Numbers.clj
(def leds {"0" (list " _  " "| | " "|_| ") "1" (list "  " "| " "| ")
"2" (list " _ " " _| " "|_ ") "3" (list "_ " "_| " "_| ")
"4" (list " " "|_| " " | ") "5" (list " _ " "|_ " " _| ")
"6" (list " _ " "|_ " "|_| ") "7" (list "_ " " | " " | ")
"8" (list " _ " "|_| " "|_| ") "9" (list " _ " "|_| " " _| ")})

(defn toList [number]
(map str(seq(str number))))

(defn get_led [x n num]
(cond
(> (count x) 0)
(concat (nth (get leds (first x)) n) (get_led (rest x) n num))
(and (= (count x) 0) (< n 2))
(concat "" "\n" (get_led (toList num) (+ 1 n) num))
(and (= (count x) 0) (= n 2))
(concat "" "\n")))

(defn showLED [num]
(do (print (apply str (get_led (toList num) 0 num))))(symbol ""))

Wanna see it in action? Of course you want to -:)


Well...let's go back and keep learning -:D

Greetings,

Blag.
Development Culture.

Permalink

Recorded talk about destructuring in Clojure (in Spanish)

I recently started to work at Codesai.

Some members of the team wanted to learn Clojure, so we started a small Clojure/ClojureScript study group.

We created a slack called clojuresai where I'm posting some readings (we're reading Clojure for the Brave and True) and exercises (we're working through a selection of exercises from 4Clojure) each week and where they can ask doubts about the exercises and readings, and comment things they find interesting.

Some colleagues from the Clojure Developers Barcelona meetup that are beginning to learn clojure have also joined the study group.

Now and then, we do introductory talks using Google Hangouts (Codesai's team is distributed) to go deeper in some of the topics they've just read in the book.

So far we've done two talks. The first one was about Clojure collections. Unfortunately, we didn't record it, but you can find the examples we used here.

Today we did a second remote talk. This time it was about destructuring in Clojure, and since some of the members of the study group members weren't able to make it, we decided to record it.

This is the resulting video:

You'll find the examples we used in this previous post.

I'd like to especially thank Ángel and Magento Barcelona for letting me do the talk today from their office (their internet connection is much better than mine).

Recently, Miguel de la Cruz (thanks a million Miguel!!) has joined our study group as a tutor. He knows a lot of Clojure so he will help answering doubts and giving some talks that we'll hopefully share with you soon, as well.

Permalink

Clojure in Berlin: Zalando

A growing fashion company

Zalando is Europe's leading online platform for fashion. It originated in 2008 as an online shoes retailer and has since expanded into 15 markets across Europe.

They now have over 11,000 employees with 1600 working in tech. The website has over 160 million visits per month (~65% mobile) with 19 million active customers. The company made over 3 billion EUR in net sales in 2015.

Their technology HQ is in Berlin and they have offices in other parts of Germany, Helsinki and Dublin.

I caught up with software engineer Dmitrii Balakhonskii to learn more.

Background

Jon: Could you give me some background on the tech at Zalando?

Dmitrii: In the beginning there was a uniform monolithic technology stack; mostly Java applications running in Tomcat clusters.

There was a strict policy on choosing technologies and so developers were restricted; not able to choose the technologies they'd love to work with. As the tech team grew the delivery speed was slowing down. Also there were not as many candidates applying and a lack of excitement about what we were doing.

Jon: What changed?

Dmitrii: The idea came to use a totally new approach to managing developers: what we call Radical Agility. In March 2015 the teams became automomous teams and treated as first class citizens. The management team defined what to expect and the team decided on how to deliver. The teams could decide which technologies to use.

The overall tech team became more diverse, using Scala, Clojure, and Go in some teams. This diversity made the jobs of the developers more interesting, fun and inspiring.

Jon: How do you co-ordinate across teams?

Dmitrii: We have common standards; everyone builds Microservices using REST API-first. First a team writes a description of an API then publishes it and aligns with the other teams when they implement.

Jon: What is the mix of technologies?

Dmitrii: There is more Scala used and the majority is still doing Java. We have around 10 teams using Clojure. As a company we are mostly JVM based.

Clojure

Jon: How did Clojure get started at Zalando?

Dmitrii: At one stage we decided to start using AWS. To provide all teams with a uniform approach a special Platform team was organised.

They built a middleware product; an abstraction layer on top of the cloud written in Clojure called STUPS. Nearly everyone who deploys into the cloud now uses STUPS.

Tobias Sarnowski originally introduced Clojure. People saw products successfully built using Clojure and followed suit, embracing the language.

We've now been using Clojure for 2 years, growing and improving. We have our own technology radar - inspired by the JUXT Clojure radar, Clojure is almost in adopt.

Jon: What do you yourself work on?

Dmitrii: I'm maintaining Pier One. It's a Docker registry that satisfies our company requirements such as OAuth, S3 backend, and immutable tags. We have about 40k Docker images it manages.

It's written in Clojure and I'm happy with it; the codebase is small and easy to maintain.

All the topics on the interwebs

Jon: Has Clojure grown outside of the deployment layer?

Dmitrii: Yes, one use case is for internal tools such as proxies, infrastructure tools and CI servers. We have a team that provides continuous integration services to the other teams. We have about 200 CI workers that are powered down at night, but then you can still use them if you want to.

Hunter Kelly presented at last years Clojure/Conj on using Clojure and Spark to 'understand fashion in a more quantitative manner'.

We have written libraries in Clojure to help teams write Microservices. Friboo is an example that offers common components and encourages an API First approach using REST and Swagger.

People

Jon: Do you think that Clojure has made a difference to IT at Zalando?

Dmitrii: It's not so much the technology that makes a difference, but the people being inspired and motivated when they make a decision. Both Scala and Clojure are successful here for this reason. There is no overall trend - our goal is to be more attractive to developers.

When we hire developers they don't immediately join a specific team. The first month they attend workshops and get to know the company. Then the teams present themselves to the new hires and the new developers get to decide which team to join. This makes teams compete to be attractive.

Upskilling

Living Clojure

Jon: How do you go about training and upskilling?

Dmitrii: For training we have weekly Clojure workshops. We also have a Clojure mentoring program where people can register and do exercises from the Living Clojure book by Carin Meier.

Jon: How have people taken to learning Clojure?

Dmitrii: I had a developer join our team who was new to Clojure. After three months he wrote a proxy server that wrapped a legacy server and was highly performant. Three months was enough for him to deliver production code.

We have seen that people are interested in growing and developing their careers. We also have a management team to take people out of their comfort zone and to challenge them with new ideas.

State of Clojure

Clojure Bridge

Jon: What is the state of Clojure in Berlin?

Dmitrii: Clojure in Berlin is quite active - we have monthly meet ups with about 50 people in attendance. I go from time to time and see new faces there. There are other companies here doing Clojure.

We also have a ClojureBridge event 3 times a year - organised by fellow Clojure enthusiasts. I worked as a coach there.

Jon: Anything that interests you in the development of the Clojure language?

Dmitrii: I'm very exciting about a big new feature; clojure.spec. There is lots to explore here. There is very few alternatives to this in other languages. It's very nicely designed and the way of the future.

Introduction to Clojure Spec

Jon: What's so radical about Spec?

Dmitrii: It's an alternative way of ensuring the data has the shape you expect it to have. It's the answer from the dynamic typing world to the compile time checks in static languages, but solved in a nicely dynamic and idiomatic way. The presence of Clojure.spec will make Clojure easier to sell.

Technologies

Jon Pither: Any technologies you'd like to give a shout out to?

Dmitrii: Stuart Sierra's Component framework which is a big thing. All our services are based on Component. Also Midje, it's really a step up from clojure.test. I like that you can mock and check everything in every possible way; very convenient.

Resources

Check out the Zalando tech blog.

Permalink

Macchiato: ClojureScript Arrives on the Server

I recently started the Macchiato project to provide a platform for building ClojureScript based apps on top Node.js.

First, let's look at some of the reasons for running ClojureScript on the server. The JVM is an excellent platform, it's mature, performant, and has a large ecosystem around it. This makes it a solid choice for a wide range of applications.

However, there are situations where the JVM might not be a good fit. It's a complex piece of technology that requires experience to use effectively. It has a fairly large footprint even from small applications. The startup times can be problematic, especially when it comes to loading Clojure runtime.

Meanwhile, Node.js also happens to be a popular platform with a large ecosystem around it. It requires far less resources for certain types of applications, has very fast startup times, and its ecosystem is familiar to many JavaScript developers.

Another appeal for Node based servers comes from building full stack ClojureScript single-page applications, since using Node on the server facilitates server-side rendering for any React based libraries.

While there are a few existing experiments using ClojureScript on Node, such as Dog Fort, none of these appear to be actively maintained. Since ClojureScript and its ecosystem have evolved in the meantime, I wanted to create a fresh stack using the latest tools and best practices.

Overview

My goal for Macchiato is to provide a stack modeled on Ring based around the existing Node ecosystem, and a development environment similar to what's available for Clojure on the JVM.

The Stack

I think it makes sense to embrace the Node ecosystem and leverage the existing modules whenever possible. For example, Ring style cookies map directly to the cookies NPM module. Conversely, there are a number of excellent ClojureScript libraries available as well, such as Timbre, Bidi, and Mount.

I used a Ring inspired model where I created wrappers around Node HTTP request and response objects. This allowed adapting parts of Ring, such as its session store implementation, with minimal changes.

The ClientRequest object is translated to a Clojure map, and the response map is written to the ServerResponse object. The request handler is implemented as follows:

(defprotocol IHTTPResponseWriter
  (-write-response [data res] "Write data to a http.ServerResponse"))

(defn response [req res opts]
  (fn [{:keys [cookies headers body status]}]
    (cookies/set-cookies cookies req res (:cookies opts))
    (.writeHead res status (clj->js headers))
    (when (-write-response body res)
      (.end res))))

(defn handler [handler-fn & [opts]]
  (let [opts (or opts {})]
    (fn [req res]
      (handler-fn (req->map req res opts) (response req res opts)))))

The handler accepts a handler-fn function that's passed the request map produced by the req->map helper. The handler-fn is expected to return a request handler function that will be used to generate the response. This function should accept the request map and the response call back function that writes the response map to the ServerResponse object. The IHTTPResponseWriter protocol is used to serialize different kinds of responses.

Concurrent Request Handling

JVM servers commonly use a listener thread for accepting client requests, the connections are then passed on to a thread pool of request handlers. This allows the listener to continue accepting connections while the requests are being processed.

Since Node is single threaded, long running request handlers block the server until they finish. While async operations can be used to handle IO in the background, any business logic will end up preventing the server from accepting new connections while it's running.

One way around this is to use the cluster module that spins up a single listening process that forks child processes and dispatches the requests to them. Setting this up is pretty straight forward:

(defstate env :start (config/env))

(defstate http :start (js/require "http"))

(defn app []
  (mount/start)
  (let [host (or (:host env) "127.0.0.1")
        port (or (some-> env :port js/parseInt) 3000)]
    (-> @http
        (.createServer
          (handler
            router
            {:cookies {:signed? true}
             :session {:store (mem/memory-store)}}))
        (.listen port host #(info "{{name}} started on" host ":" port)))))

(defn start-workers [os cluster]
  (dotimes [_ (-> os .cpus .-length)]
    (.fork cluster))
  (.on cluster "exit"
       (fn [worker code signal]
         (info "worker terminated" (-> worker .-process .-pid)))))

(defn main [& args]
  (let [os      (js/require "os")
        cluster (js/require "cluster")]
    (if (.-isMaster cluster)
      (start-workers os cluster)
      (app))))

However, it's worth noting that unlike threads, processes don't share memory. So, each child that gets spun up will require its own copy of the memory space.

The Template

I setup a template that creates a minimal app with some reasonable defaults. This template is published to Clojars, and you can try it out yourself by running:

lein new macchiato myapp

The template is setup similarly to Luminus. The source code for the project is found in the src folder, and the env folder contains code that's specific for dev and prod environments.

The project.clj contains dev and release profiles for working with the app in development mode and packaging it for production use. The app can be started in development mode by running:

lein build

This will clean the project, download NPM modules, and start the Figwheel compiler. Once Figwheel compiles the sources, you can run the app with Node in another terminal as follows:

node target/out/myapp.js

The app should now be available at http://localhost:3000.

Figwheel also starts the nREPL at localhost:7000. You can connect to it from the editor and run (cljs) to load the ClojureScript REPL.

Packaging the app for production is accomplished by running:

lein package

This will print out package.json for the app and generate the release artifact called target/release/myapp.js.

Looking Forward

Overall, I think that ClojureScript on top of Node is ready for prime time. It opens up server-side Clojure development to a large community of JavaScript developers, and extends the reach of Clojure to any platform that supports Node.

While the initial results are very promising, there is still much work to be done in order to provide a solid stack such as Luminus. If you think this project is interesting, feel free to ping me via email or on the Clojurians slack. I would love to collaborate on making Macchiato into a solid choice for developing Node based applications.

Permalink

Schema & Clojure Spec for the Web Developer

This post will walk through the main differences of Schema and Spec from the viewpoint of a Clojure(Script) web developer. There is also some thinking aloud how we could achieve best of both worlds and peek into some evolving libraries. This is a first part of a blog series.

Clojure Spec

Clojure Spec is a new Clojure(Script) core library for specifying Clojure applications & data for communication, validation, parsing and generative testing. It is similar to Plumatic Schema but also has some cool new features like spec destructuring, multispecs and a inbuilt serialization format. Spec is still in alpha, and will ship with Clojure 1.9.0. There is a great introduction talk by Arne Brasseur from ClojuTRE 2016. Carin Meier's talk "Genetic programming with clojure.spec" on EuroClojure 2016 was a mind-blower, sadly the video is not on Internet.

Schema

We at Metosin are big fans of Schema. For the last three years, it has enabled us to build robust and beautifully documented apps both for both Clojure & ClojureScript. Many of our open source libs have been built on top of Schema. These include ring-swagger, compojure-api and kekkonen. There is also others like pedestal-api and yada using Schema currently.

Differences

This post is not a complete comparison of the two, but instead highlights some of the key differences that a normal Clojure Web Developer (like me!) would see in the daily work: how to define and transform models (both at design- and runtime), validating/transforming values from external sources, api-docs and getting human-readable error messages for the end users. Things like function specs/schemas and generative testing are left out.

Defining the models

Schema

With Schema, models are defined as Clojure data structures and Schema predicates or Java Classes. Schema maps are closed by default, allowing no extra keys. Schemas are easy to reason about as they are defined in the same form as the values it represents. Errors are presented in a hybrid of human/machine -readable format.

(ns user.schema)

(require '[schema.core :as s])

(def age
  (s/constrained s/Int #(> % 18) 'over-18))

(s/defschema Address
  {:street s/Str
   :zip s/Str})

(s/defschema Person
  {::id s/Int
   :age age
   :name s/Str
   :likes {s/Str s/Bool}
   (s/optional-key :languages) #{s/Keyword}
   :address Address})

(def liisa
  {::id 1
   :age 63
   :name "Liisa"
   :likes {"coffee" true
           "maksapihvi" false}
   :languages #{:clj :cljs}
   :address {:street "Amurinkatu 2"
             :zip "33210"}})

(s/check Person liisa) ; => nil

(s/check Person {:age "17", :bogus "kikka"})
; {:user.schema/id missing-required-key,
;  :age (not (integer? "17")),
;  :name missing-required-key,
;  :likes missing-required-key,
;  :address missing-required-key,
;  :bogus disallowed-key}

Reusing schemas is done either by predefining common parts of it (like the age and address above) or by transforming existing schemas. As the schemas as just data, transformations can also be done at runtime. For more complex transformations, there are external libraries like Schema Tools and Schema-bijections.

;; reuse at compile-time
(s/defschema PersonView
  (select-keys
    Person
    [::id :likes :address]))

(s/check
  PersonView
  (select-keys
    liisa
    [::id :likes :address])) ; => nil

;; reuse at runtime!
(let [keys [::id :likes :address]]
  (s/check
    (select-keys Person keys)
    (select-keys liisa keys))) ; => nil

Above Schemas visualized with schema-viz:

Spec

With Spec, models are defined using clojure.spec macros and function predicates. Maps are defined using keysets instead of key-value pairs. All map keys need to be globally registered. Calling s/form on any given spec returns the original source code for it, and should later enable spec serialization. Errors are reported in machine-readable format.

(ns user.spec)

(require '[clojure.spec :as s])

(s/def ::id integer?)
(s/def ::age (s/and integer? #(> % 18)))
(s/def ::name string?)
(s/def ::likes (s/map-of string? boolean?))
(s/def :user.address/street string?)
(s/def :user.address/zip string?)

(s/def ::languages
  (s/coll-of keyword? :into #{}))

(s/def ::address
  (s/keys :req-un [:user.address/street
                   :user.address/zip]))

(s/def ::person
  (s/keys :req [::id]
          :req-un [::age
                   ::name
                   ::likes
                   ::address]
          :opt-un [::languages]))

(def liisa
  {::id 1
   :age 63
   :name "Liisa"
   :likes {"coffee" true
           "maksapihvi" false}
   :languages #{:clj :cljs}
   :address {:street "Amurinkatu 2"
             :zip "33210"}})

(s/valid? ::person liisa) ; => true

(s/explain-data
  ::person {:age "17", :bogus "kikka"})
; {:clojure.spec/problems
;  ({:in [], :path [],
;    :pred (contains? % :user.spec/id),
;    :val {:age "17",
;          :bogus "kikka"},
;    :via [:user.spec/person]}
;    {:in [], :path [],
;     :pred (contains? % :name),
;     :val {:age "17",
;           :bogus "kikka"},
;     :via [:user.spec/person]}
;    {:in [], :path [],
;     :pred (contains? % :likes),
;     :val {:age "17",
;           :bogus "kikka"},
;     :via [:user.spec/person]}
;    {:in [], :path [],
;     :pred (contains? % :address),
;     :val {:age "17",
;           :bogus "kikka"},
;     :via [:user.spec/person]}
;    {:in [:age], :path [:age],
;     :pred integer?, :val "17",
;     :via [:user.spec/person
;           :user.spec/age]})}

Spec promotes application level reuse as all the specs are found in the registry. New specs can be composed with the clojure.spec macros like and, or and merge. Due to use of macros, creating specs at runtime is not easy - and would pollute the global spec registry. Spec is still young but there is already many evolving utility libs for it. We are doing the Spec Tools and there is at least Schpec and Spectrum out there.

;; reuse specs at compile-time
(s/def ::person-view
  (s/keys :req [::id]
          :req-un [::likes :address]))

(s/valid?
  ::person-view
  (select-keys
    liisa
    [::id :likes :address])) ; => true

(s/valid?
  (s/keys :req [::id]
          :req-un [::likes ::address])
  (select-keys
    liisa
    [::id :likes :address])) ; => true

;; runtime (bad idea but works)
(let [req-keys [::id]
      req-un-keys [::likes ::address]
      value-keys [::id :likes :address]]
  (s/valid?
    (eval
      `(s/keys :req ~req-keys
               :req-un ~req-un-keys))
    (select-keys
      liisa value-keys))) ; => true

Transforming values

For web app runtime, it's important to be able to both validate/conform values from external sources. Different wire-formats have different capabilities for presenting types. In string-based formats (like ring :query-params & :path-params) all values have to be represented and parsed from Strings. JSON supports maps, vectors, numbers, strings, booleans and null, but not for example Dates or Keywords. Both EDN and Transit can be extended to support any kind of values.

Schema

In Schema, there is coercion. Given a Schema and a separate matcher at runtime, one can validate and transform values from different formats into Clojure data. Schema ships with matchers for both string and json formats. Matchers can be easily extended.

(require '[schema.coerce :as sc])

;; define a transformation function
(def json->Person
  (sc/coercer
    Person
    sc/json-coercion-matcher))

;; :languages from [s/Str] => #{s/Keyword}
(json->Person
  {::id 1
   :age 63
   :name "Liisa"
   :likes {"coffee" true
           "maksapihvi" false}
   :languages ["clj" "cljs"]
   :address {:street "Amurinkatu 2"
             :zip "33210"}})
; {:user.schema/id 1,
;  :age 63,
;  :name "Liisa",
;  :likes {"coffee" true,
;          "maksapihvi" false},
;  :languages #{:clj :cljs},
;  :address {:street "Amurinkatu 2"
;            :zip "33210"}}

Spec

Spec has a conform, which works like coercion but the transforming function is directly attached to the Spec instead of passed in at runtime. Because of this, it's not suitable for runtime-driven transformations.

(s/def ::str-keyword
  (s/and
    (s/conformer
      (fn [x]
        (if (string? x)
          (keyword x)
          x)))
    keyword?))

(s/conform ::str-keyword "clj") ; => :clj
(s/conform ::str-keyword :clj) ; => :clj

To support more Schema-like runtime conformations, we have the following options:

1. Better clojure.spec/conform

Current conform takes only the spec and a value as arguments (s/conform spec x). It could have a 3-arity version where we could pass in a runtime provided callback function to selectively conform based on the spec value (s/conform spec x spec->conformer-fn). I have been mumbling about this in the Clojure Slack & in Google Groups. Would be simple, but not likely going to happen.

2. Dynamic conforming

Clojure has the Dynamic Scope, which could be used to pass conforming callback to the conformer function at runtime. The Conformer could read this value and conform accordingly. This requires a special "dynamic conformer" to be attached to all specs. Default operation would be no-op. There could also be set of predefined "Type Predicates" which would have a dynamic conformer attached. There could be a special dynamic-conform to set set the variable and call vanilla conform.

There is an implementation of this in Spec-tools, more about that in the next part of this blog.

3. Generate differently conforming Specs

Specs could be walked with clojure.spec/form generating (and registering) differently conforming specs for all conforming modes. All the new specs would have to have new generated names, e.g. :user/id => :user$JSON/id. Seems easy and elegant, but there are few challenges on the way:

  • Due to the current implementation of s/keys, fully qualified spec keys can't be exposed this way. In Spec, by design, the keys and the values are not separate and thus a qualified spec key can't be mapped to multiple, differently conforming versions. I tried to create a modified version of s/keys for this but ended up copying most of the clojure.spec to properly support it. Maybe later.

  • the s/form has a nasty bug in alpha-14, some specs still emit non-qualified forms. This should be fixed soon.

4. Create an extra layer of "Easy Data Specs"

One could invent a new and more data-driven format having it's own mechanisms for runtime conforming. But - adding a "easy" abstraction layer comes with a cost and most likely will backfire eventually. For now at least, it's good to work with Specs directly, as we are all still learning.

5. Generating Schemas from Specs

Tried this too, was a bad idea: there would be two sets of errors messages depending on where it was raised. It's better to have Specs (or Schema) all the way down.

Api-docs

This is important. With Schema, we have tools like ring-swagger, which transforms nested Schemas into Swagger JSON Schema enabling beautiful api-docs.

;; [metosin/ring-swagger "0.22.12"]
(require '[ring.swagger.swagger2 :as rs])

(rs/swagger-json
  {:paths
   {"/echo-person"
    {:post
     {:summary "Echoes a person"
      :parameters {:body Person}
      :responses {200 {:schema Person}}}}}})
; ... valid swagger spec returned

For Spec, there aren't any finalized solution for this yet. Andrew Mcveigh is working on something and we have a Spec -> JSON Schema transformer in Spec-tools, but it's not complete yet and has some hacks while waiting for the core s/form - bug to be fixed. The Swagger transformations can be used separately and plan is for it to be eventually merged into ring-swagger for easy transition. Something like (but with qualified keys?):

(require '[spec-tools.swagger :as swagger])

(swagger/swagger-object
  {:paths
   {"/echo-person"
    {:post
     {:summary "Echoes a person"
      :parameters {:body ::person}
      :responses {200 {:schema ::person}}}}}})

Human-readable error messages

Neither of the two libraries has solved this one for good. There are some promising experiments out there, looking forward to seeing them mature and get integrated into tooling.

Conclusion

As per today, Schema is a proven solution for building robust runtime-validating web apps and is not going away. There is good set of existing web libs using it already providing both runtime coercion & api-docs. Schema can be used in the ClojureScript for things like dynamic form and server request validation. We have been using Schema in most of our projects and will continue to use and support it in our libs.

Spec is awesome and without a doubt will be de facto data description library for Clojure. For now, the runtime conforming & api-docs story is not on par with Schema but it will be, eventually. Not all web apps need currently the runtime conforming feature, end2end Clojure(Script) apps can transfer data in Transit using just runtime validation instead of conforming. Also, for Spec, we have to remember that it's still in Alpha, so things might change.

Road ahead

Spec is still under development and there are lot of community libs evolving around it. We too are building tools to help adopting Spec for web & api development, more on spec-tools & friends on Part2. Our web-libs will support spec as soon as.

Exciting times to be a Clojure(Script) web developer :)

Tommi (@ikitommi)

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.