crepl: Link to examples

crepl: Link to examples

crepl is a collaborative editor where you can write and run ClojureScript together. You can now create links to new crepl pads that will be loaded with the code from an external source. Once you are in a crepl you can invite others to join you by sending them the url in the browser navigation bar. For instance try out these examples:

Read more

Permalink

JShell: The Java Shell and the Read-Eval-Print Loop

Let's talk about JShell. We can explore it with the JDK 9 Early Access Release. As of now, the general availability of JDK9 is scheduled for 27 July, 2017, and the JShell feature was proposed as part of JEP 222. The motivation behind it is to provide interactive command line tools to quickly explore the features of Java.

From what I've seen, it is a very useful tool to get a glimpse of Java features very quickly, which is especially useful for new learners. Already, Java is incorporating functional programming features from Scala. Consider the move to a REPL (Read-Eval-Print Loop) interactive shell for Java, just like Scala, Ruby, JavaScript, Haskell, Clojure, and Python.

Permalink

Where Should Tests Live?

I came across a design question when working on elm-test recently:

What if you could write tests in the same file as the code they're testing?

There are a few testing systems that support this:

  • clojure.spec (to vastly oversimplify) lets you define tests inline in the same file as your business logic.
  • Rust allows putting tests in the same file as the code being tested, although the documentation recommends putting them in a separate file (without elaboration as to why).
  • Doctests (such as in Python, Elixir, Go, and Haskell) allow writing tests in the same file—theoretically for documentation purposes, although potentially for testing purposes instead.
  • EUnit in Erlang runs all functions in a given module that end in _test as tests, including business logic modules.
  • Racket supports inline tests.
  • Pyret supports "testing blocks" which can live in any source file.
  • Test::Inline adds support for this to Perl.

I've never used any of these, but I'm wondering how the experience would be in practice. Is it amazing? Unexciting? An outright bad idea? What are the pros and cons compared to putting tests in a separate file?

If you've tried writing tests like this before, what did you think?

Permalink

Bayadera = Bayes + Clojure + GPU (slides for my Bob Konferenz 2017 talk)

Hi Clojurians! I wished I had time to write a few posts since EuroClojure, but I got carried away by some other stuff (there are a few cool things coming up in the next version of Neanderthal, and some Clojure CUDA stuff after that!). The good news is that I have more interesting themes to write about soon!

For now, I hope that the slides for my upcoming talk at Bob Konferenz, that takes place on 24th of February in Berlin will give you an interesting preview of Bayadera, a Clojure library for high performance Bayesian Data Analysis.

The slides are available here, just open them in the web browser. You can also read the abstract of the talk here.

Please note that there are two directions to browse: down/up, and left/right. Down/up arrows move you through slides, while left/up arrows move you through sections. Normally, you'd press down-down-down until there are no more slides, and then right to cross to the next section. ESC gives you the bird's view of the structure.

Permalink

Java Generational Garbage Collection

Java Generational Garbage Collection

How the JVM makes Clojure's data structures possible.

Persistent data structures require really good garbage collection. Lisp has always had persistent data structures. The cons list is persistent because it can share structure. When Clojure came out, it featured immutable persistent data structures. And not just the list. It also has vectors, maps, and sets.

Because they're immutable, once an object is instantiated, it is never changed. That means Clojure has to use a "copy-on-write" discipline. Instead of modifying the object, you make a modified copy. That means you've got lots of garbage. You're making copies all the time.

Luckily, because the data structures are persistent, it means that they share a lot of their structure. The amount of garbage memory per modification can be quite low relative to the amount of memory that can be reused.

But it's still a lot of garbage. Every seq allocates. Every time you modify a vector or a map, it allocates. Garbage is the name of the game in Clojure. That's why garbage collection was invented. A lot of research has gone into garbage collection, particularly for reducing the amount of time the GC has to pause.

One of the coolest ways to reduce that pause time is to consider the age of the objects. Most objects that are old (they were instantiated a while ago) tend to stick around. So why look at them frequently to check? It also turns out that most objects are discarded very soon after being used. So you get rid of most of your garbage very quickly. This type of memory management is called Generational Garbage Collection.

Generational GC is one of the reasons Clojure can be so fast. Clojure creates so much "ephemeral" garbage, it's important to be able to allocate and collect it quickly. Java's memory management has been tuned and worked on for years. I tried to find numbers to quantify how much effort has been put into it, but I couldn't. I think it's safe to say that it's in the millions of dollars. Allocation and collection are down to a handful of instructions per object.

Clojure's data structures exercise the JVM's fast GC. While using them is never going to be as fast as pure Java, the JVM does allow Clojure to be practical. Of course, the memory usage still needs to be configured and managed, but the JVM allows us to program at a much higher level and to take advantage of shared-memory parallelism.

If you're not experienced in the JVM, all of those details can be overwhelming. People have told me flat out that they would love to write Clojure but they're afraid of the JVM. This is a shame, because the JVM is what enables Clojure. I wanted to create something that would help people feel comfortable with the JVM without spending years gaining experience.

So I created a five-hour course called JVM Fundamentals for Clojure. It won't turn you into a JVM expert, but it will give you an in-depth tour. It teaches you all of those things that I do day-to-day that I've seen people get stuck with. How do you deal with out of memory errors? How do you navigate this huge standard library? All of that stuff is covered.

You can get the course by buying either a watch online or a watch-online-and-download version. Bot give you lifetime access. Another way to get it is to buy a membership. That way, you get the JVM course and more than 30 hours of other material on everything from web development to macros.

Of course, not everything is rosy on the JVM. Besides the complexity, there are a few things that are not so great that actually make it a less-than-ideal host. We're going to get into some of those next time.

Get this lesson and ten more in your inbox. Sign up below.

Permalink

Literate [Clojure] Programming: Publishing Documentation In Multiple Formats

This blog post is the sixth, and the last, of a series of blog posts about Literate [Clojure] Programming in Org-mode where I explain how I develop my [Clojure] applications using literate programming concepts and principles.

This last post introduce a tool that leverage the side effect of having all the codes neatly discussed: it gives the possibility to automatically generate all different kinds of project documentation with a single key-binding. The documentation that we will generate are:

  1. Human readable HTML general project documentation
  2. Programming API documentation
  3. Book style complete project documentation in PDF

This series of blog posts about literate [Clojure] programming in Org-mode is composed of the following articles:

  1. Configuring Emacs for Org-mode
  2. Project folder structure
  3. Anatomy of a Org-mode file
  4. Tangling all project files
  5. Publishing documentation in multiple formats (this post)
  6. Unit Testing

Documentation, Documentation, Documentation!

Not all documentation is equal and depending on what you are looking for, different kinds and different formats of documentation may work better depending on a task. If I want to read in depth the architecture of a project, if I want to understand the underlying assumptions and decisions that lead to a certain design, if I want to read text with figures about data structures I would prefer reading article like documentation in form of a HTML page or a PDF document.

However, if I am programing something using a certain API, what I will be looking for is the documentation of the API with maybe some supporting material as required. I will be looking to understand how a specific function works, how its parameters are meant to be used, etc. I may also simply be looking for a consolidated list of available function calls. That kind of documentation is quite different than the previous one.

The beauty of Literate Programming is that as a software developed, I don’t have to write each of these types of documentation independently. I have everything I need in my literate programming files (in this case, in my Org-mode files) to generate all kinds of different purposes documentation.

Note that everything that is discussed in this series of blog posts has been applied to the org-mode-tests-utils project. I would strongly suggest you to browse that project along with its folder structure to see how this works in a real [tiny] project.

Now, let’s see how this can be accomplished.

publish.org

What we want to do to generate all the different kinds of documentation is simply having to open a Org-mode file, and to click C-c C-v t which will run all the code blocks in the file that will generate all the documentation we want. That file is called publish.org and is located in the /org/ root folder.

This file is split into three general sections:

  1. Generate HTML Documentation
  2. Generate API Documentation
  3. Generate PDF Documentation

Each of these sections contains the ELisp code blocks that are used to generate the different kind of documentation.

Generate HTML Documentation

What the HTML documentation generator does is to search and find [recursively] all the .org files that exists in the /org/ folder of your project. Then it will generate a sitemap.org file if it is not already existing that it will use to generate the HTML documentation.

Themes

Different themes and styles can be defined for the generated HTML pages. To enable a theme, you simply have to select the proper :html-head setting.

The main themes come from the org-html-themes extension. The theme currently being used is called readtheorg.

Publishing Options

A series of settings can be configured to create the documentation the way you want. Here are the main settings:

  • Settings
    • :base-directory
      • The base directory is the current directory which is the [project]/org/ directory where all the Org files are defined
    • :recursive
      • We specify that we want Org-mode to generate HTML files for each Org file in all children folder (recursively)
    • :publishing-directory
      • We specify where we want to publish the HTML documentation from the Org files
    • :publishing-function
      • We specify that we want to publish everything in HTML
    • :section-numbers
      • We don’t want any kind of section numbers generated by Org-mode
    • :with-toc
      • We want to include a table of content for each generated documentation file
    • :auto-sitemap
      • We want to generate a sitemap automatically. The file is named sitemap.html
    • :html-head
      • We want to specify a style sheet that will be used by each generated HTML file. It should be located in doc/html/css/

Additional settings and configurations are available from these two web pages:

Publish

To publish in HTML, you simply have to run the following code blocks:

(defun org-publish-org-sitemap-includes (project &optional sitemap-filename)
  "Create a sitemap of pages in set defined by PROJECT.
Optionally set the filename of the sitemap with SITEMAP-FILENAME.
Default for SITEMAP-FILENAME is `sitemap.org'."
  (let* ((project-plist (cdr project))
         (dir (file-name-as-directory
               (plist-get project-plist :base-directory)))
         (localdir (file-name-directory dir))
         (exclude-regexp (plist-get project-plist :exclude))
         (files (nreverse
                 (org-publish-get-base-files project exclude-regexp)))
         (sitemap-filename (concat dir (or sitemap-filename "sitemap.org")))
         (sitemap-title (or (plist-get project-plist :sitemap-title)
                            (concat "Sitemap for project " (car project))))
         (sitemap-sans-extension
          (plist-get project-plist :sitemap-sans-extension))
         (visiting (find-buffer-visiting sitemap-filename))
         file sitemap-buffer)
    (with-current-buffer
        (let ((org-inhibit-startup t))
          (setq sitemap-buffer
                (or visiting (find-file sitemap-filename))))
      (erase-buffer)
      (insert (concat "#+TITLE: " sitemap-title "\n\n"))
      (while (setq file (pop files))
        (let ((link (file-relative-name file dir))
              (oldlocal localdir))
          (when sitemap-sans-extension
            (setq link (file-name-sans-extension link)))
          ;; sitemap shouldn't list itself
          (unless (equal (file-truename sitemap-filename)
                         (file-truename file))     
            (let ((entry
                   (org-publish-format-file-entry
                    org-publish-sitemap-file-entry-format file project-plist)))
              (insert (concat "* " entry "\n"
                              "#+INCLUDE: " link "\n"))))))
      (save-buffer))
    (or visiting (kill-buffer sitemap-buffer))))
(setq org-publish-project-alist
      '(("org-mode-clj-tests-utils--doc-html"
         :base-directory "."
         :publishing-directory "../doc/html"
         :publishing-function org-html-publish-to-html
         :section-numbers nil
         :recursive t
         :exclude "fulldoc\\.org\\|project\\.org\\|tangle\\-all\\.org\\|setup\\.org\\|publish\\.org"
         :with-toc t
         :auto-sitemap t
         :sitemap-function org-publish-org-sitemap-includes

                                        ; ReadTheOrg Theme
         :html-head "<link rel=\"stylesheet\" type=\"text/css\" href=\"themes/styles/readtheorg/css/htmlize.css\"/>
<link rel=\"stylesheet\" type=\"text/css\" href=\"themes/styles/readtheorg/css/readtheorg.css\"/>
<script src=\"https://ajax.googleapis.com/ajax/libs/jquery/2.1.3/jquery.min.js\"></script>
<script src=\"https://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/js/bootstrap.min.js\"></script>
<script type=\"text/javascript\" src=\"themes/styles/lib/js/jquery.stickytableheaders.js\"></script>
<script type=\"text/javascript\" src=\"themes/styles/readtheorg/js/readtheorg.js\"></script>")))
(setq org-publish-use-timestamps-flag nil)
(setq org-export-html-style-include-scripts nil
      org-export-html-style-include-default nil)

(org-publish-all)

Once everything is properly configured, (org-publish-all) is run and the HTML documentation get generated in the /doc/html/ folder. You can see what the generated HTML documentation looks like here.

Generate API Documentation

Programming API documentation is also easy to generate. In this case, what we are doing is to use the tangled code to generate the API documentation. Here we are using Clojure’s Codox API code generator to generate the documentation, but we could have used any other such libraries to do the same.

To generate the API documentation, we simply have to run this Clojure code block and use Codox’s API to generate the full documentation.

(use 'codox.main)

(generate-docs {:output-path "doc/api"})

The generated documentation will appear in the /doc/api/ folder. You can see what the generated API documentation looks like here.

Generate PDF Documentation

Finally, we can leverage Org-mode’s internal (org-publish-current-project) internal function to generate a PDF version of each Org-mode file within the /org/ folder (recursively). We leverage the org-latex-publish-to-pdf publishing function to generate the files in PDF.

(setq org-publish-project-alist
      '(("org-mode-clj-tests-utils--doc-pdf"
         :base-directory "."
         :publishing-directory "../doc/pdf"
         :publishing-function org-latex-publish-to-pdf
         :recursive t
         :section-numbers nil
         :with-toc t
         :auto-sitemap t)))
(org-publish-current-project)

The generated PDF documentation will appear in the /doc/pdf/ folder. You can see what the generated API documentation looks like here.

Conclusion

This is what conclude my series of six blog posts about how I do Literate Programming [in Clojure using Org-mode]. As I mentioned in another article, the problem of writing readable code which is well commented, well documented and well tested is that ideally we would have to focus on all these aspects at the same time, but given the development environments used by most people, it is not possible. You will plan an aspect of your program and write the code. Then if you are really lucky and you will find (or take) the time to write some documentation and create some unit tests. The problem is that each of these tasks are siloed: they are performed in isolation with 4 different states of minds, at 4 different times and hopefully within 4 weeks. The worse happens when you start fixing bugs or improving the code: comments, documentation and unit tests will often remain unchanged and lagging behind.

This is what Literate Programming is for me: a way to perform all these tasks at once, with the same state of mind, at the same time. This is a process to put in place, a new way to work. The problem is to put in place that process, that way to work, that enables you do to all this at once. I hope I have been able to put in place and explains a Literate Programming process that can work for you and shows the benefits of doing so.

I have the feeling that it will become more and more important to write readable code and to write about the thought process that lead to that written code. Much of the code we are writing in these days is code that manipulates and transform data, code that implement machine learning workflows and such. The kind of code that greatly benefit to be readable by many people other than the ones that write the code.

Permalink

Clojure in London: Social Superstore

Clojure has helped us to focus on delivering business value

Social Reselling

Social Superstore allows people to setup their own 'store' to review and promote existing products to their social networks. Entrepreneurs are given a helping hand to resell without having to worry about managing stock, logistics and advertising. Celebrities also use it to make product recommendations to their following. The platform has worked well with fashion bloggers who can link subscribers to their own store of selected products.

The tech stack is predominantly Clojure based. They have seven Clojure developers and four developers working on native applications.

James Henderson - an ever present London Clojurian and creator of a host of widely used Clojure open source libraries - grew the current Clojure team from ground up. Dave Aitken is a developer using Clojure for the first time commercially having come from a background of C# and Javascript.

Malcolm Sparks and I paid a visit to their office in Farringdon, London for a chat.

Migrating from Angular.js to ClojureScript

Jon: So what's the tech story here?

James: There has always been Clojure and AWS since a consultancy bootstrapped the MVP. It originally had a Clojure backend and an Angular front end. About 6 months in we started a ClojureScript prototype and we decommissioned the Angular app.

Jon: How long did it take you to move from Angular? What was the motivation?

James: It took about 2-3 months to decommission Angular. The code-base was getting harder to maintain; features coming in were taking longer to develop and the results were buggy. We also wanted the benefits of using ClojureScript; the state management you get with ClojureScript and React.js - with frameworks such as Om and Reagent. We were also a Clojure team.

Jon: You were able to go 'full stack'?

James: Yes, all the devs can work on both the front and back end.

Dave: One of the benefits is that we can put logic in cljc and share code across layers.

Jon: Looking back, do you think the mature ClojureScript solution is better than what you had with Angular?

James: Yes, with Angular there was lots of magic involved. ClojureScript very much suited our team, although the second generation of the front-end was always going to be better.

DevOps

Jon: What is your non-Clojure stack?

James: PostgreSQL, AWS, SQS, Kubernetes, and Docker.

Jon: How do you provision to AWS?

Dave: We use CloudFormation and Chef; they meet in the middle.

Jon: What's your opinion of Kubernetes?

Dave: Kubernetes is great. It was a bit fiddly initially but that's improved a lot. We get smooth, zero downtime deployment. You can throw containers at it, and it becomes super-easy to try out a new service. Upgrading versions of Kubernetes itself has been fine.

Jon: Have you looked at the AWS equvialent of ECS (Elastic Container Services)?

Dave: ECS, certainly at the time, wasn't as feature rich. Kubernetes also gives you a bit more cloud portability; you can port your applications to the Google Container Engine.

Clojure Stack

Jon: What's in your Clojure application stack?

James: We're fairly vanilla Clojure. We use Ring with a general set of custom middleware, Bidi for routing, Medley for a whole host of utility functions, Nomad for configuration. We are using Bounce for lifecycled components.

Jon: Bounce is what you wrote as an alternative to Component? How does it differ?

James: Minor things - I wouldn't push Bounce heavily as it has 'grown' into our application. I wanted to avoid having a partially started system in the REPL which is difficult to recover from. I also personally prefer using vanilla values and functions as opposed to protocols. Mount is more similar to Bounce but I'm not keen on the top-level state; Bounce pulls ideas from both Component and Mount.

Jon: What else are you using?

Dave: I like Yesql for SQL usage; it has a good level of abstraction.

James: We don't have a traditional REST API; we've moved to a command/query API - taking inspiration from Bobby Calderwood's talk at the Conj. We compose API requests as Clojure data structures and the front-end requests just the data it needs as part of 'pull' model. We developed a library for this but it's currently in house, we stole ideas from GraphQL.

ClojureScript

Jon: Have you used Om Next?

James: We tried it on some hack nights but we're very much tied to Reagent right now.

Jon: How have you found Reagent?

James: Generally, it's very good. I would say that there are a few pitfalls that we've learned to avoid, for example where you can potentially close over the wrong variable leading to components remounting/rerendering too often, or not often enough. To get around this we've taken some inspiration from Elm; we keep state at the top in a separate tree and use stateless views with events firing out from those views.

We have a giant multi-method that works off incoming events, which is a lot like Elm.

Dave: Elm is strong with types. Without types it can get confusing - the state tree, the view tree, and events flying about. Having some lightweight assistance on keeping track of the types is helpful; ensuring the trees and data types are correlated.

James: We've stolen a lot of ideas from Elm.

Dave: We also use Devcards, it's really good for prototyping.

The Types Debate

Jon: Where do you guys stand on the types debate?

James: Clojure has a lot of benefits of being dynamically typed; it's second to none for playing with data and prototyping.

A challenge we have is handling data coming from a variety of disparate sources, where the data quality can be poor. Originally having a dynamic structure was useful, but now that the system has matured, we could benefit from more rigorous typing. We've gone for Schema, not so much Spec because it wasn't around.

Dave: By not having types you're taking on tech debt in a way, which is good when you're finding your feet. Once code solidifies you've then got work to add rigour to it.

James: - Our typing is particularly focused on system boundaries - our APIs and data coming from external sources.

As a hobbyist Haskeller I miss ADTs from Clojure.

Dave: Always the grass is greener.

James: Without a shadow of a doubt.

Hiring/Training

Jon: How have you found hiring?

Dave: Everyone on the team applied because it was a Clojure job.

James: We haven't had as many people to interview, but we've had a better hit rate of success. Overall Clojure has been very good.

Jon: How about training?

James: Learning the basics of Clojure is great - it's a really simple language. In my experience, though, people can find it quite difficult making the jump from a hobbyist to writing serious Clojure in production. In OOP, the larger architectural patterns are well-documented; in FP, it's quite a different mindset.

One thing that helps is watching tech talks as a group and going to conferences.

Large Code-bases

Jon: What's the size of your Clojure code-base?

James: About 25K LOC Clojure, 30K ClojureScript.

Jon: Have you learnt any lessons about refactoring on a mature Clojure code-base?

James: The biggest thing is using schema on the system boundaries. In terms of refactoring, the REPL is very handy - we have lots of comment blocks with examples on how to use the code and example data structures. REPL driven development is a big win for Clojure; we iterate quite fast.

Jon: Do you have much automated testing?

Dave: We do have smoke tests that catch quite a lot of things. We also have a lot of monitoring using Kibana and LogStash.

James: A principal we follow is to reduce magic where possible. When you make a change you want to be confident about the impact of that change. Here schemas help us to ensure that impact of changes are well known and small.

We've also split out IO from non-IO code. Before, lots of code pulled out data from the DB and munged it etc. Now that we've split it out, the amount of IO code has become smaller; reducing the surface area for things to go wrong.

State of Clojure

Jon: Your view on the state of Clojure?

James: Clojure has been very stable since I came here in January 2015. We have a good handle on how to architect Clojure code; and for data transformations it's particularly well suited.

ClojureScript, less so. Having a ClojureScript project a couple of years old you can now see the 'lava layers'; the record of everything that was best practice at the time. It's still changing quickly.

The decision to go on the JVM has benefited us hugely. The industry has 20-30 years of knowing how to deploy Java applications.

Dave: Things like New Relic working straight away has been a plus.

Jon: What do you think the state of Clojure is in London?

James: London seems to be a hub for Clojure. There are lot of community events: dojos, Clojure Bridge, multiple conferences.

Clojure itself has stablised, now coming up to 10 years old. It's not as 'new' as in the early days.

Wrap Up

Jon: Any last words?

Dave: Thank heavens we're not using Java.

James: Clojure has helped us to focus on delivering business value, to help us get to the core of our business requirements.

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.