I had the great honor to appear on Conversations about Software Engineering. We talked about functional programming and my book, Grokking Simplicity. https://cdn.podigee.com/media/podcast_998_case_conversations_about_software_engineering_episode_476020_eric_normand_on_functional_thinking.mp3
Let's start right in the middle of the story, and play with cosine similarity. I'll
tell you why in a few minutes.
I assume that most programmers have forgotten their math classes, so
"cosine similarity" sounds somewhat grandiose. Quick skimming at Wikipedia might puzzle
you even more, but, scrolling down to the definition, you can
see that it's just a normalized dot product. I know, I know - now you might wonder what
dot product really is. Please read my recent linear algebra Hello World article if that's the case.
A little warm up coding
We are programmers; it will all be clear after we code this.
Here are two vectors:
(defx(dv 1 2 3))(defy(dv 30 20 10))
Dot product
Now, the dot product can be naively implemented as following.
(defnnaive-dot[x y](reduce + (map * x y)))
(naive-dot x y)
=> 100.0
It's much easier and faster to use the staple linear algebra function called dot:
(dot x y)
=> 100.0
Normalization
Next, the normalization part. That might seem grandioze too, but
it is, again, a staple LA function nrm2 (the second norm of a vector). Not
only that it's a staple function, but it's just a square root of the
dot product of a vector with itself.
(defnnaive-nrm2[x](sqrt (naive-dot x x)))
(naive-nrm2 x)
=> 3.7416573867739413
A better choice in higher level code is to use the optimized nrm2 function available in Neanderthal:
(nrm2 x)(nrm2 y)
Cosine similarity
This is cosine similarity between vectors x and y:
(/ (dot x y)(nrm2 x)(nrm2 y))
=> 0.7142857142857142
OK, nice exercise, but not that interesting. There's no particularily
useful insight there. I implemented everything naively, and then wrote
my own cosine-similarity, which is, after all, the same thing as any
library's cosine-similarity (it's probably faster but that's irrelevant here).
We now have a way to compute a measurement of similarity of two vectors (cosine similarity),
which is a number between -1 (opposite), 0 (not similar at all), 1 (they match), or somewhere in between.
In our case, x and y are somewhat similar (0.714) which does not tell us that much, but it can
tell us that they are more similar that z if cosine similarity with z is, say, 0.5.
So what?
This was only a warm-up, and the insight follows after the intermezzo: we can compute
cosine similarity of many vectors in a batch much faster than one by one. But why? And when?
Well, if we would have only blindly used what the library is providing,
we probably wouldn't even been asking these questions!
Now, the motivation for all this
A few weeks ago a Clojurist was asking about a weird result returned by
a function in a Clojure/Java machine learning library. There's nothing unexpected
there; bugs are caught in all software all the time, and the author of
that (open source and free) library fixed the bug immediately. It's a
good pretext to make a point, though, so let's make it :)
We often rely on software libraries. Layers of abstraction make our work
possible. Something we stretch the tendency to rely on 3rd party
abstractions too much, intentionally giving up control of the central
part of our code. The irony is that many times the simplest thing is to
implement the thing using the right lower layer, than to configure a
function call of the higher layer.
In this case, the bug was that a cosine similarity funciton returned a value larger than 1. The
user (correctly) wondered whether it's an impossible value for cosine
similarity. Several people helped with theoretical knowledge and
practical guesses where the bug might be. How such a simple thing such
as cosine similarity could be the source of any fuss? Probably because
most programmers rely on libraries so much that they don't build
confidence even for the simplest things related to math.
That's why I like to not skip the step of implementing custom functions for central parts of
the algorithms that I rely on. Sometimes the existing solutions serve
well, and I use them in the final code, but by building at least a naive
prototype I build up my understanding and confidence. I also found out
that my solution is often much better than the one implemented in a
library. If that seems unexpected, consider that machine learning
libraries are frequently written by grad students on their path to
discovery. It's a domain expert with poor programming skills.
Or it might be a case of a good programmer who only barely understands the domain…
Is that it? NO :)
So, we implemented our naive cosine similarity (nothing special).
(defncosine-similarity[x y](/ (dot x y)(nrm2 x)(nrm2 y)))
We choose this, or the (hypothetical) library function with the same name,
and proceed with looping, mapping, reducing and other Clojure-fu?
NO! This would limit us to calling such function for each pair of vectors.
What works well in number crunching is finding an operation that does the same
for the whole bunch in one sweep!
But, how do we do that? If we hadn't explored the internals of cosine similarity,
that'd be a mystery. But, since we had, we can recognize that matrix multiplication
is a bunch of dot products. Explaining the details of this insight would digress
too much; I cover this kind of knowledge in the Numerical Linear Algebra for Programmers book.
So, let's put these two vectors, x and y, in a matrix and call it xs.
(defxs(dge 2 3
[[1 2 3][30 20 10]]{:layout:row}))
When we multiply two matrices, we get a third matrix, and each entry in the result
is the dot product of the respective row of the first matrix and column of the second.
For example, entry (2,3) is the dot product of the second row of the first and the third
column of the second matrix.
In our trivialized case, we have one matrix, xs, with two rows, and we don't have
any other matrix. What now? Since we want all combinations of dot products of our vectors,
we can simply matrix multiply xs by its transpose!
Do you recognize the familiar number 100.0? That's the dot product of x and y.
Notice that this matrix is symmetric, so we can optimize this by using a dedicated function
optimized for matrix multiply with transpose, mmt.
(defx-dots(mmt xs))
Now we only need to normalize these. But how? There's no trick with matrix multiplication for nrm2. Or is there?
Remember that we also learned how nrm2 is computed: as the square root of the dot product of a vector with itself.
The diagonal holds the dot products of x and x \((1,1)\), and y \((2,2)\)! We only need to square the diagonal.
While we're at it, we might also invert it, so we can later deal with multiplication instead of division (but this is a minor detail).
All these vector operations that we need are available in Neanderthal's vect-math namespace.
(defx-norms(inv! (sqrt! (copy (dia x-dots)))))
Now we have the dot products in matrix x-dots, and the norms in vectorx-norms.
Whoops… How do we combine that? The formula for cosine similarity uses each combination of
norms for vector pairs, not single norms. No problem! We pull the rk function from our pocket.
It does the outer product of two vectors, making a matrix that contains the product of each pair of their entries!
Again, this is not some one-off trick, but a standard linear algebra stuff explained in the book.
BAM! Here it is! 0.71! Ones on the diagonal show that each vector is perfectly similar to itself :)
This might seem much fuss for nothing! Isn't the vector code simpler? Yes, but it only can compute one combination of two vectors.
In this example, we've done the most trivial example with the same two vectors. But this code computes as many vectors as we give it!
No need for loops or recursions!
Pimp my cosine similarity!
I've added some bookkeeping code. Clean up our memory and reuse matrices if possible for maximum speed gains.
See? All the combinations of the vectors with one single blow! And, if we have thousands of vectors
it's orders of magnitude faster than calling the single cosine similarity function thousands of times!
A post from Arne Brasseur about some changes for how Lambda Island makes videos. Making changes like this can be tough, but it seems like this is going to be more sustainable.
Libraries & Books.
re-frame-flow helps you visualise a chain re-frame events.
Methodical is a replacement for Clojure’s multimethods. It adds some neat features from the Common Lisp Object System, like adding wrapper methods, and the ability to call the next most specific method (kinda like calling a “superclass” but against the multimethod hierarchy). 0.12.0 is out now.
OpenJDK has a new project Lilliput. The goal is to explore reducing Java’s object header size (in Hotspot) from 128 bits to 64 bits or less. As you can imagine, if succesulf, this would have a substantial impact on Java’s memory footprint.
Tools.
A patch for adding Clojure language support to Git’s diffs. You can add this yourself to your own Git config, but it’ll be nice to have it built-in.
An interesting proposal for a web app architecture. The server would send DOM updates over a websocket to the client (similar to Phoenix LiveView). See more discussion on ClojureVerse.
I’m Daniel Compton. I maintain public Maven repositories at Clojars, private ones at Deps, and help fund OSS Clojure projects (along with tons of generous members like Latacora, Roam, Pitch, Nubank, Cisco, JUXT, Metosin, Solita, Adgoji, Nextjournal, Flexiana, Toyokumo, Griffin, and [Parkside][parkside]) at Clojurists Together. If you’ve enjoyed reading this, tell your friends to sign up at therepl.net, or post a link in your company chatroom. If you’ve seen (or published) a blog post, library, or anything else Clojure/JVM related please reply to this to let me know about it.
If you’d like to support the work that I’m doing, consider signing up for a trial of Deps, a private, hosted, Maven Repository service that I run.
With ever-growing workloads on the cloud, companies face significant challenges in managing productivity and spending, and maximizing impact to their businesses. Compute Software is addressing a huge market opportunity to help companies make good business decisions and run optimally on the public cloud. We're building a powerful platform to give customers the tools they need to gain transparency, optimize their usage, and execute change across their organizations.
We're a small, distributed team, currently spanning California to Quebec, and we offer early stage market-rate compensation (including equity), health and dental insurance, and 401K benefits. You'll be joining a venture capital-backed, distributed team with ambitious goals, and you will have the ability to make a direct and lasting impact on the team and product.
Your Role
Be one of the earliest employees and join our early-stage engineering team as a Senior Software Engineer. You will be essential in shaping the features and functionality of our SaaS platform, culture and processes.
You'll spend your day enveloped in Clojure. The backend is written in Clojure and driven by data in Datomic and InfluxDB. The frontend is written in ClojureScript using re-frame and communicates with the backend using Pathom. We deploy to AWS Fargate and Datomic Ions.
For product development, we follow Shape Up. We use Notion, Slack, and ClubHouse.
What will you do at Compute Software?
Write Clojure and ClojureScript.
Design, build, and deploy features and bug fixes across the entire stack.
Become an expert in all the nuances of the various cloud platforms like Amazon Web Services, Google Cloud, and Microsoft Azure.
Provide product feedback and evaluate trade-offs to impact product direction.
Debug production problems.
What We're Looking For Passion - you are excited about the large, high-growth cloud computing market and figuring out how to help customers, who are using cloud computing solutions today. You are excited by being one of the earliest employees and getting to work with a small team.
Engineering experience - you're a Clojure practitioner with 6+ years of professional experience. You know what it takes to create large-scale b2b software. You can create effective and simple solutions to problems quickly, and communicate your ideas clearly to your teammates.
Product-minded - you love building products and you care about the details of creating a great user experience. You have an interest in how users will use our platform and the impact we will have on them. You can balance your consideration of the product and user requirements with technical complexity and implementation details to make appropriate decisions when things are unclear.
Effective communication - you're great at communicating. If something is unclear you reach out and ask questions. You're comfortable owning, communicating and presenting information on specific projects or initiatives, both in writing and in person.
Organizational and project management - you are highly organized and able to self-manage projects in a fast-moving company. You are able to take high level goals and break them down into achievable steps.
Conceptually metadata is an ability for the symbol or the collection to have some additional information for the Clojure compiler. It is implemented as a map data structure and is often used to convey the type information to the compiler, documentation and compilation warnings.
Welcome to the Clojure Deref! This is a periodic link/news roundup for the Clojure ecosystem. (@ClojureDerefRSS)
Highlights
HOPL IV (History of Programming Languages) at PLDI 2021 is happening on Monday and Tuesday and includes a talk from Rich Hickey about the History of Clojure paper. Registration is still available and the conference is online and features many other fine language developers!
The results are out from the JVM Ecosystem Report 2021 and Clojure continues to make a strong showing as one of the most popular JVM languages (other than Java), rising from 2.9% last year to 8.4% this year. Lots of other interesting tidbits in there as well.
Sponsorship Spotlight
Lately Christophe Grand and Baptiste Dupuch have been teasing their work on a new ClojureDart runtime with Flutter support. You can support their work on GitHub: cgranddupuchba. Nubank (users of both Clojure and Flutter) are now supporting both!
Podcasts and videos
LispCast - Eric Normand talks about stratified design
defn - Vijay Kiran and Ray McDermott interview Paula Gearon
refl - Michiel Borkent released a new example project to clean up reflection configs for GraalVM for Clojure projects
Fun and Games
Fidenza - Tyler Hobbs has a long history of doing interesting generative art with Clojure and he has published a rundown of his newest generative algorithm. Not explicitly Clojure but fascinating to read.
I have been writing Clojure professionally for the past 5 years, but I still, have a hard time recommending Clojure to friends and clients. My friend Anand, co-founder of Pabio aptly said that:
JavaScript for me is a language for making prototypes, and not a system that you want running after a decade from now.
Although I’m a Clojure fan-boy and advocate, the realist in me knows that Anand is right. Clojure has an adoption problem.
What is Clojure?
Clojure is a functional LISP that with a futuristic feedback loop. It’s hosted, has a strong following, and often reaches the first page of Hacker News. I’ll not go into the details of the language as manyarticlesalreadyexist about this topic. Clojure’s simplicity and expressiveness is a recurring theme in most descriptions of the language.
But this article is not about Clojure’s beauty. It’s about the challenges (and possible solutions) that stand in the way of mass adoption. It’s not about being a successful artist, it’s about being successful at the art bazaar.
What is the adoption problem?
Despite the widespread praise about the Clojure language, we cannot refute the fact that it’s hard to find Clojure developers. On the other hand, it’s also hard to find a Clojure job.
The problem turns into a chicken and egg paradox. There won’t be enough developers until there are enough jobs, and there won’t be enough jobs until there is a big talent pool.
A small talent pool leads to a small open-source footprint and prevents network effects. If few developers write Clojure, fewer new developers will be introduced to the language, amplifying the chicken and egg problem. This is countered to some extent because Clojure is hosted and can tap into vibrant eco-systems of JavaScript and Java.
Writing software is the easy part of running a software company
All software is a tool to solve a real-world problem. Companies and organizations exist to solve real-world problems. Until the 1950s, real-world problems were solved by war. Post the second world war, we shifted to data-oriented solutions.
As software developers, we concern ourselves with the issues about our domain. For a person who can code, every problem is an opportunity to make a SaaS company. But if we zoom out and look at some successful companies, we’d know that larger risks exist in the market.
Markets change and we need to react. There is a competition or lack of demand. There are legal issues and privacy issues. A bad culture could lead to people issues. And most of these are beyond our control. Looking at the full picture, we see that software is the most predictable and risk-free aspect of a software company. Since this blog’s audience is mainly software developers, I usually don’t talk beyond software. But this specific topic of basing your stack on Clojure requires me to address the other things that go into making good software.
All good software companies start with a strong grasp of the market and grow with a strong culture and the ability to react. If you have zero users, you will probably be solving technical problems. If you have hundreds or thousands of users (depending on your market) you’ll be solving sales and marketing problems.
Once you have crossed maybe a million users mark, you step into the realm of people and cultural problems. And post 10 million, you have to face political problems.
All languages are tools and all tools are good enough
From a zoomed-out perspective, all languages are tools. And all mainstream tools like Java, JavaScript, TypeScript, Ruby, etc are good enough. An experienced JS developer is as effective as an experienced Clojure developer at building software.
Although all languages are good enough, some of them offer special capabilities. Like C++ and Rust are close to the metal. Go, Elixir and Clojure have excellent concurrency primitives. Java, JS has the largest talent pools. Python has the support of the ML community. TypeScript has Microsoft.
These are broad generalizations. Most languages have overlapping strong points. Python has a large talent pool along with being the language of choice for data science.
Then how do you decide what tool to use? And when do you pick up something like Clojure? To answer these questions, we need to analyze the usefulness of tools from the perspective of a “software company”.
Properties of good tools
We have already ascertained that writing software is the easy part and risks lie outside our servers: in the market, in the culture, in the political scenario. That being the lens of our camera’s perspective, a good tool should have the following properties.
Make your process easy
A table-saw makes it easier to cut wood. Hasura GraphQL engine makes it easier to ship APIs. Postgres makes it easy to store data. Functional languages make it easy to handle side effects.
Be easy to comprehend
Complicated concepts do not survive the test of time, and are eventually replaced with simpler alternatives. It’s hard to find a counter-example because most tools that reach mainstream usage are easy to comprehend.
Extend like legos
Good tools are pluggable because an architect relies on multiple tools. REST APIs are a great concept. The SaaS community shifted radically from the server-rendered MVC pattern to REST APIs. You can argue that the shift was due to the explosion of clients (mobile, desktop, IoT), but it still serves the argument. A tool that can be modified and extended can serve more purposes.
Last for long, ideally forever
There is a cost associated with mastering a tool. And if the tool dies before you have reaped a significant return on investment, then you made a bad deal.
Get out of your way
A tool is designed to ease your process, not dictate it. It’s okay to have rules and limitations on what a tool can do, but it should never dictate the how.
Make you feel confident about the work produced
A tool is no good if you cannot be confident that your app will run. Tests, Types, Pure Functions, and Side-Effect free systems are examples of tools that provide this confidence.
Enable you to react to market
Ultimately, you survive because you adapt to the market’s condition. According to Rich Hickey, the single largest factor that decides how quickly you can react is the code you have right now. He calls it the “elephant” in the room. I believe the same, but Rich Hickey saying this adds a lot more credibility:
Simplicity is hard work. But, there’s a huge payoff. The person who has a genuinely simpler system — a system made out of genuinely simple parts, is going to be able to affect the greatest change with the least work. He’s going to kick your ass. He’s gonna spend more time simplifying things upfront and in the long haul he’s gonna wipe the plate with you because he’ll have that ability to change things when you’re struggling to push elephants around
Clojure is almost as pure and sophisticated as Haskell, and as easy as Python or JavaScript. The syntax might seem off, but the returns on learning the syntax are astronomical, because of the REPL.
It gets your way and lets you tap into rich library eco-systems of Java and JavaScript. Many functional languages suffer due to a lack of open-source network effects. Clojure bypasses the issue by feeding off network effects of other ecosystems.
And no praise is enough to describe the REPL. It accelerates development to an extent that most other languages can’t even imagine. I always wondered if there was a REPL for other languages. There are simpler versions for JavaScript like Quokka.js. u/didibus pointed out Clojure properties that enable the Clojure REPL on this Reddit thread.
It would be unfair to say that Clojure is the only exceptional tool so please consider this to be a personal opinion. Clojure is balanced. It makes hardcore mathematical concepts, commercially feasible.
Should you choose Clojure?
This is the trick question. There is no right answer but experiences and opinions to build on.
In my opinion, you should not learn Clojure at your own expense. If you are building a company and have no prior Clojure experience, then it would be a terrible choice. It will slow you down and will hinder you from focusing on the market.
Are you building a startup or are you churning out code? If you have time to consider these questions then you’re not selling enough. You need to stay focused on the business.
If you want to tinker, then do that on other people’s dime or save that for a hobby project.
If you already know Clojure, it becomes your super-power. It gets out of your way and helps you focus on the market. It may be hard to find Clojure talent, but it’s not hard to nurture it. Clojure might not be the easiest language to get started with, but senior developers can pick it just like any other language. Here’s what Reddit thinks about the Clojure hiring situation. From my personal experience, Clojure companies hire regardless of Clojure experience.
u/bostonou introduced Clojure at Vanderbilt university:
I helped introduce Clojure at Vanderbilt University working in genetics research. I then moved to a Clojure startup, did some Clojure consulting, and now do Clojure at CircleCI. Overall I’ve been doing Clojure professionally for ~8 years.
We’ve used Clojure at the heart of our stack for the last five years and I consider it one of our superpowers.
It was less a leap of faith but a decision based on exceptionally positive previous experiences. I first trialed and adopted Clojure at a health tech company back in 2013 and then built a marketplace startup for hobbyists and collectors with it in 2014. Both times were productive and fast time to market. The composability of lisp and FP makes rapid reliable development not only achievable but the norm.
How tough is it to hire Clojure talent?
Adam from WeFarm also has a unique perspective on hiring Clojure developers:
Over the years I’ve also discovered that hiring for Clojure is also different in a positive way. In more mainstream languages I’d get a huge stack of CVs. Amongst them would be great and not-so-great candidates and it’d be a lot of effort filtering the wheat from the chaff. With Clojure roles, the pile of CVs is a fraction the size but you can guarantee 99% of the candidates have a curious and investigative character that enjoys learning new things. Exactly the characteristics I look for in candidates!
The “curious investigative character” of people who write Clojure has held in my personal experience. No one would take the trouble of learning a language like Clojure (or Haskell) unless they deeply care about their work. This property will hold until Clojure jobs become as plenty as JS jobs.
At my company Status, we hire regardless of Clojure experience. Since we are an open-source company, we pick up simple issues that we believe any engineer should be able to solve regardless of the language. All prospective hires are expected to pick one issue and raise a PR as a coding challenge (candidates are compensated for their work). Status is hiring for its React Native based app written in ClojureScript if you are interested!
I chose to use Clojure due to the incredible caliber of the average Clojure developer, because it’s virtually perfect for web applications and data processing, and because developer productivity is much higher than in other languages I’ve used (Python, Perl, Ruby, JS).
If you are a part of a big company, training talent internally too is a viable option. I have seen a company take over a 5-year-old codebase and develop a 20 person team around it. The entire transition took about 8 months.
Conclusion
I’m a Clojure fan, but more than that I’m a problem solver. And Clojure is not the right tool for everything. If your problem involves handling a ton of data and building APIs and UIs, Clojure is a great system. If you want to quickly build a team around your project then probably not. Clojure is a super-power that enables you to better react to changes outside the system.
Recommending Clojure was hard because of a lack of talent and resources. But I feel this is changing. To contribute to the change, I built a course about how to build React applications with Clojure. I also collected a list of free resources that you can use to jump-start your Clojure journey.
Last month marked the five year anniversary of Lambda Island. Five years since I
quit my job, built a video platform, and figured out how to put together
tutorial videos. It’s been quite a ride. All this time I’ve been fortunate to be
part of the Clojure community, to watch it grow and evolve, and to help
individuals and companies to navigate these waters.
I learned some hard lessons along the way. Like how hard it is to bootstrap an
educational content business catering to a niche audience, or how lonely and
stressful it can be in business to go it alone.
And with these lessons came changes. I started doing more consulting work again.
To some extent it was a necessity, but it also provided me with an opportunity
to get involved with many Clojure projects out there in the wild. To work with
talented individuals, to learn what amazing things people were doing, and to
help shape these companies, products, and the greater narrative around Clojure
as a technology and community.
Out of this work grew many open source libraries and tools, like
Kaocha,
Regal,
Glögi and
Corgi, which became part of the Lambda
Island brand, and indispensible items in people’s toolbelts.
But I was still going it alone, and I didn’t like that, so in 2019,
Gaiwan was born, a bona fide Clojure shop, and slowly but
surely I started putting together a team. Fast forward to today. We are now a
team of 6, spread around the globe, working with amazing clients like
Nextjournal, Eleven,
Pitch, IT Revolution, and
A-Knowledge.
We’ve been busy! And not just with client work. We’ve continued to grow and
improve our open source offerings. We’re still running
ClojureVerse and
Clojurians-log for the
community. Our blog has gotten regular
updates. We’ve reached a lot of people on YouTube during the Advent of
Code,
and we even organized a conference, Heart of
Clojure.
What we haven’t been doing is posting new video content to Lambda
Island. I could say that with everything else
happening it has been nearly impossible for me to carve out the time to work on
new episodes. That’s not wrong, but the real issue is that I burned out on
making them. I would easily spend a full week on a 15 minute episode, slaving
over the script, agonizing over the details, and meticulously editing the final
result. Then it would go live and…crickets. Freebies would maybe get a few
comments on Reddit and a hundred views. Paywalled episodes would get a few
dozen. Neither would really bring in new revenue, and I’d be scrambling
afterwards to catch up with client commitments.
In hindsight it was a pretty toxic feedback loop. I suffered under my own
perfectionism, and with that continued to set the bar high for each subsequent
episode.
But this isn’t a story of doom and gloom. Instead it’s one of necessary change.
We’re a team now, and everyone on the team is excited to help plot the future of
Lambda Island. We don’t know exactly where we’ll end up, but we know the general
direction we’re setting out in.
One thing that is clear is that Lambda Island is going to start looking and
sounding a bit different. Before, Lambda Island was very much my voice,
literally and figuratively. One person’s style of writing, level of experience,
and view on technology. Going forward we are going to be a multitude of
voices. The first place you are going to notice this is on this blog, where both
Ariel and Alys have already started publishing some
articlesof their
own.
Meanwhile Felipe—with help from Ariel—has been hard at work behind the scenes to
give the site a major upgrade, the biggest overhaul the site has seen since we
launched in 2016. The result should look and feel much more fresh and modern, be
faster, more responsive and accessible, and allow for better attribution of
authors.
We also know that we want to put more effort into fostering a vibrant community
around Lambda Island. One that values curiosity, creativity, and exploration;
and where beginners can feel welcome and supported. We set up a Lambda Island
community on Discord and started inviting a few dozen individuals while we set
things up and get a feel for the place. We’ll soon be sending out more invites,
starting with Lambda Island subscribers and to our open source supporters on
Open Collective. The whole Gaiwan
team hangs out there as well, and so going forward this will be the best way to
get in touch with us and get involved in conversations. Don’t be a stranger!
As for video content… we don’t know yet. The most likely scenario is that you
are going to start seeing more uncut and unscripted content, like the Advent of
Code videos, instead of the highly condensed and polished episodes from before.
Unfortunately Pascal’s shorter letter applies here as well… I would have made
a shorter video, but I did not have the time.
It does seem a lot of people enjoy these uncut videos, bloopers included. You
get to really see how someone works in realtime, how they use their tools and
think through problems. For a language with a highly idiosyncratic and
interactive workflow like Clojure this is invaluable.
Lambda Island is changing. Change can be a little uncomfortable, but it’s a
constant of life. With this post I want to pave the way for the rest of the team, so that
they can be comfortable making their mark. We’ll talk in more details about some of
the things we are up to in upcoming posts. Meanwhile if you’d like to reach out
then you can contact me on Discord as Arne#3086. Don’t be a stranger!
Long time no see, my friends. We have been busy building stuff at Magnet and in this hot day, hiding from the scorching sun, I would like to tell you about a very special product that we have created in this hectic year; a year where we are all navigating each in our precarious way through the changes this post-pandemic (fingers cross!) world has brought to us.
enerGIS is how we called this business venture and it’s led by Matthew Barlow, a brilliant mind and better guy. enerGIS is a web application that assists decision makers (governments, green consultancies, energy producers,…) understand solar and wind resource potential across a territory. It analyses, among other things, parameters such as distance to infrastructure, load centre proximity, land use classification and more — such as topography, bathymetry and environmental restrictions.
According to researchers from World Resources Institutes land planning will play a pivotal role for the successful deployment of any renewable energy project. We have several country datasets already available and we are adding new ones in the coming days, with the mission to map the whole world, contributing to a greener and more sustainable future.
The International Energy Agency (IAE) has designed a pathway to reach the goal of net-zero CO2 emissions in the energy sector by 2050. As part of this challenge, by 2030, the solar and wind energy capacity added to the system will have to be three times the capacity installed in 2020. This is certainly challenging, but it’s worth noting that many developing countries are still in the infancy stage of leveraging their renewable energy potential.
And here is where enerGIS comes to play: a simple web application that can be suitable for decision makers who want to analyse the potential of a country or territory, an online tool that takes the information out of the technical back rooms and brings it out in a simple, clear way.
I would not go into the technicalities of the project, colleagues will comment on these in later posts (of course it has been built in Clojure(script)!), or explain the features that the tool brings (you can find out more in here), I just wanted to announce that the product is available in a SaaS model (in private beta right now) as well as client-side deployment for Enterprise versions, and we plan to make it available to the public during this summer.
A picture is worth more than a thousand words, so here you have a sneak peek of enerGIS:
Recently FOMO got to me and I jumped on crypto speculation game. You gots to do bunch of accounting/counting/reconciling/tracking in it. Sounds boring, but clojure made it fun. And that’s what I want to share here.
:clojureD was recently held online, the talks should be up on YouTubesoonish.
Nubank raised a staggering $750M last week, on top of a $400M round in January. Nubank plans to use the funding to launch in Mexico and Colombia, launch new products and services, and hire more employees. That still seems like a lot of money, I’m excited to see if there’s anything else they plan to do with it.
The Stack Overflow Developer Survey is on right now. I’d encourage all Clojure developers to fill it out to help the Clojure community be represented.
Chris Nuernberger gave a talk on High Performance Data for the London Clojurians Meetup. Chris (along with others in the scicloj community) has been pushing Clojure a long way in the data analysis/data processing space.
Alex Miller has started Clojure Deref, a link/news roundup for the Clojure ecosystem. If you like this, you should check out Deref too.
Jordan Miller continues her podcast Lost in Lambduhhs with Alex Miller.
Nathan Marz gives us a tour through their 250k line Clojure codebase. That’s a pretty large Clojure codebase, and it sounds especially dense with 400 custom macros (Clojure’s source has 127). However, they’re building a programming language on top of Clojure so it’s perhaps not too surprising. Nathan Marz answered a few followup questions on the Clojure mailing list, including a few more hints on what they’re doing with continuations. I’m looking forward to seeing more.
Libraries & Books.
mprop is a nice extension to test.check with two useful extensions: localising test.check failures to the assertion, not just the test; and scaling the number of tests based on environment so you can run 10x more in CI than in local dev.
snoop adds some syntactic sugar to define Malli function schemas.
Foundations.
Crux has a Bibliography of books and papers that inspired Crux’s philosophy and architecture.
Tools.
Tips and tricks on how to use Clojure with GraalVM from Lee Read
Michiel Borkent continues churning out useful Clojure tools This time it’s scittle, a lightweight tool for running ClojureScript directly in the browser with no precompilation. View source on the linked page to see what it looks like, very cool!
An alpha for Expectations 2.0.0 is out, if you use Expectations, now would be a good time to give your feedback.
A list of OSS datalog databases (from the Crux team). The comparison looks fair, no one vendor gets all ✔️’s.
Learning.
An interactive guide on how to build a tetris game in ClojureScript from Shaun Lebron. Even if you don’t want to build a game, it’s worth reading, just to see an example of how you can make code explanations much more tangible.
Misc.
I’ve recently started stretching to end my day. It’s a good wind down routine to relax your muscles. There are a bunch of videos on YouTube to pick from, depending on how much time you have and how flexible you are to start with.
I’m Daniel Compton. I maintain public Maven repositories at Clojars, private ones at Deps, and help fund OSS Clojure projects (along with tons of generous members like Latacora, Roam, Pitch, Nubank, Cisco, JUXT, Metosin, Solita, Adgoji, Nextjournal, Flexiana, Toyokumo, and Griffin) at Clojurists Together. If you’ve enjoyed reading this, tell your friends to sign up at therepl.net, or post a link in your company chatroom. If you’ve seen (or published) a blog post, library, or anything else Clojure/JVM related please reply to this to let me know about it.
If you’d like to support the work that I’m doing, consider signing up for a trial of Deps, a private, hosted, Maven Repository service that I run.
I was working on re-implementing the Stochastic gradient descent algorithm in
Clojure based on the blog post How to Implement Linear Regression From Scratch
in Python. Because I wasn't happy about the performance when the dataset is in
the order of thousands rows of thousands features, so I started looking into
ways to go lower-level to use the Java arrays and found this amazing article on
Clojure Goes Fast: Java arrays and unchecked math. (It's a great article that
goes into details of Clojure type-hinting and other tips. Highly recommended!)
Following the instructions in the Clojure Goes Fast article, I'm quite amazed by
the performance boost. Below is a comparison of the same predict function: one
expects Clojure vector and the other Java arrays. The only thing novel thing
that didn't cover in the Clojure Goes Fast article is the usage of
areduce. Instead of the explicit loop-recur shown in the article, I stumbled
upon this handy clojure.core/areduce form and used it to better convey the
implementation of the predict function.
Evaluation count : 834 in 6 samples of 139 calls.
Execution time mean : 736.026481 µs
Execution time std-deviation : 26.489882 µs
Execution time lower quantile : 715.074957 µs ( 2.5%)
Execution time upper quantile : 777.624246 µs (97.5%)
Overhead used : 6.224349 ns
Evaluation count : 106956 in 6 samples of 17826 calls.
Execution time mean : 5.865484 µs
Execution time std-deviation : 221.419267 ns
Execution time lower quantile : 5.580842 µs ( 2.5%)
Execution time upper quantile : 6.125437 µs (97.5%)
Overhead used : 6.224349 ns
It's about 736 µs versus 5.87 µs. That's about a 125X speedup by using Java array.
It's really nice that I don't have to sacrifice the functional programming
constructs (like map1 and reduce) because I'm working on a different data
structure. Under the hood, areduce is a macro that does the loop-recur for
you. Here's the source code of clojure.core/areduce:
(defmacro areduce
"Reduces an expression across an array a, using an index named idx,
and return value named ret, initialized to init, setting ret to the
evaluation of expr at each step, returning ret."
{:added "1.0"}
[a idx ret init expr]
`(let [a# ~a l# (alength a#)]
(loop [~idx 0 ~ret ~init]
(if (< ~idx l#)
(recur (unchecked-inc-int ~idx) ~expr)
~ret))))
Using this macro has an advantage over using the loop-recur: you don't need to
explicitly call the unchecked-inc-int to get the full benefit of speeding up
looping on the array a.
Clojure has number special forms. Among them are let and
binding that are implemented as macros. For new comers to Clojure,
above can be confusing to understand, which I hope to address in this
blog post. If you have read "Programming Clojure" book, you might be
thinking that it is easy to use them correctly considering following
straightforward example:
(def x 7) ;; bind the var x to the value 7
(defn print-x [x]
(println x))
(let [x 9]
(print-x)) ;; prints 7
(binding [x 9]
(print-x)) ;; prints 9
Why? Because let creates a lexical scoped vars, which is local to
the code block or functions where it is initially defined while def
creates a global var that is visible to all functions under the
namespace that you are in. Therefore, in the example above, let just
created a new "x" with the value of 9. But it is not local to the
function "print-x". That is to say, "print-x" have no knowledge of the
local scoped binding that is created by let. binding macro instead
creates a new binding for the already existed var, "x" with the value
of 9. Since "print-x" is evaluated within the binding form, the
newly bound value is visible to any chained calls within the form. All
nice and easy. However, it is not that obvious to understand and use
them correctly. Look at following example:
(def x 7)
(def y 9)
(let [x 5 y x] y) ;; returns 5
(binding [x 5 y x] y) ;; returns 7
let example is just fine. It created a new var "y" with the value of
5 bound to it. But what is going on with the binding example? If we
dig little deeper to Clojure API doc, we learn that the new bindings
in binding are made in parallel, not sequential as it is in let.
Therefore, existing var "y" gets bound to the root binding of "x" not
the overshadowed value of "x", which is the case in the let example.
OK, so far so good. Let us look at another example:
All the lines but the last two are pretty straight forward. But how
about the last line? Shouldn't it call "print-y" function? Not really.
It is because the anonymous function is evaluated outside of the
binding form. The "#<user$eval_28$fn_30
user$eval_28$fn_30@bdb503>" is an indication that anonymous function
did not get evaluated. The extra parens evals above form. Ok,
now, how do you make sure it evaluates within the binding form? Just
enclose the anonymous function within parenthesis. What do we learn
from all of the above? Here are the summary that helps one use let
and binding correctly:
Use binding only if you are trying to alter a behavior dynamically
for special-vars that is already defined. Enclose your special vars
within * as recommended by idiomatic clojure to make your intent
clear.
Use let for locally scoped vars and avoid shadowing already
defined vars.
Remember let binding is sequential and binding binding is done in parallel.
Pay close attention to the scope of the binding form. Any
expression that is not evaluated inside the form, will not see the
binding. That is to say, binding in binding form is thread local
starting from the binding form until it ends including any
expression that is chained within the form, which is what the
thread-local are all about.
Lastly avoid using binding as much as possible. I have seen many
production bugs related to issues I outlined here.
Photo by 
enerGIS in action