Eric Normand on Functional Thinking – Conversations about Software Engineering [PODCAST]

I had the great honor to appear on Conversations about Software Engineering. We talked about functional programming and my book, Grokking Simplicity. https://cdn.podigee.com/media/podcast_998_case_conversations_about_software_engineering_episode_476020_eric_normand_on_functional_thinking.mp3

The post Eric Normand on Functional Thinking – Conversations about Software Engineering [PODCAST] appeared first on LispCast.

Permalink

Pimp my Clojure number crunching

Let's start right in the middle of the story, and play with cosine similarity. I'll tell you why in a few minutes.

I assume that most programmers have forgotten their math classes, so "cosine similarity" sounds somewhat grandiose. Quick skimming at Wikipedia might puzzle you even more, but, scrolling down to the definition, you can see that it's just a normalized dot product. I know, I know - now you might wonder what dot product really is. Please read my recent linear algebra Hello World article if that's the case.

A little warm up coding

We are programmers; it will all be clear after we code this.

Here are two vectors:

(def x (dv 1 2 3))

(def y (dv 30 20 10))

Dot product

Now, the dot product can be naively implemented as following.

(defn naive-dot [x y]
 (reduce + (map * x y)))
(naive-dot x y)
=> 100.0

It's much easier and faster to use the staple linear algebra function called dot:

(dot x y)
=> 100.0

Normalization

Next, the normalization part. That might seem grandioze too, but it is, again, a staple LA function nrm2 (the second norm of a vector). Not only that it's a staple function, but it's just a square root of the dot product of a vector with itself.

(defn naive-nrm2 [x]
 (sqrt (naive-dot x x)))
(naive-nrm2 x)
=> 3.7416573867739413

A better choice in higher level code is to use the optimized nrm2 function available in Neanderthal:

(nrm2 x)
(nrm2 y)

Cosine similarity

This is cosine similarity between vectors x and y:

(/ (dot x y) (nrm2 x) (nrm2 y))
=> 0.7142857142857142

OK, nice exercise, but not that interesting. There's no particularily useful insight there. I implemented everything naively, and then wrote my own cosine-similarity, which is, after all, the same thing as any library's cosine-similarity (it's probably faster but that's irrelevant here).

We now have a way to compute a measurement of similarity of two vectors (cosine similarity), which is a number between -1 (opposite), 0 (not similar at all), 1 (they match), or somewhere in between. In our case, x and y are somewhat similar (0.714) which does not tell us that much, but it can tell us that they are more similar that z if cosine similarity with z is, say, 0.5.

So what?

This was only a warm-up, and the insight follows after the intermezzo: we can compute cosine similarity of many vectors in a batch much faster than one by one. But why? And when? Well, if we would have only blindly used what the library is providing, we probably wouldn't even been asking these questions!

Now, the motivation for all this

A few weeks ago a Clojurist was asking about a weird result returned by a function in a Clojure/Java machine learning library. There's nothing unexpected there; bugs are caught in all software all the time, and the author of that (open source and free) library fixed the bug immediately. It's a good pretext to make a point, though, so let's make it :)

We often rely on software libraries. Layers of abstraction make our work possible. Something we stretch the tendency to rely on 3rd party abstractions too much, intentionally giving up control of the central part of our code. The irony is that many times the simplest thing is to implement the thing using the right lower layer, than to configure a function call of the higher layer.

In this case, the bug was that a cosine similarity funciton returned a value larger than 1. The user (correctly) wondered whether it's an impossible value for cosine similarity. Several people helped with theoretical knowledge and practical guesses where the bug might be. How such a simple thing such as cosine similarity could be the source of any fuss? Probably because most programmers rely on libraries so much that they don't build confidence even for the simplest things related to math.

That's why I like to not skip the step of implementing custom functions for central parts of the algorithms that I rely on. Sometimes the existing solutions serve well, and I use them in the final code, but by building at least a naive prototype I build up my understanding and confidence. I also found out that my solution is often much better than the one implemented in a library. If that seems unexpected, consider that machine learning libraries are frequently written by grad students on their path to discovery. It's a domain expert with poor programming skills. Or it might be a case of a good programmer who only barely understands the domain…

Is that it? NO :)

So, we implemented our naive cosine similarity (nothing special).

(defn cosine-similarity [x y]
  (/ (dot x y) (nrm2 x) (nrm2 y)))

We choose this, or the (hypothetical) library function with the same name, and proceed with looping, mapping, reducing and other Clojure-fu? NO! This would limit us to calling such function for each pair of vectors. What works well in number crunching is finding an operation that does the same for the whole bunch in one sweep!

But, how do we do that? If we hadn't explored the internals of cosine similarity, that'd be a mystery. But, since we had, we can recognize that matrix multiplication is a bunch of dot products. Explaining the details of this insight would digress too much; I cover this kind of knowledge in the Numerical Linear Algebra for Programmers book.

So, let's put these two vectors, x and y, in a matrix and call it xs.

(def xs (dge 2 3
             [[1 2 3]
              [30 20 10]]
             {:layout :row}))

When we multiply two matrices, we get a third matrix, and each entry in the result is the dot product of the respective row of the first matrix and column of the second. For example, entry (2,3) is the dot product of the second row of the first and the third column of the second matrix.

In our trivialized case, we have one matrix, xs, with two rows, and we don't have any other matrix. What now? Since we want all combinations of dot products of our vectors, we can simply matrix multiply xs by its transpose!

(mm xs (trans xs))
=>
#RealGEMatrix[double, mxn:2x2, layout:column, offset:0]
   ▥       ↓       ↓       ┓
   →      14.00  100.00
   →     100.00 1400.00
   ┗                       ┛

Do you recognize the familiar number 100.0? That's the dot product of x and y.

Notice that this matrix is symmetric, so we can optimize this by using a dedicated function optimized for matrix multiply with transpose, mmt.

(def x-dots (mmt xs))

Now we only need to normalize these. But how? There's no trick with matrix multiplication for nrm2. Or is there?

Remember that we also learned how nrm2 is computed: as the square root of the dot product of a vector with itself. The diagonal holds the dot products of x and x \((1,1)\), and y \((2,2)\)! We only need to square the diagonal. While we're at it, we might also invert it, so we can later deal with multiplication instead of division (but this is a minor detail). All these vector operations that we need are available in Neanderthal's vect-math namespace.

(def x-norms (inv! (sqrt! (copy (dia x-dots)))))

Now we have the dot products in matrix x-dots, and the norms in vector x-norms. Whoops… How do we combine that? The formula for cosine similarity uses each combination of norms for vector pairs, not single norms. No problem! We pull the rk function from our pocket. It does the outer product of two vectors, making a matrix that contains the product of each pair of their entries! Again, this is not some one-off trick, but a standard linear algebra stuff explained in the book.

(mul! (view-ge x-dots) (rk x-norms x-norms))
=>
#RealGEMatrix[double, mxn:2x2, layout:column, offset:0]
   ▥       ↓       ↓       ┓
   →       1.00    0.00
   →       0.71    1.00
   ┗                       ┛

BAM! Here it is! 0.71! Ones on the diagonal show that each vector is perfectly similar to itself :)

This might seem much fuss for nothing! Isn't the vector code simpler? Yes, but it only can compute one combination of two vectors. In this example, we've done the most trivial example with the same two vectors. But this code computes as many vectors as we give it! No need for loops or recursions!

Pimp my cosine similarity!

I've added some bookkeeping code. Clean up our memory and reuse matrices if possible for maximum speed gains.

Note the plural in the function name.

(defn cosine-similarities [a]
 (let-release [a-dots (mmt a)]
    (with-release [a-norms (inv! (sqrt! (copy (dia a-dots))))
                   ab-norms (rk a-norms a-norms)]
      (mul! (view-ge a-dots) ab-norms)
      a-dots)))

Here's a matrix with 4 vectors.

(def a (dge 4 3
            [[1 2 3]
             [-3 8 0]
             [30 20 10]
             [-7 1 -5]]
            {:layout :row}))

And, here are their cosine similarities.

#RealUploMatrix[double, type:sy, mxn:4x4, layout:column, offset:0]
   ▥       ↓       ↓       ↓       ↓       ┓
   →       1.00    *       *       *
   →       0.41    1.00    *       *
   →       0.71    0.22    1.00    *
   →      -0.62    0.39   -0.74    1.00
   ┗                                       ┛

See? All the combinations of the vectors with one single blow! And, if we have thousands of vectors it's orders of magnitude faster than calling the single cosine similarity function thousands of times!

Vectorization FTW!

Permalink

The REPL

The REPL

Automatic lambdas and Lillilput
View this email in your browser

The REPL

-main

  • A post from Arne Brasseur about some changes for how Lambda Island makes videos. Making changes like this can be tough, but it seems like this is going to be more sustainable.

Libraries & Books.

  • re-frame-flow helps you visualise a chain re-frame events.
  • Methodical is a replacement for Clojure’s multimethods. It adds some neat features from the Common Lisp Object System, like adding wrapper methods, and the ability to call the next most specific method (kinda like calling a “superclass” but against the multimethod hierarchy). 0.12.0 is out now.
  • recife is a library for running TLC/TLA+ models for model checking.

Foundations.

  • OpenJDK has a new project Lilliput. The goal is to explore reducing Java’s object header size (in Hotspot) from 128 bits to 64 bits or less. As you can imagine, if succesulf, this would have a substantial impact on Java’s memory footprint.

Tools.

  • patch for adding Clojure language support to Git’s diffs. You can add this yourself to your own Git config, but it’ll be nice to have it built-in.
  • syntactic differ for Scheme. Similar to the excellent but underused autochrome.

Recent Developments.

Learning

Misc.

Colin Fleming shared this to the Clojure subreddit - A strong commitment to backwards compatibility means keeping your mistakes

I’m Daniel Compton. I maintain public Maven repositories at Clojars, private ones at Deps, and help fund OSS Clojure projects (along with tons of generous members like LatacoraRoamPitchNubankCiscoJUXTMetosinSolitaAdgojiNextjournalFlexianaToyokumoGriffin, and [Parkside][parkside]) at Clojurists Together. If you’ve enjoyed reading this, tell your friends to sign up at therepl.net, or post a link in your company chatroom. If you’ve seen (or published) a blog post, library, or anything else Clojure/JVM related please reply to this to let me know about it.

If you’d like to support the work that I’m doing, consider signing up for a trial of Deps, a private, hosted, Maven Repository service that I run.

Thanks!

Twitter
Copyright © 2021 Daniel Compton, All rights reserved.


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by Mailchimp

Permalink

Senior Software Engineer (Clojure)

Senior Software Engineer (Clojure)

Compute Software | USA and Canada
remote
Startup that helps enterprises to run optimally on the public cloud.

About Us

With ever-growing workloads on the cloud, companies face significant challenges in managing productivity and spending, and maximizing impact to their businesses. Compute Software is addressing a huge market opportunity to help companies make good business decisions and run optimally on the public cloud. We're building a powerful platform to give customers the tools they need to gain transparency, optimize their usage, and execute change across their organizations.

We're a small, distributed team, currently spanning California to Quebec, and we offer early stage market-rate compensation (including equity), health and dental insurance, and 401K benefits. You'll be joining a venture capital-backed, distributed team with ambitious goals, and you will have the ability to make a direct and lasting impact on the team and product.

Your Role

Be one of the earliest employees and join our early-stage engineering team as a Senior Software Engineer. You will be essential in shaping the features and functionality of our SaaS platform, culture and processes.

You'll spend your day enveloped in Clojure. The backend is written in Clojure and driven by data in Datomic and InfluxDB. The frontend is written in ClojureScript using re-frame and communicates with the backend using Pathom. We deploy to AWS Fargate and Datomic Ions.

For product development, we follow Shape Up. We use Notion, Slack, and ClubHouse.

What will you do at Compute Software?

  • Write Clojure and ClojureScript.
  • Design, build, and deploy features and bug fixes across the entire stack.
  • Become an expert in all the nuances of the various cloud platforms like Amazon Web Services, Google Cloud, and Microsoft Azure.
  • Provide product feedback and evaluate trade-offs to impact product direction.
  • Debug production problems.

What We're Looking For Passion - you are excited about the large, high-growth cloud computing market and figuring out how to help customers, who are using cloud computing solutions today. You are excited by being one of the earliest employees and getting to work with a small team.

Engineering experience - you're a Clojure practitioner with 6+ years of professional experience. You know what it takes to create large-scale b2b software. You can create effective and simple solutions to problems quickly, and communicate your ideas clearly to your teammates.

Product-minded - you love building products and you care about the details of creating a great user experience. You have an interest in how users will use our platform and the impact we will have on them. You can balance your consideration of the product and user requirements with technical complexity and implementation details to make appropriate decisions when things are unclear.

Effective communication - you're great at communicating. If something is unclear you reach out and ask questions. You're comfortable owning, communicating and presenting information on specific projects or initiatives, both in writing and in person.

Organizational and project management - you are highly organized and able to self-manage projects in a fast-moving company. You are able to take high level goals and break them down into achievable steps.

Permalink

Clojure metadata

Conceptually metadata is an ability for the symbol or the collection to have some additional information for the Clojure compiler. It is implemented as a map data structure and is often used to convey the type information to the compiler, documentation and compilation warnings.

Permalink

Clojure Deref (June 18, 2021)

Welcome to the Clojure Deref! This is a periodic link/news roundup for the Clojure ecosystem. (@ClojureDeref RSS)

Highlights

  • HOPL IV (History of Programming Languages) at PLDI 2021 is happening on Monday and Tuesday and includes a talk from Rich Hickey about the History of Clojure paper. Registration is still available and the conference is online and features many other fine language developers!

  • The results are out from the JVM Ecosystem Report 2021 and Clojure continues to make a strong showing as one of the most popular JVM languages (other than Java), rising from 2.9% last year to 8.4% this year. Lots of other interesting tidbits in there as well.

Sponsorship Spotlight

Lately Christophe Grand and Baptiste Dupuch have been teasing their work on a new ClojureDart runtime with Flutter support. You can support their work on GitHub: cgrand dupuchba. Nubank (users of both Clojure and Flutter) are now supporting both!

Podcasts and videos

Blogs, discussions, tutorials

Libraries and Tools

Some interesting library updates and posts this week:

Fun and Games

  • Fidenza - Tyler Hobbs has a long history of doing interesting generative art with Clojure and he has published a rundown of his newest generative algorithm. Not explicitly Clojure but fascinating to read.

  • ClojureScript racing game - Ertuğrul Çetin published this game this week

Throwback Friday (I know, I’m doing it wrong)

In this recurring segment, we harken back to a talk from an older time to a favorite talk of yore. This week, we’re featuring:

In this lovely story from 2012, Carin Meier talks about monads through the lens of Clojure and Alice in Wonderland.

Permalink

Should you adopt Clojure at your company?

TL; DR; If you have already learnt it or have the time to do so, then yes

Photo by Matt Palmer on Unsplash

I have been writing Clojure professionally for the past 5 years, but I still, have a hard time recommending Clojure to friends and clients. My friend Anand, co-founder of Pabio aptly said that:

JavaScript for me is a language for making prototypes, and not a system that you want running after a decade from now.

Although I’m a Clojure fan-boy and advocate, the realist in me knows that Anand is right. Clojure has an adoption problem.

What is Clojure?

Clojure is a functional LISP that with a futuristic feedback loop. It’s hosted, has a strong following, and often reaches the first page of Hacker News. I’ll not go into the details of the language as many articles already exist about this topic. Clojure’s simplicity and expressiveness is a recurring theme in most descriptions of the language.

But this article is not about Clojure’s beauty. It’s about the challenges (and possible solutions) that stand in the way of mass adoption. It’s not about being a successful artist, it’s about being successful at the art bazaar.

What is the adoption problem?

Despite the widespread praise about the Clojure language, we cannot refute the fact that it’s hard to find Clojure developers. On the other hand, it’s also hard to find a Clojure job.

The problem turns into a chicken and egg paradox. There won’t be enough developers until there are enough jobs, and there won’t be enough jobs until there is a big talent pool.

A small talent pool leads to a small open-source footprint and prevents network effects. If few developers write Clojure, fewer new developers will be introduced to the language, amplifying the chicken and egg problem. This is countered to some extent because Clojure is hosted and can tap into vibrant eco-systems of JavaScript and Java.

Writing software is the easy part of running a software company

All software is a tool to solve a real-world problem. Companies and organizations exist to solve real-world problems. Until the 1950s, real-world problems were solved by war. Post the second world war, we shifted to data-oriented solutions.

As software developers, we concern ourselves with the issues about our domain. For a person who can code, every problem is an opportunity to make a SaaS company. But if we zoom out and look at some successful companies, we’d know that larger risks exist in the market.

Markets change and we need to react. There is a competition or lack of demand. There are legal issues and privacy issues. A bad culture could lead to people issues. And most of these are beyond our control. Looking at the full picture, we see that software is the most predictable and risk-free aspect of a software company. Since this blog’s audience is mainly software developers, I usually don’t talk beyond software. But this specific topic of basing your stack on Clojure requires me to address the other things that go into making good software.

All good software companies start with a strong grasp of the market and grow with a strong culture and the ability to react. If you have zero users, you will probably be solving technical problems. If you have hundreds or thousands of users (depending on your market) you’ll be solving sales and marketing problems.

Once you have crossed maybe a million users mark, you step into the realm of people and cultural problems. And post 10 million, you have to face political problems.

All languages are tools and all tools are good enough

From a zoomed-out perspective, all languages are tools. And all mainstream tools like Java, JavaScript, TypeScript, Ruby, etc are good enough. An experienced JS developer is as effective as an experienced Clojure developer at building software.

Although all languages are good enough, some of them offer special capabilities. Like C++ and Rust are close to the metal. Go, Elixir and Clojure have excellent concurrency primitives. Java, JS has the largest talent pools. Python has the support of the ML community. TypeScript has Microsoft.

These are broad generalizations. Most languages have overlapping strong points. Python has a large talent pool along with being the language of choice for data science.

Then how do you decide what tool to use? And when do you pick up something like Clojure? To answer these questions, we need to analyze the usefulness of tools from the perspective of a “software company”.

Properties of good tools

We have already ascertained that writing software is the easy part and risks lie outside our servers: in the market, in the culture, in the political scenario. That being the lens of our camera’s perspective, a good tool should have the following properties.

Make your process easy

A table-saw makes it easier to cut wood. Hasura GraphQL engine makes it easier to ship APIs. Postgres makes it easy to store data. Functional languages make it easy to handle side effects.

Be easy to comprehend

Complicated concepts do not survive the test of time, and are eventually replaced with simpler alternatives. It’s hard to find a counter-example because most tools that reach mainstream usage are easy to comprehend.

Extend like legos

Good tools are pluggable because an architect relies on multiple tools. REST APIs are a great concept. The SaaS community shifted radically from the server-rendered MVC pattern to REST APIs. You can argue that the shift was due to the explosion of clients (mobile, desktop, IoT), but it still serves the argument. A tool that can be modified and extended can serve more purposes.

Last for long, ideally forever

There is a cost associated with mastering a tool. And if the tool dies before you have reaped a significant return on investment, then you made a bad deal.

Get out of your way

A tool is designed to ease your process, not dictate it. It’s okay to have rules and limitations on what a tool can do, but it should never dictate the how.

Make you feel confident about the work produced

A tool is no good if you cannot be confident that your app will run. Tests, Types, Pure Functions, and Side-Effect free systems are examples of tools that provide this confidence.

Enable you to react to market

Ultimately, you survive because you adapt to the market’s condition. According to Rich Hickey, the single largest factor that decides how quickly you can react is the code you have right now. He calls it the “elephant” in the room. I believe the same, but Rich Hickey saying this adds a lot more credibility:

Simplicity is hard work. But, there’s a huge payoff. The person who has a genuinely simpler system — a system made out of genuinely simple parts, is going to be able to affect the greatest change with the least work. He’s going to kick your ass. He’s gonna spend more time simplifying things upfront and in the long haul he’s gonna wipe the plate with you because he’ll have that ability to change things when you’re struggling to push elephants around

Rich Hickey [https://www.infoq.com/presentations/Simple-Made-Easy/]

Clojure is an exceptional tool

Clojure is almost as pure and sophisticated as Haskell, and as easy as Python or JavaScript. The syntax might seem off, but the returns on learning the syntax are astronomical, because of the REPL.

It gets your way and lets you tap into rich library eco-systems of Java and JavaScript. Many functional languages suffer due to a lack of open-source network effects. Clojure bypasses the issue by feeding off network effects of other ecosystems.

And no praise is enough to describe the REPL. It accelerates development to an extent that most other languages can’t even imagine. I always wondered if there was a REPL for other languages. There are simpler versions for JavaScript like Quokka.js. u/didibus pointed out Clojure properties that enable the Clojure REPL on this Reddit thread.

It would be unfair to say that Clojure is the only exceptional tool so please consider this to be a personal opinion. Clojure is balanced. It makes hardcore mathematical concepts, commercially feasible.

Should you choose Clojure?

This is the trick question. There is no right answer but experiences and opinions to build on.

In my opinion, you should not learn Clojure at your own expense. If you are building a company and have no prior Clojure experience, then it would be a terrible choice. It will slow you down and will hinder you from focusing on the market.

Paul Lam’s (VP Engineering at Motiva AI) comment on this topic captures the essence of this idea:

Are you building a startup or are you churning out code? If you have time to consider these questions then you’re not selling enough. You need to stay focused on the business.
If you want to tinker, then do that on other people’s dime or save that for a hobby project.

If you already know Clojure, it becomes your super-power. It gets out of your way and helps you focus on the market. It may be hard to find Clojure talent, but it’s not hard to nurture it. Clojure might not be the easiest language to get started with, but senior developers can pick it just like any other language. Here’s what Reddit thinks about the Clojure hiring situation. From my personal experience, Clojure companies hire regardless of Clojure experience.

u/bostonou introduced Clojure at Vanderbilt university:

I helped introduce Clojure at Vanderbilt University working in genetics research. I then moved to a Clojure startup, did some Clojure consulting, and now do Clojure at CircleCI. Overall I’ve been doing Clojure professionally for ~8 years.

Adam Nielson, CTO of WeFarm said:

We’ve used Clojure at the heart of our stack for the last five years and I consider it one of our superpowers.
It was less a leap of faith but a decision based on exceptionally positive previous experiences. I first trialed and adopted Clojure at a health tech company back in 2013 and then built a marketplace startup for hobbyists and collectors with it in 2014. Both times were productive and fast time to market. The composability of lisp and FP makes rapid reliable development not only achievable but the norm.

How tough is it to hire Clojure talent?

Adam from WeFarm also has a unique perspective on hiring Clojure developers:

Over the years I’ve also discovered that hiring for Clojure is also different in a positive way. In more mainstream languages I’d get a huge stack of CVs. Amongst them would be great and not-so-great candidates and it’d be a lot of effort filtering the wheat from the chaff. With Clojure roles, the pile of CVs is a fraction the size but you can guarantee 99% of the candidates have a curious and investigative character that enjoys learning new things. Exactly the characteristics I look for in candidates!

The “curious investigative character” of people who write Clojure has held in my personal experience. No one would take the trouble of learning a language like Clojure (or Haskell) unless they deeply care about their work. This property will hold until Clojure jobs become as plenty as JS jobs.

At my company Status, we hire regardless of Clojure experience. Since we are an open-source company, we pick up simple issues that we believe any engineer should be able to solve regardless of the language. All prospective hires are expected to pick one issue and raise a PR as a coding challenge (candidates are compensated for their work). Status is hiring for its React Native based app written in ClojureScript if you are interested!

Anson MacKeracher, CTO of LegalMate says:

I chose to use Clojure due to the incredible caliber of the average Clojure developer, because it’s virtually perfect for web applications and data processing, and because developer productivity is much higher than in other languages I’ve used (Python, Perl, Ruby, JS).

If you are a part of a big company, training talent internally too is a viable option. I have seen a company take over a 5-year-old codebase and develop a 20 person team around it. The entire transition took about 8 months.

Conclusion

I’m a Clojure fan, but more than that I’m a problem solver. And Clojure is not the right tool for everything. If your problem involves handling a ton of data and building APIs and UIs, Clojure is a great system. If you want to quickly build a team around your project then probably not. Clojure is a super-power that enables you to better react to changes outside the system.

Recommending Clojure was hard because of a lack of talent and resources. But I feel this is changing. To contribute to the change, I built a course about how to build React applications with Clojure. I also collected a list of free resources that you can use to jump-start your Clojure journey.

Originally published at https://krimlabs.com on June 17, 2021.

https://krimlabs.com/courses/tinycanva-clojure-for-react-developers

You might also like:

Permalink

Lambda Island is Changing

Last month marked the five year anniversary of Lambda Island. Five years since I quit my job, built a video platform, and figured out how to put together tutorial videos. It’s been quite a ride. All this time I’ve been fortunate to be part of the Clojure community, to watch it grow and evolve, and to help individuals and companies to navigate these waters.

I learned some hard lessons along the way. Like how hard it is to bootstrap an educational content business catering to a niche audience, or how lonely and stressful it can be in business to go it alone.

And with these lessons came changes. I started doing more consulting work again. To some extent it was a necessity, but it also provided me with an opportunity to get involved with many Clojure projects out there in the wild. To work with talented individuals, to learn what amazing things people were doing, and to help shape these companies, products, and the greater narrative around Clojure as a technology and community.

Out of this work grew many open source libraries and tools, like Kaocha, Regal, Glögi and Corgi, which became part of the Lambda Island brand, and indispensible items in people’s toolbelts.

But I was still going it alone, and I didn’t like that, so in 2019, Gaiwan was born, a bona fide Clojure shop, and slowly but surely I started putting together a team. Fast forward to today. We are now a team of 6, spread around the globe, working with amazing clients like Nextjournal, Eleven, Pitch, IT Revolution, and A-Knowledge.

We’ve been busy! And not just with client work. We’ve continued to grow and improve our open source offerings. We’re still running ClojureVerse and Clojurians-log for the community. Our blog has gotten regular updates. We’ve reached a lot of people on YouTube during the Advent of Code, and we even organized a conference, Heart of Clojure.

What we haven’t been doing is posting new video content to Lambda Island. I could say that with everything else happening it has been nearly impossible for me to carve out the time to work on new episodes. That’s not wrong, but the real issue is that I burned out on making them. I would easily spend a full week on a 15 minute episode, slaving over the script, agonizing over the details, and meticulously editing the final result. Then it would go live and…crickets. Freebies would maybe get a few comments on Reddit and a hundred views. Paywalled episodes would get a few dozen. Neither would really bring in new revenue, and I’d be scrambling afterwards to catch up with client commitments.

In hindsight it was a pretty toxic feedback loop. I suffered under my own perfectionism, and with that continued to set the bar high for each subsequent episode.

But this isn’t a story of doom and gloom. Instead it’s one of necessary change. We’re a team now, and everyone on the team is excited to help plot the future of Lambda Island. We don’t know exactly where we’ll end up, but we know the general direction we’re setting out in.

One thing that is clear is that Lambda Island is going to start looking and sounding a bit different. Before, Lambda Island was very much my voice, literally and figuratively. One person’s style of writing, level of experience, and view on technology. Going forward we are going to be a multitude of voices. The first place you are going to notice this is on this blog, where both Ariel and Alys have already started publishing some articles of their own.

Meanwhile Felipe—with help from Ariel—has been hard at work behind the scenes to give the site a major upgrade, the biggest overhaul the site has seen since we launched in 2016. The result should look and feel much more fresh and modern, be faster, more responsive and accessible, and allow for better attribution of authors.

Front page of the newly redesigned Lambda Island website

We also know that we want to put more effort into fostering a vibrant community around Lambda Island. One that values curiosity, creativity, and exploration; and where beginners can feel welcome and supported. We set up a Lambda Island community on Discord and started inviting a few dozen individuals while we set things up and get a feel for the place. We’ll soon be sending out more invites, starting with Lambda Island subscribers and to our open source supporters on Open Collective. The whole Gaiwan team hangs out there as well, and so going forward this will be the best way to get in touch with us and get involved in conversations. Don’t be a stranger!

As for video content… we don’t know yet. The most likely scenario is that you are going to start seeing more uncut and unscripted content, like the Advent of Code videos, instead of the highly condensed and polished episodes from before. Unfortunately Pascal’s shorter letter applies here as well… I would have made a shorter video, but I did not have the time.

It does seem a lot of people enjoy these uncut videos, bloopers included. You get to really see how someone works in realtime, how they use their tools and think through problems. For a language with a highly idiosyncratic and interactive workflow like Clojure this is invaluable.

Lambda Island is changing. Change can be a little uncomfortable, but it’s a constant of life. With this post I want to pave the way for the rest of the team, so that they can be comfortable making their mark. We’ll talk in more details about some of the things we are up to in upcoming posts. Meanwhile if you’d like to reach out then you can contact me on Discord as Arne#3086. Don’t be a stranger!

Permalink

enerGIS, renewable resource planning at your fingertips.

Long time no see, my friends. We have been busy building stuff at Magnet and in this hot day, hiding from the scorching sun, I would like to tell you about a very special product that we have created in this hectic year; a year where we are all navigating each in our precarious way through the changes this post-pandemic (fingers cross!) world has brought to us.

enerGIS is how we called this business venture and it’s led by Matthew Barlow, a brilliant mind and better guy. enerGIS is a web application that assists decision makers (governments, green consultancies, energy producers,…) understand solar and wind resource potential across a territory. It analyses, among other things, parameters such as distance to infrastructure, load centre proximity, land use classification and more — such as topography, bathymetry and environmental restrictions.

According to researchers from World Resources Institutes land planning will play a pivotal role for the successful deployment of any renewable energy project. We have several country datasets already available and we are adding new ones in the coming days, with the mission to map the whole world, contributing to a greener and more sustainable future.

The International Energy Agency (IAE) has designed a pathway to reach the goal of net-zero CO2 emissions in the energy sector by 2050. As part of this challenge, by 2030, the solar and wind energy capacity added to the system will have to be three times the capacity installed in 2020. This is certainly challenging, but it’s worth noting that many developing countries are still in the infancy stage of leveraging their renewable energy potential.

And here is where enerGIS comes to play: a simple web application that can be suitable for decision makers who want to analyse the potential of a country or territory, an online tool that takes the information out of the technical back rooms and brings it out in a simple, clear way.

I would not go into the technicalities of the project, colleagues will comment on these in later posts (of course it has been built in Clojure(script)!), or explain the features that the tool brings (you can find out more in here), I just wanted to announce that the product is available in a SaaS model (in private beta right now) as well as client-side deployment for Enterprise versions, and we plan to make it available to the public during this summer.

A picture is worth more than a thousand words, so here you have a sneak peek of enerGIS:

enerGIS in action

enerGIS, renewable resource planning at your fingertips. was originally published in magnet.coop on Medium, where people are continuing the conversation by highlighting and responding to this story.

Permalink

Fun of clojure - wrap some code around data

Recently FOMO got to me and I jumped on crypto speculation game. You gots to do bunch of accounting/counting/reconciling/tracking in it. Sounds boring, but clojure made it fun. And that’s what I want to share here.

Permalink

The REPL

The REPL

View this email in your browser

The REPL

Notes.

Lots going on in Clojure this week!

-main

  • :clojureD was recently held online, the talks should be up on YouTubesoonish.
  • Nubank raised a staggering $750M last week, on top of a $400M round in January. Nubank plans to use the funding to launch in Mexico and Colombia, launch new products and services, and hire more employees. That still seems like a lot of money, I’m excited to see if there’s anything else they plan to do with it.
  • The Stack Overflow Developer Survey is on right now. I’d encourage all Clojure developers to fill it out to help the Clojure community be represented.
  • Chris Nuernberger gave a talk on High Performance Data for the London Clojurians Meetup. Chris (along with others in the scicloj community) has been pushing Clojure a long way in the data analysis/data processing space.
  • Alex Miller has started Clojure Deref, a link/news roundup for the Clojure ecosystem. If you like this, you should check out Deref too.
  • Jordan Miller continues her podcast Lost in Lambduhhs with Alex Miller.
  • Nathan Marz gives us a tour through their 250k line Clojure codebase. That’s a pretty large Clojure codebase, and it sounds especially dense with 400 custom macros (Clojure’s source has 127). However, they’re building a programming language on top of Clojure so it’s perhaps not too surprising. Nathan Marz answered a few followup questions on the Clojure mailing list, including a few more hints on what they’re doing with continuations. I’m looking forward to seeing more.

Libraries & Books.

  • mprop is a nice extension to test.check with two useful extensions: localising test.check failures to the assertion, not just the test; and scaling the number of tests based on environment so you can run 10x more in CI than in local dev.

People are worried about Types. ?

  • Jacek Schae talked to Tommi Reiman about Malli
  • snoop adds some syntactic sugar to define Malli function schemas.

Foundations.

  • Crux has a Bibliography of books and papers that inspired Crux’s philosophy and architecture.

Tools.

  • Tips and tricks on how to use Clojure with GraalVM from Lee Read
  • Michiel Borkent continues churning out useful Clojure tools This time it’s scittle, a lightweight tool for running ClojureScript directly in the browser with no precompilation. View source on the linked page to see what it looks like, very cool!
  • An alpha for Expectations 2.0.0 is out, if you use Expectations, now would be a good time to give your feedback.
  • Christophe Grand and Baptiste Dupuch continue to do good work on extending Clojure to Dart
  • list of OSS datalog databases (from the Crux team). The comparison looks fair, no one vendor gets all ✔️’s.

Learning.

  • An interactive guide on how to build a tetris game in ClojureScript from Shaun Lebron. Even if you don’t want to build a game, it’s worth reading, just to see an example of how you can make code explanations much more tangible.

Misc.

I’ve recently started stretching to end my day. It’s a good wind down routine to relax your muscles. There are a bunch of videos on YouTube to pick from, depending on how much time you have and how flexible you are to start with.


I’m Daniel Compton. I maintain public Maven repositories at Clojars, private ones at Deps, and help fund OSS Clojure projects (along with tons of generous members like LatacoraRoamPitchNubankCiscoJUXTMetosinSolitaAdgojiNextjournalFlexianaToyokumo, and Griffin) at Clojurists Together. If you’ve enjoyed reading this, tell your friends to sign up at therepl.net, or post a link in your company chatroom. If you’ve seen (or published) a blog post, library, or anything else Clojure/JVM related please reply to this to let me know about it.

If you’d like to support the work that I’m doing, consider signing up for a trial of Deps, a private, hosted, Maven Repository service that I run.

Thanks!

Twitter
Copyright © 2021 Daniel Compton, All rights reserved.


Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by Mailchimp

Permalink

Better performance with Java arrays in Clojure

I was working on re-implementing the Stochastic gradient descent algorithm in Clojure based on the blog post How to Implement Linear Regression From Scratch in Python. Because I wasn't happy about the performance when the dataset is in the order of thousands rows of thousands features, so I started looking into ways to go lower-level to use the Java arrays and found this amazing article on Clojure Goes Fast: Java arrays and unchecked math. (It's a great article that goes into details of Clojure type-hinting and other tips. Highly recommended!)

Following the instructions in the Clojure Goes Fast article, I'm quite amazed by the performance boost. Below is a comparison of the same predict function: one expects Clojure vector and the other Java arrays. The only thing novel thing that didn't cover in the Clojure Goes Fast article is the usage of areduce. Instead of the explicit loop-recur shown in the article, I stumbled upon this handy clojure.core/areduce form and used it to better convey the implementation of the predict function.

The results are quite promising.

(require '[criterium.core :as c])

(defn predict-v [xs [bias & coefs]]
  (reduce + bias (map * xs coefs)))

(defn predict-a [^doubles xs ^doubles coefs]
  (areduce xs idx ret (aget coefs 0)
           (+ ret (* (aget xs idx) (aget coefs (inc idx))))))

(let [xs      (range 5000)
      coefs   (range 5001)
      xs-v    (into [] xs)
      coefs-v (into [] coefs)
      xs-a    (into-array Double/TYPE xs)
      coefs-a (into-array Double/TYPE coefs)]
  (c/quick-bench (predict-v xs-v coefs-v))
  (c/quick-bench (predict-a xs-a coefs-a)))

The *out*:

Evaluation count : 834 in 6 samples of 139 calls.
             Execution time mean : 736.026481 µs
    Execution time std-deviation : 26.489882 µs
   Execution time lower quantile : 715.074957 µs ( 2.5%)
   Execution time upper quantile : 777.624246 µs (97.5%)
                   Overhead used : 6.224349 ns
Evaluation count : 106956 in 6 samples of 17826 calls.
             Execution time mean : 5.865484 µs
    Execution time std-deviation : 221.419267 ns
   Execution time lower quantile : 5.580842 µs ( 2.5%)
   Execution time upper quantile : 6.125437 µs (97.5%)
                   Overhead used : 6.224349 ns

It's about 736 µs versus 5.87 µs. That's about a 125X speedup by using Java array.

It's really nice that I don't have to sacrifice the functional programming constructs (like map1 and reduce) because I'm working on a different data structure. Under the hood, areduce is a macro that does the loop-recur for you. Here's the source code of clojure.core/areduce:

(defmacro areduce
  "Reduces an expression across an array a, using an index named idx,
  and return value named ret, initialized to init, setting ret to the
  evaluation of expr at each step, returning ret."
  {:added "1.0"}
  [a idx ret init expr]
  `(let [a# ~a l# (alength a#)]
     (loop  [~idx 0 ~ret ~init]
       (if (< ~idx l#)
         (recur (unchecked-inc-int ~idx) ~expr)
         ~ret))))

Using this macro has an advantage over using the loop-recur: you don't need to explicitly call the unchecked-inc-int to get the full benefit of speeding up looping on the array a.

Reference Links

Footnotes

1 Besides the clojure.core/areduce shown in this article, there's also the clojure.core/amap.

Permalink

Let and Binding macros in Clojure

Clojure has number special forms. Among them are let and binding that are implemented as macros. For new comers to Clojure, above can be confusing to understand, which I hope to address in this blog post. If you have read "Programming Clojure" book, you might be thinking that it is easy to use them correctly considering following straightforward example:

(def x 7) ;; bind the var x to the value 7
(defn print-x [x]
  (println x))

(let [x 9]
  (print-x)) ;; prints 7

(binding [x 9]
  (print-x)) ;; prints 9

Why? Because let creates a lexical scoped vars, which is local to the code block or functions where it is initially defined while def creates a global var that is visible to all functions under the namespace that you are in. Therefore, in the example above, let just created a new "x" with the value of 9. But it is not local to the function "print-x". That is to say, "print-x" have no knowledge of the local scoped binding that is created by let. binding macro instead creates a new binding for the already existed var, "x" with the value of 9. Since "print-x" is evaluated within the binding form, the newly bound value is visible to any chained calls within the form. All nice and easy. However, it is not that obvious to understand and use them correctly. Look at following example:

(def x 7)
(def y 9)

(let [x 5 y x] y) ;; returns 5
(binding [x 5 y x] y) ;; returns 7

let example is just fine. It created a new var "y" with the value of 5 bound to it. But what is going on with the binding example? If we dig little deeper to Clojure API doc, we learn that the new bindings in binding are made in parallel, not sequential as it is in let. Therefore, existing var "y" gets bound to the root binding of "x" not the overshadowed value of "x", which is the case in the let example. OK, so far so good. Let us look at another example:

(defn print-x [& args]
  (println "print-x prints: " args))
(defn print-y [& args]
  (println "print-y prints: " args))
(let [print-x print-y] (print-x 123)) ;; print-y prints: (1 2 3)
(binding [print-x print-y] (print-x 123)) ;; print-y prints: (1 2 3)
(binding [print-x print-y] #(print-x 123)) ;; prints
					     ;; cryptic "#<user$eval__28$fn__30
					     ;; user$eval__28$fn__30@bdb503>"
((binding [print-x print-y] #(print-x 123))) ;; print-x prints: (1 2 3)

All the lines but the last two are pretty straight forward. But how about the last line? Shouldn't it call "print-y" function? Not really. It is because the anonymous function is evaluated outside of the binding form. The "#<user$eval_28$fn_30 user$eval_28$fn_30@bdb503>" is an indication that anonymous function did not get evaluated. The extra parens evals above form. Ok, now, how do you make sure it evaluates within the binding form? Just enclose the anonymous function within parenthesis. What do we learn from all of the above? Here are the summary that helps one use let and binding correctly:

  • Use binding only if you are trying to alter a behavior dynamically for special-vars that is already defined. Enclose your special vars within * as recommended by idiomatic clojure to make your intent clear.
  • Use let for locally scoped vars and avoid shadowing already defined vars.
  • Remember let binding is sequential and binding binding is done in parallel.
  • Pay close attention to the scope of the binding form. Any expression that is not evaluated inside the form, will not see the binding. That is to say, binding in binding form is thread local starting from the binding form until it ends including any expression that is chained within the form, which is what the thread-local are all about.
  • Lastly avoid using binding as much as possible. I have seen many production bugs related to issues I outlined here.

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.