Blockchain Without the Crypto

Permalink

The Humble Programmer

We read from and comment on Edsger Dijkstra's 1972 Turing Award Lecture called The Humble Programmer. Is the problem with programming that we don't recognize our own limitations? We'll explore that and more.

The post The Humble Programmer appeared first on LispCast.

Permalink

Senior Data Engineer (Boston/Remote) at Flying Machine Studios

Permalink

Meta Reduce 2021.1: Summer Time

It’s been quite a while since my last “Meta Reduce” post. I’m happy to report that since writing it in April, I got vaccinated against COVID-19, and the cases in Bulgaria have dropped down significantly. One can never know what life has in store for us next, but the summer has been pretty pleasant and almost normal so far.

I finally did a bit of international travel for the first time since the pandemic started:

• In June I’ve spend a couple of weeks in Ukraine, visiting the family of my wife.
• In early July I finally managed to go to Venice, after a couple of failed attempts (the second one was in March 2020).
• In mid July I spent some quality time in Greece on the islands of Zakynthos and Santorini. If you’re into peaceful vacations, I can’t recommend the northern part of Zakynthos highly enough!

Clearly I really missed traveling, as I think I overdid it in the past couple of months. Now it’s time for a quiet month at home. I love Sofia in August, as it’s almost totally empty and very peaceful. I also hate traveling to the seaside during the high season. I guess I don’t have a lot of fondness for big groups of humans. Go figure!

There’s not much I can report on the OSS front. I took a 1 month vacation from work in mid-July and I’ve been trying to unwind a bit and stay away from programming for a while. Things are pretty quiet on the RuboCop and CIDER fronts. UNIX socket support for nREPL was finally merged a couple of days ago and I expect that relatively soon we’ll release nREPL 0.9 (e.g. early September).

We’ve also made some headway with the new library enrich-classpath, that makes it easy to download the sources and the javadoc for Java libraries. That’s extremely handy for CIDER (Orchard) users, as it makes it possible to navigate to the sources and javadoc of Java libraries. That has always been possible in CIDER, but the setup process was admittedly somewhat complex (you had to meddle with the classpath manually) and enrich-classpath makes this simple. Kudos to vemv (I don’t know his real name) for his work on the new library, and his recent efforts to improve cider-nrepl and refactor-nrepl!

Another cool recent development is the launch of nrepl-cljs-sci, created by Kimmo Koskinen. People have wanted an nREPL server for Node.js for a very long time, and now they have one. The project is still in its infancy and will require a lot of polish and some documentation, but it’s off to a very promising start.

I’ve used the extra time on my hands during my vacation to do some reading (I’m currently re-reading “A Wizard of Earthsea”), play StarCraft 2 (I totally suck there and random kids wipe the floor with me), watch “Better Call Saul”, migrate my personal email from Gmail to Fastmail, and provide technical support for my parents (that requires a lot of time!).

I also plan to do a bit of extra blogging in the next couple of weeks, but with all the heat here, and my excessive laziness, there’s no guarantee that this is going to happen. At least I finally finished the article I started to write last summer about the brief return to Linux. I’ve got plenty of ideas for articles, though, and I’m optimistic that at least 20% of them are going to become actual articles eventually. Of course, there’s always the possibility that I’ll actually learn how to play StarCraft II, become a pro gamer and put programming behind me.¹

That’s all I have for you today. Short and sweet. Enjoy the summer (winter)! May it be as normal and delightful as possible! Keep chilling!

Permalink

CIDER is Now on Discord

Over the past few months I’ve been trying to find better ways to support the community around CIDER (and my other bigger OSS projects), that’s why I was more focused on documentation and exploring new communication channels, instead of new features and fixes. Over the years I’ve tried a lot of different support channels with varying degrees of success:

• Mailing List (Google Groups) - never gained any traction. I’ve noticed the Clojure mailing list hasn’t been that active in recent years as well, so I think mailing lists are losing popularity in general. A pity, if you ask me. At any rate - given that no one uses the mailing list I definitely considering to shut it down completely.
• Gitter chat - never gained any traction. Gitter was somewhat popular a few years ago, but that’s no longer the case.
• GitHub Discussions - it’s still new, but there’s not much traction there.
• StackOverflow - never gained much traction either, although it’s certainly more used for CIDER than any of the above options.
• Slack (Clojurians) - the #cider, #emacs and #nrepl channels there are quite active. CIDER’s channel alone has over 1700 users, which probably makes it one of most popular channels on Clojurians. Seems that Slack was a big success for Clojure in general, and for CIDER in particular.

A lot of support also happens in GitHub issues, but I consider this accidental rather than intended and now I can easily convert all issues that should have been discussions into discussions.

So, it seems that Slack is the solution for support/discussions/etc, right? It’s big, it’s active, so it must be great, right? Unfortunately, I don’t think that’s exactly the case. Sure, it has a lot of users, but it’s really painful for me to use Slack because of several things:

• The logs disappear fairly quickly on the free plans that the Clojurians Slack is using. Often by the time I see something, all the context for it has disappeared. I know the logs are stored somewhere else, but I don’t really want to have to dig for them. I prefer them in chat.
• There’s not enough granularity in the conversations as they happen in just a couple of channels and support requests, feature requests, etc quickly get intertwined. I could have created more channels like cider-support, cider-beginners, etc, but experience with Slack has taught me that most people probably won’t bother to use them (unless constantly prodded by someone). And Slack’s threads are notoriously easy to miss. In our company we had even forbidden them for a while, as so many people hated how they were implemented.
• Slack has been betting progressing more enterprise and less hackerish every year. I still remember it when it was hip, simple and cool. Now it’s something very different.
• It uses a ton of RAM.

All this means that I don’t really enjoy using Slack much. Frankly, the only reason that I still frequent the Clojurians Slack is that I’m using Slack for work anyways. I’ve been pondering about alternatives of Slack for a while now, but I couldn’t find something compelling. Until I did. At the end of last year I stumbled upon Discord, the famous chat app for gamers, and I was shocked to realize that it’s a pretty general purpose chat app, that fits my needs way better than Slack:

• It’s simpler (less features, less hassle). It actually reminds me of the early days of Slack in some ways.
• It uses way less resources.
• The history doesn’t disappear on the free plan. That’s big!
• I can run my own Discord “server” where I get to define the channel structure and how channels can be used. Probably not very important for you, but quite important for me.
• It does threads better.

I know that some people don’t like Discord for various reasons (mostly related to some privacy fiasco in the past), but from my perspective it’s better than Slack in every way (at least for my CIDER/OSS projects use-case).

CIDER’s Discord server has been up for a few months and has the following channel structure:

• #general
• #support
• #ideas
• #nrepl
• #orchard
• #clojure-mode
• #clj-refactor
• #inf-clojure

A bunch of CIDER-related channels (the first 3), plus a few extra channels for my CIDER-related projects. Pretty simple, and super focused, at least from my perspective.

The Discord server is not particularly active yet (it has only a 110 users), but I haven’t advertised it much, so I’m optimistic about the future. My plan is to gradually reduce my presence in Slack and focus entirely on Discord. It’d be cool if more people adopted Discord (not just for CIDER, there’s a pretty big Clojure community on Discord in general), but I totally understand everyone who’s happy with Slack and wants to stay there. I have a feeling that part of its popularity is coming from the fact that so many companies are using Slack today as the official company chat, so adding one extra workspace there is not much hassle for many people. Given the size of CIDER’s Slack community, I think it’s going to continue being helpful even without me there. You can join CIDER’s Discord server here and decide for yourself if it’s better than Slack.

That’s all I have for you today. Keep chatting!

Permalink

Why we killed elixir

First this is my opinion as CTO, I don't care about languages, I hate them all.

Here is my hatelist of languages I know enough to say I hate them, from most hated to least hated:

# most hated
java perl javascript
elixir ruby
python cpp objc clojure
go dart
c
# least hated

All those languages have something to offer, when you decide to use one, you better know how it will bite you in the butt. Most overpromise and underdeliver, their philosophies work great on paper, but poorly in practice. By 'in practice' I mean a diverse team of people working on a non trivially sized project with different experiences, preferences and backgrounds.

When REKKI started the devs thought they are building a chat app, instead of ordering app, and they chose elixir, with phoenix to be able to trivially do things like 'someone is typing' in the chat. However what they didn't realize was that there is no way two chefs are in the app in the same time. You can read more about that here: https://dev.to/rekki/work-in-the-kitchen-4ifm

We moved from elixir to go around 1 year ago, it was pretty smooth, we just made few go services and moved endpoints that we needed. We made a super simple k8s deployment infra make build push deploy and you are good to go.

This enabled all frontend devs to write go, learn more sql, make simple endpoints for themselves, and change something if they need. All of a sudden each frontend dev became a full stack dev.

But elixir has so many powerful concepts, the genserver, hot code reloading, concurrency, its functional, phoenix is pretty popular and has great community, so why would we move away from it?

Let It Fail

First the "let it fail" thing, look, in our company things end up in a master database, for 99.9% of the things, you open a transaction, write your stuff and close it.

GenServer is great and all, but honestly for us its completely useless, it requires way too much discipline to not create inconsistencies in the data model, so if you do want to have things fail you have to actually handle the failure of a weak data model.

Lets look at this example:

User types "hello" message, this ends up in GenServer that sends it in rabbitmq, RMQ is great piece of software, if somethings goes in it is very unlikely it will disappear, except when there is no memory in one of the nodes because of a bug in another queue that the consumer was not consuming, so now lets examine this "let it fail" thing.

What exactly can fail?

Nothing really.. what about we just write the message in the database and return
ok or error to the user?

We had so many outages because the data model has to be very relaxed in order to support the "let it fail" thing, and so often we ended up with partial data in the database..

Checkout https://dev.to/rekki/mutation-is-life-boring-technology-11h0 for example

hot code reloading

For me this is complete myth, to build a robust hot code reloading deployment pipeline requires enormous amount of effort. To properly handoff and transform state properly..

Its ok if you want to replace one module or so, or for some prod debugging.

Performance

Runtime performance wise it is slow, about 2-3 times slower than go, and 4-5 than c, which is OK for most things, unless you want to rank a million items with some semi complicated formula, and at point you will have to scatter gather.

Compile time.. omg it is horrible, like early rust horrible.

Type Safety

We had to have huge amount of useless tests just to avoid having typos.. typos I repeat. Its like perl without strict. Running the linter is slower than flow. Of course we use typespec and etc, it just has the same problem as flow, things become any fast, and any spreads like a virus.

And worse you cant delete anything, first the tests will fail, and then even when you fix them you have no confidence your changes are good. Of course typesafe code does not guarantee things are ok, but I have seen elixir bugs that are just pathetic for year 2020.

Reading and Writing code

This is personal now, some people like it some dont, I wont make strong argument, for me reading clojure is 10x faster than reading elixir, but I also used clojure more.

The problem is it has steep learning curve, you cant get a js dev and ask him to change elixir code.

Scale

Do you really think the issues with our scale will be bound to elixir? Most companies will be fine by using bash+netcat webservers.

This is a joke obviously nobody should do that.

while true; do 
  echo -ne "HTTP/1.0 200 OK\r\nContent-Length: 5\r\n\r\nhello" | nc -l -p 8080
done

Concurrency

What concurrency? Most languages are ok with that, elixir is no better.

Finding developers

IDK, now its hard to find any kind of developers, but choosing niche language makes it even harder. Also people join just because of the tech, which is not good.

So since we wanted to unblock the frontend, and simplify the action-at-a-distance architecture, I had few reasonable choices, go and java amongst the top on the list, and since it is just easier to write ok-ish code in go than in java, I chose go.

So now we are a go shop, we have 20-30 services, each dev can make their own service and deploy it.

Things are pretty nice. I can say, after 1 year, it was a huge enabler and multiplier of people, I don't think we would've done half the things we did have we not switched, and the cost of switching was pretty small.

Go has its baggage as well.. but at least we can delete it.

Permalink

Development Diary #6

Permalink

Clojure Deref (July 30, 2021)

Permalink

Working at Multis - Use Clojure to Build the Best Crypto Wallet

Multis (YC19) is a French startup, founded in 2019, building a best-in-class business wallet for crypto-native companies who want to professionalize their treasury management in digital and traditional currencies.At Multis, they believe that digital currency will birth an open financial system and support billions of entrepreneurs worldwide through instant payment networks and inclusive financial services. Their goal is to be businesses' financial OS and provide them with services and tools they'll need along their way.ProductAfter going through a shift in direction, it seems like this past year has been a defining one for Multis, who after a thorough research process, have starte...

Tags: Ethereum, Clojurescript, Blockchain, Clojure, Crypto, Tech Startups, Bitcoin, Defi

Permalink

Speed up your test builds

How to speed up your build using Kaocha and CircleCI Tag: Clojure Visit https://juxt.pro for more.

Permalink

It’s hard to get job offers at tech companies right out of school. What's going on?

I used to work in engineering and data science at a tech recruiting company.

We had a hard time placing new grads.

In this article I want to describe what tech hiring is like. If you’re just starting your career I hope this article can help you come up with a strategy to stand out from the competition and get the job you want.

It's hard for new grads to get their first job

According to data I've seen, candidates who get a lot of job offers have a mix of experience (preferably from a famous company or school), the right skillset and the right salary expectations. In that order.

Experience is particularly important.

The data I've seen suggests that there's a 3 year career-start dead zone -- you may be an amazing engineer but during those first years you'll have to knock on a lot of doors to get your interviews.

In an industry "short on talent" where companies happily pay head-hunting firms $20,000 for introductions, why do new engineers struggle to even set up brief meetings?

A lot of new engineers are talented, passionate and filled with potential. They're less expensive and fairly abundant.

You'd think companies would be taking advantage of this talent pool but many of them aren't.

What's going on?

I believe that companies are being careful. Making ANY hire is expensive. A bad hire can be catastrophic.

Let me describe what the interview process is like from the employer side.

How does interviewing work inside companies?

In most companies finding candidates is the job of the HR / recruiting department or external recruiters.

Companies have limited resources so out of the 12 applications they get for a position they'll only likely screen a couple.

A basic code screen requires ~2 hours from an engineer whose salary after taxes and benefits runs about $100 / hr. Time spent interviewing is time NOT spent coding. When you factor in time for context switching and getting lunch an interview takes about half of an engineers' day.

Recruiters usually don't have engineering backgrounds themselves but they're the ones who take the first look at your resume. They tend to work by keying in on keywords.

Because of this a resume focused on 'Rust', or 'Clojure' skills will get weeded out. While an engineer looking at this resume would see "rockstar coder with an interest in functional languages", a recruiter only sees "not a node engineer".

A company usually spends one to two full days interviewing a candidate.

The process is elaborate because the stakes are high. While rejecting a candidate is painful, hiring the wrong candidate is disastrous.

Hiring the wrong person is really bad

The hiring process itself is expensive but the real costs come after the hire.

Closing costs

The company spends a pile of money before the new hire even walks in the door.

If a recruiting firm sourced the candidate the company has to pay a finders fee - typically ~20000$ (15% - 20% of a first years’ salary).

Then there are all the costs associated with hiring. Payroll, IT, etc...

Ramp-up

Like most jobs, it takes a while for a new employee to become valuable. With engineers I've it takes about 3 months.

During this time the engineer needs a lot of hand holding.

An experienced engineer will need help with the quirks of a company's system, but a new engineer will need help with system setup and basics.

Code quality

A newer engineer also requires more babysitting.

They may not have mastered architectural patterns. They may not have experience with testing and code reviews.

The company is exposing itself to a lot of risk if code ships with bugs or the codebase gets crufty.

Older engineers write bad code too but if they worked for Apple the odds are that they have fewer holes in their coding skills -- or at least that's how the thinking goes.

Nobody likes firing people + so hiring mistakes don't get fixed

It's easy to hire someone it's really really hard to fire them.

There have been a few times in my career where I worked with someone who obviously should have gotten fired.

One person was showing up to work drunk and trying to write code. Another really took advantage of the 'unlimited vacation' policy by basically never showing up to work.

In each case it took months for the management to finally admit that a mistake had been made.

Firing someone after investing team time and resources sucks.

If a person has a strong track record the likelihood you'll have to undo your hiring mistake drops.

Newer engineers are up against a lot... but we all got started somehow

I don't want to leave you with the idea that getting work in tech is impossible. Lots of people have broken into the industry even without CS-degrees.

I know this because this was actually my path.

When I graduated I had a lot of coding skills but my degree was in economics.

My own story involved building up a portfolio by doing small-time contracts for clients with small budgets. I then used that portfolio to get bigger more prestigious clients.

Over the years I have learned that getting hired takes strategy based on a realistic assessment of your strengths and weaknesses.

Not all 'employers' are the same.

Some companies are in high demand. Everybody wants to work with them so they're inundated with resumes from rockstars. Other companies need people but don't know how to find them. They'll hire the first good engineer who walks through their door.

Little changes in a resume (like emphasizing your Node skills over your older .Net skills when applying to newer companies, or changing the market you're searching for a job in) are the difference between 20 job offers in a week and crickets.

A little bit of thought about strategy can really help.

I thought it would be useful to end this article with some strategy recommendations.

I hope some of these approaches are helpful to you.

Strategies for getting in the door when you have no work history

To begin with, set realistic expectations - expect the hunt can be hard.

If you're sending out lots of resumes and not hearing back know that this happens to lots of candidates -- it's fairly normal so don't lose hope. All it takes is one company to say yes to hiring to get things rolling.

Sending out resumes can work but in my experience it's better to see if you can bypass HR by talking directly to an engineering manager.

The best way I know of to do this is to attend tech-focused meetups and events. Search for ruby or node or whatever tech meetups and go to the events. After the event employers get up and see if anyone is looking for work. Talk about some of the stuff you've done or things you're passionate about and you'll likely get invited to come in for an interview.

Another strategy you can use is to work with an employer in a contract (often contract-to-hire) role.

As we talked about earlier, the big things that are keeping the employer from working with you are cost and risk. If an employer doesn't have to actually commit to hiring someone full time they're way more likely to take a chance on someone with a bit less experience.

You can find contract work at events, on contract-centric work sites like Upwork. Smaller companies in particular like to work with contractors. This is how I got started and I highly recommend this approach.

Sometimes you really DO want a role in a hard-to-get-into (popular) big tech company and when that’s the case you need to make your resume stand out.

As someone who can't rely on work history to get in the door, you’ll need something else to make your resume stand out.

Some things that you can do ~

Niche ~ get specific about who you want to work for and bring more to the table than just programming skills.

Most candidates are telling employers, ‘you need Java? I have Java’ but the employer really actually needs ‘Java’ to solve educational or fashion or recruiting problems…

If you can bring a passion for the space along with the skills you have a much greater chance of getting an interview.

Demonstrate work the employer is looking for ~ a portfolio with relevant projects is pretty awesome here.

One hack I've seen done is to give yourself a homework assignment. Research a company you'd like to work for and put together a mini project (as if they had hired you). Send the project to them as part of your application.

Demonstrate high quality engineering skills.

Most new grads are missing professional engineering experience. They don't know about testing of version control workflows. They have nothing a potential employer can look at.

The solution? Work on open source projects, go to hack-a-thons, get involved in the engineering community.

All of these things can help you get interviews... once you get interviews all you have to do is pass them.

If you're interested in learning about passing coding interviews I put together a 50 minute Skillshare course about coding interview techniques. It's free for dev.to readers until August 30th. Check it out.

I hope this article has been useful for you and that you have some new ideas for strategies you can use in your job hunt. Good luck and may you find exactly the work you’re looking for.

Permalink

Clojure/ClojureScript Developer(Mobile)

We are looking for a Clojure/ClojureScript developer willing to join new product development. Even you are not Clojure/ClojureScript guru but willing to explore it being a fan of functional development with an understanding of native mobile applications development, modern JavaScript frameworks (like React, Agular, etc.) – you are welcome on board! The product is an AWS hosted multi-module payment and analytical platform for the healthcare services, written in Clojure/Golang/Python language stack. The product encompasses a few applications for customer journeys (web, mobile), data science/data analytics platform, multiple integrations with federal and governmental resources, and complex micro-service architecture. The product’s domain is Healthcare/Fintech, hence all the compliances and accent on security and high-performance

Requirements:

• 3+ years strong development experience
• Experience in building native mobile applications, understanding of React Native
• Strong understanding of Functional Languages principles, OOP, SQL and databases
• Good knowledge of JavaScript, understanding of React or Angular
• At least basic experience with Clojure/ClojureScript (couple months or more), willingness to advance with Clojure and ClojureScript.
• English – Upper-Intermediate
• Strong understanding of software

Would be a plus: 

• Experience with Java, Scala, Python, Lisp or ideally Clojure as a core tool

Responsibilities:

• Work as a part of our product development team
• Participate in solution design and development, deliver high-quality code
• Regularly communicate with the team members in Ukraine and at client side, participate in status meetings, design sessions, and brainstorming
• Provide estimation and reporting for assigned tasks

We offer friendly working conditions with competitive compensation and benefits including:

• Comfortable working environment
• Friendly team and management
• Competitive salary
• Free English classes
• Regular performance-based compensation review
• Flexible working hours
• 100% paid vacation, 4 weeks per year
• 100% paid sick-leaves
• Corporate and team building events

Apply Now 

Permalink

Clojure microservices for JavaScript developers part 3

This series was co-written by Musa Barighzaai and Tyler Sullberg.

This is the third and final post in a series of posts for JavaScript developers about how to set up Clojure microservices. The previous posts were:

• Thinking in JavaScript vs Thinking in Clojure
• Creating Your First Clojure Microservice

Those previous posts are useful context, but you can clone the repo and jump into this post without reading them.

Using the Clojure test API

Compared to JavaScript, a convenient feature of Clojure is that it comes with a built-in unit testing library, clojure.test. In a Leinegen project by convention, you have an src directory which we know holds all the code, and for testing a test directory to separate your code from your tests. For any file under src that you want to test, you create a matching file under /test suffixing with \_test.

Imagine we have a file,adder.clj, in the following file structure of a Leiningen project.

src/
├─ adder/
│  ├─ adder.clj

To write unit tests for adder.clj we would create the following test directory:

src/
├─ adder/
│  ├─ adder.clj
test/
├─ adder/
│  ├─ adder_test.clj

Notice how the test directory mirrors our src directory, and how we add a \_test suffix on the file we want to test.

We want to test the following function in adder.clj.

(defn add-numbers [x y]
  (+ x y))

To write a test in adder.clj for the add-numbers function, you first bring in the Clojure Core testing framework.

(ns adder
  (:require [clojure.test :refer [deftest]))

Making use of the deftest macro, writing tests is as easy as:

(deftest test-add-numbers
  (is (= 4 (add-numbers 2 2))))

In our sample project, we have unit tests for all our HTTP endpoint handlers. Here’s an example of one (found in test/clojure_for_js_devs/handlers_test.clj) . This is testing the handler for our /counter path. The /counter path adds the requester’s IP address as a key to Redis and adds 1 to the value, keeping a count of the number of times an IP has pinged /counter.

(testing "counter-handler"
    (let [response "Counter: 44"
          req (->  (ring-mock/request :get "/counter"))]
      (bond/with-stub! [[redis/getKey (constantly 44)] [redis/incr (constantly nil)]]
        (is (= (handlers/counter-handler req {}) response)))))

There is a lot of going on here, so let us break it down for you.

First, we initialize a request object, req, for the endpoint we want to test (/counter). We are making use of a request mocking library that comes with ring-clojure (our HTTP server framework). This saves us time; we don’t need to write out a full request map.

Next, we use the library bond/with-stub!. In JavaScript the most popular testing framework is Jest. If you have worked with Jest before, bond gives you similar features. jest.mock() lets you mock modules and also state what you want the mock function’s return value to be. That’s what bond/with-stub does for us in Clojure. Since these are unit tests, we want to mock our call to Redis, specifically the keys getKey and incr. For getKey we want to return 44 anytime it’s called in this test, and similarly, for incr we want to return nil.

Finally, we make our assertion that our call to handlers/counter-handler will match our response, Counter: 44. Note that the final parameter in handlers/counter-handler is our Redis component, but in this test, we pass in an empty map {}. Since we are stubbing our Redis calls we can pass an empty map for this parameter, because Redis won’t be required in our test.

How about writing integration tests? In JavaScript, one of the useful features of Jest is the setup and teardown of tests. This feature comes in handy when writing integration tests. For example, if we had an application that queries a database of cities, you would do something like this in Jest:

beforeAll(() => {
  initializeCityDatabase();
});

afterAll(() => {
  clearCityDatabase();
});

test("city database has Vienna", () => {
  expect(isCity("Vienna")).toBeTruthy();
});

The beforeAll/afterAll allows you to run code before and after tests.

Clojure has built-in support for both test setup and teardown. It also makes use of fixtures. Here’s a full example:

(ns adder_test
  (:require [clojure.test :refer [testing use-fixtures]))

(defn my-fixture [f]
  ;; The function you want to run before all tests
  (initializeCityDatabase)

  (f)  ;;Then call the function we passed.
  ;; The function you want to run after all tests
  (clearCityDatabase)
 )
lang:clojure
(use-fixtures :once my-fixture)

(deftest city-db
  (is (= "'Vienna'" (IsCity))))

Here we are calling use-fixtures with :once, which means run my fixture only once, around all the tests in the namespace. You can also pass :each to run your fixture repeatedly for each test. use-fixtures here works the same as beforeAll/afterAll in Jest. We wrap our test (seen in my-fixture as (f)) with the method we want to be called before and after.

You can see a full example of this in the sample project, in test/clojure_for_js_devs/test_core.clj. In post 2 we discussed the purpose of a system-map. A system map allows our application to manage the lifecycle of each software component that it depends on. In our case, our components are our Redis connection and an HTTP server:

(defn- test-system
  []
  (component/system-map
   :redis (redis/new-redis "redis://localhost:6379")
   :http-server (component/using
                 (http/new-server "localhost" 0)
                 {:redis :redis})))


(defn- setup-system
  []
  (alter-var-root #'system (fn [_] (component/start (test-system)))))

(defn- tear-down-system
  []
  (alter-var-root #'system (fn [s] (when s (component/stop s)))))

(defn init-system
  [test-fn]
  (setup-system)
  (test-fn)
  (tear-down-system))

We have a method setup-system that uses the component library (from post 1) which starts the required components (HTTP server, and Redis). In addition to setup-system, we have tear-down-system, which runs component/stop to shut down all components once the tests have been completed.

Now, how do you run tests in Clojure? Without a test runner, you can call (run-all-tests) in your REPL to run all tests in all namespaces. Or, if you’re following along using Leiningen, you can call lein test.

You may have used test runners like Jest with features that run tests automatically when changes are detected in the code. The Clojure community also has various test runners with similar features. The most popular one is Kaocha. We’re using Kaocha in our sample project.

To set up Kaocha, start by adding it as a dependency in the project.clj file under your dev dependencies:

  :profiles {:uberjar {:aot :all}
                 :dev {:dependencies [[lambdaisland/kaocha "1.0.829"]
                                  [circleci/bond "0.5.0"]
                                  [ring/ring-mock]]}}

In JavaScript, you can create custom scripts for custom commands you want to run. For example, npm run prod:ci. Leinegen also has this feature known as an alias. We can create an alias test that loads the lambdaisland/kaocha dependency.

:aliases {"kaocha" ["run" "-m" "kaocha.runner"]}

Finally, add the Koacha configuration file. Create a tests.edn file in your root project directory with the following config:

#kaocha/v1
{:kaocha/color? true}

Now if you call lein test, kaocha will execute the tests and report the results.

Running Clojure tests in CircleCI

If you’re looking for an introduction to the importance of continuous integration (CI), I highly recommend checking out https://circleci.com/continuous-integration/. In this section we will go over our CI workflow and how to run Clojure tests in CircleCI,

In our sample project under .circleci/config.yml we have a workflow for testing our project every time we push a commit to our repo.

version: 2.1
jobs:
  build:
    docker:
      - image: circleci/clojure:lein-2.9.5
      - image: redis:4.0.2-alpine
        command: redis-server --port 6379
    working_directory: ~/repo
    environment:
      LEIN_ROOT: "true"
      JVM_OPTS: -Xmx3200m
    steps:
      - checkout
      - restore_cache:
          keys:
            - v1-dependencies-{{ checksum "project.clj" }}
            - v1-dependencies-
      - run: lein deps
      - save_cache:
          paths:
            - ~/.m2
          key: v1-dependencies-{{ checksum "project.clj" }}
      - run:
          name: install dockerize
          command: wget https://github.com/jwilder/dockerize/releases/download/$DOCKERIZE_VERSION/dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && sudo tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && rm dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz
          environment:
            DOCKERIZE_VERSION: v0.3.0
      - run:
          name: Wait for redis
          command: dockerize -wait tcp://localhost:6379 -timeout 1m
      - run: lein test

workflows:
  build:
    jobs:
      - build

The jobs key is where we define the jobs that can be used in our CI pipeline. Our pipeline has a single job, “build”, which will build our application and then run our tests. “Build” will use Docker images to set up our CI environment. Our Clojure application will run using a pre-built CircleCI Docker image: circleci/clojure:lein-2.9.5. These images are typically extensions of official Docker images and include tools especially useful for CI/CD. In our case, circleci/clojure:lein-2.9.5 comes with Clojure and lein installed. We also use the Redis Docker image since our integration tests will require hitting a Redis server.

Here is a review of all the steps for our “build” job:

1. We first check out our code from our repo.
2. We then wrap the lein deps call that fetches all our dependencies with caching. We don’t need to fresh install all our dependencies on every CI run, so caching will speed up our CI builds.
3. We then bring in Dockerize, a tool which gives our CI the ability to wait for other services to be available before running tests. In our case, we use Dockerize to wait for Redis before running our integration tests.
4. Finally, we run lein test to run all our tests.

We now have our CI pipeline running on every commit, automatically ensuring no one pushes a breaking change to our Clojure project.

And that’s it! You now have a Clojure microservice with basic tests using a continuous integration workflow. We hope you found this series valuable and have gained the confidence to use Clojure in your next project.

Permalink

Clojure microservices for JavaScript developers part 2

This series was co-written by Tyler Sullberg and Musa Barighzaai.

In the previous post, we explored high-level differences between thinking in Clojure compared to thinking in JavaScript. We are now ready to start building our first Clojure microservice.

The microservice we are going to build will be very simple. It will be an HTTP server that uses a Redis data store to count how many times a given IP address has pinged the /counter endpoint.

We will walk you through each step of how to build this service in this tutorial. If you would rather, you can find the completed project here.

Please note that when you are developing in Clojure, you will mostly be working in the REPL, but for the sake of clarity, we won’t use the REPL in this tutorial. We highly recommend that you check out these instructions for using the REPL and start making the REPL central to your Clojure development workflow.

Before we can start building out the project, we should introduce ourselves to the tool that will manage our project structure, Leiningen.

Using Leiningen to automate Clojure project management

Leiningen is a tool for automating aspects of Clojure project management. Leiningen is the most popular tool for setting up Clojure projects. We are particularly fond of it at CircleCI because it was built by our very own Phil Hagelberg.

The closest analogs to Leiningen in the JavaScript world are Yarn and npm. Like Yarn and npm, Leiningen is how you will manage dependencies and publish packages. There are important differences in how Leiningen works, though.

• Leiningen is opinionated about your project structure.
• Leiningen will install dependencies (Maven under the hood) to a common location on your machine that can be used by multiple projects. Yarn and npm install dependencies into the node_modules folder of each project.
• Leiningen runs your tests and configures your REPL.
• Leiningen creates your Uberjar file. Your Uberjar file is a stand-alone executable JAR file that contains your project and its dependencies.
• And more.

Setting up your first Clojure project

First, download Leiningen.

Next, we will create our Leiningen application clojure-for-js-devs

Lang:shell
lein new app clojure-for-js-devs

Now open the clojure-for-devs directory in an editor and you will see that Leiningen has built out a project for you.

Most of these files are self-explanatory - the src directory and test is where we will write our tests. Similar to ES or CommonJS modules, each .clj file we create in these directories will be its own namespace. The project.clj is the most important file of this project.

(defproject clojure-for-js-devs "0.1.0-SNAPSHOT"
 :description "Simple app to demonstrate how to construct Clojure microservices."
 :url "https://github.com/tsully/clojure-for-js-devs"
 :dependencies [[org.clojure/clojure "1.10.1"]]
 :main ^:skip-aot clojure-for-js-devs.core
 :target-path "target/%s"
 :profiles {:uberjar {:aot :all
                      :jvm-opts ["-Dclojure.compiler.direct-linking=true"]}})

project.clj is very similar to a package.json file:

• dependencies is where we will bring in Clojure/Java dependencies. This is the same as dependencies in a package.json.
• main is the entry point for your application. When you start a Lein application with lein run, it will run the main function in whichever namespace you specify here. There is already a main function in the clojure-for-js-devs.core namespace. This is functionally the same as the main key in a package.json.
• target-path is the directory where compiled files will be placed when you run lein uberjar to build your production application. If you have used webpack before, this is similar to the output key instructing webpack where it should output your bundles.
• profiles allows you to customize your project.clj setup for development, production, or the REPL. Profiles allow you to customize any aspect of your project.clj. You can bring in specific dependencies, include additional directories, and change compilation options, among other things. Our project.clj has only one profile, uberjar, which is used for compiling our production application. You can add other profiles like dev and repl. There is not a great comparison for “profiles” in Node.js. To recreate similar functionality in Node.js, you often need to use a combination of environment variables likeNODE_ENV), along with scripts, devDependencies, and config files to get the job done.

You can also create custom scripts in the same way that you can create npm scripts, but we will cover that in the next post.

Note that the configuration of the project.clj is wrapped in a defproject. Defproject isn’t a new Clojure construct you need to learn, it’s just a macro from Leiningen. This is why the body of defmacro seems to defy the normal Clojure syntax that we’ve grown to know and love.

Creating the components of the application

Now that we have the shell of our application, we’re ready to start creating the major components, our HTTP server and Redis client.

In JavaScript, to create our Redis client, we’d do something like:

const redis = require("redis");
const client = redis.createClient();

client.on("error", function (error) {
  console.error(error);
});

client.set("key", "value", redis.print);
client.get("key", redis.print);

And to create our HTTP server, we might do something like:

const express = require("express");
const app = express();

app.get("/", function (req, res) {
  res.send("Hello World");
});

app.listen(3000);

In other words, to create these components, we end up creating a giant object that has all of the methods and properties we need to conveniently manage the lifecycle of that component.

Of course, since Clojure is a functional language, our approach is going to look different:

• Instead of burying our functions as methods in objects, the functional approach is to define functions that can be freely composed and reused.
• Instead of defining how an object should behave via interfaces and inheritance, Clojure uses polymorphism.
• Instead of creating state, Clojure will (generally) use pure functions and immutable data structures.

Before we can show the functional approach for building the HTTP server component and Redis component, we need to give a bit of background about two Clojure constructs that may be unfamiliar to you if you’re a new Clojure developer: defprotocol and defrecord.

Learning how these concepts work will be crucial for understanding how your microservice is structured.

Understanding defprotocol and defrecord

defprotocol defines a set of methods along with their function signatures. Protocols do not have an implementation, but any data type can implement that protocol and define the implementation for each of the protocol’s methods. extend-protocol is one way of defining the implementation of a protocol for a data type.

(defprotocol Bird
 (eat-bread [this bread-type]))

(extend-protocol Bird
 java.lang.String
 ; the implementation of the eat-bread function if called with a string
 (eat-bread [name bread-type]
   (println (str name " is eating my " bread-type " loaf")))
 ; the implementation of the eat-bread function if called with a long
 java.lang.Long
 (eat-bread [number-of-birds bread-type]
   (println (str number-of-birds  " birds are eating my " bread-type " loaf"))))

(eat-bread "Andrew Bird" "sourdough")
; Andrew Bird is eating my sourdough loaf
(eat-bread 3 "sourdough")
; 3 birds are eating my sourdough loaf

In this example, we create a protocol called Bird which defines the function signature for eat-bread. Notice how the first argument of eat-bread is this. The this in the defprotocol is superficially similar to “this” in JavaScript. Iit represents the data that is implementing this function. ‘This` isn’t a protected keyword in Clojure. We could name that first parameter anything.

When we use extend-protocol on Bird, we’re creating the actual function implementations of the Bird protocol. These function implementations will be different based on the data type of the first argument (i.e. this). This is polymorphism. Different function bodies are executed based on the data type of the first argument. If it’s a string, we execute one function body, and if it’s a long, we execute another.

So, what if we want to extend the Bird protocol with our own data type rather than a primitive type? We can create our own data type with defrecord. defrecord creates a Java class that can have properties and implementations of protocols. We can use the -> constructor to make an instance of this class. Let’s make a defrecord for a Duck:

(defrecord Duck [name])

(def reggie-the-duck (->Duck "Reginald"))
(println reggie-the-duck)
; #clojure_for_js_devs.core.Duck{:name Reginald}

To give our Duck the ability to eat-bread and make-noise, we can use the Bird protocol. We can use extend-protocol to define how the function signatures defined in the Bird protocol should be implemented for the Duck data type. A better way to implement the protocol is to define the protocol’s implementation directly on the defrecord.

(defprotocol Bird
 (eat-bread [this bread-type])
 (make-noise [this repeat-num]))

(defrecord Duck [name]
 Bird
 (eat-bread [_this bread-type]
   (println (str name " is eating my " bread-type " loaf")))
 (make-noise [_this repeat-num]
   (println (repeat repeat-num "Quack! "))))

(def reggie-the-duck (->Duck "Reginald"))

(eat-bread reggie-the-duck "sourdough")
; Reginald is eating my sourdough loaf
(make-noise reggie-the-duck 3)
; Quack!  Quack!  Quack!

defrecord and defprotocol will be central to how we structure our microservice, so learning these concepts now will give you much more confidence in your ability to build Clojure microservices.

Using stuartsierra/components

The stuartsierra/component library will be the heart of how our application manages the runtime state of our Redis client and HTTP server components. stuartsierra/component is a framework for managing the lifecycle of components. It makes sure components are stopped and started in the right order and explicitly declares the shared state between components. This framework allows us to make sure that our Redis client is started before our HTTP server, and that the HTTP server knows how to interact with Redis. Components will seem a lot like OOP objects, but remember that we’re still living the Clojure paradigm of pure functions and immutable data structures.

Setting up Redis

Now we can created our first component, the Redis client. Go to your project.clj file and add stuartsierra/component and carmine, a Clojure Redis client, to the dependencies.

 :dependencies [[org.clojure/clojure "1.10.1"]
                ; Add the following two dependencies
                [com.stuartsierra/component "0.4.0"]
                [com.taoensso/carmine "2.20.0"]]

Next, create a redis.clj file in your src/clojure_for_js_devs directory with the following code.

(ns clojure-for-js-devs.redis
 (:gen-class)
 (:require [com.stuartsierra.component :as component]
           [taoensso.carmine :as car]))

(defrecord Redis [uri connection]
 component/Lifecycle
 (start [this]
   ; If there's already a connection return the instance of the Redis class
   (if (:connection this)
     this
     ; Otherwise, associate the 'connection' property of this defrecord with
     ; a map representing the Redis connection
     (do
       (println "Starting Redis component")
       (println "Redis connection URI" this)
       (assoc this :connection {:pool {} :spec {:uri uri}}))))

 (stop [this]
   (if (:connection this)
     (do
       (println "Stopping Redis component")
       (assoc this :connection nil))
     this)))

(defn new-redis
 "Create instance of Redis class"
 [uri]
 (map->Redis {:uri uri}))

Let us break down what’s going on here. We’re creating a defrecord called Redis and a function to create a new instance of Redis that takes a URI as its only argument. In the Redis component, we implement a protocol called component/Lifecycle. This protocol comes from the stuartsierra/component library and has two function signatures: start and stop. The stuartsierra/component will call the start and stop function on our new Redis defrecord to manage this component’s lifecycle.

So far, this component just keeps track of a Redis connection, but doesn’t actually have any functions for interacting with it. Instead of defining these functions inside of the Redis class, like we might with an object-oriented language, we’re going to define separate functions that can take a Redis class as an argument. Add these these to the bottom of your redis.clj file.

(defn ping
 "Check that Redis connection is active"
 [redis]
 (car/wcar (:connection redis) (car/ping)))

(defn getKey
 "Retrieve count for a key in Redis DB."
 [redis key]
 (car/wcar (:connection redis) (car/get key)))

(defn incr
 "Increment count for a key in Redis DB."
 [redis key]
 (car/wcar (:connection redis) (car/incr key)))

We need to also create a docker-compose-services.yml file in the root directory so that we can run a Redis server during development

version: "2"
services:
  redis:
    image: redis:4.0.2-alpine
    ports:
      - "127.0.0.1:6379:6379"

Creating the component system map

Next, we need to create a “component map” that will describe how our components, including Redis, will interact with each other. This component map will initially include only Redis, but will add in the HTTP server component afterwards. Move over to the src/clojure_for_js_devs/core.clj file and copy/paste the following code.

(ns clojure-for-js-devs.core
 (:gen-class
  :main true)
 (:require
  [com.stuartsierra.component :as component]
  [clojure-for-js-devs.http :as http]
  [clojure-for-js-devs.redis :as redis]))

; Create a variable called *system* that can only be defined once
; ^:dynamic means that this variable can be rebound to a new value
(defonce ^:dynamic *system* nil)

(defn main-system
 "Creates map of component dependencies with implementation of Lifecycle protocol"
 []
 (component/system-map
  :redis (redis/new-redis "redis://redis:6379")))

(defn start
 "Start components of system in dependency order. Runs SystemMap implementation of Lifecycle protocol's 'start' function"
 [system]
 (try
   (component/start system)
   (catch Exception _ex
     (println "Failed to start the system"))))

(defn stop
 "Stop components of system in dependency order. Runs SystemMap implementation of Lifecycle protocol's 'stop' function"
 [system]
 (component/stop system)
 ; dynamically rebind *system* var back to nil
 (alter-var-root #'*system* (constantly nil)))

(defn -main
 "Entry point to the application."
 []
 (let [system (start (main-system))]
   ; dynamically rebind *system* to the newly created SystemMap instance
   (alter-var-root #'*system* (constantly system))
   ; Create hook that stops the component system in a controlled manner before the JVM completely shuts down
   (.addShutdownHook (Runtime/getRuntime) (Thread. (partial stop system)))))

If you’re interested in the details of what’s going on here, the stuartsierra/component source code is a short and useful read. Here is a quick overview:

• When the application starts, the main function will be automatically called (see: the :main key of the project.clj)
• The main function will create a SystemMap (remember, an immutable data structure, not an object) that represents all of the dependency relationships between the components of the application.
• Under the surface, the SystemMap is implemented as a defrecord that implements the Lifecycle protocol’s stop and start methods. It does so in the same way that our Redis defrecord implemented the Lifecycle protocol.

The last line deserves further explanation:

(.addShutdownHook (Runtime/getRuntime) (Thread. (partial stop system)))

We’re creating a shutdown hook here to detect when the application is shutting down, so that we can cleanly stop our component system first. The . in addShutdownHook signifies that we’re using a Java method of the Runtime Java object. The . at the of Thread. signifies that we’re creating an instance of Java Thread class.

Setting up the HTTP server

Start off by adding some new dependencies to the project.clj.

  :dependencies [[org.clojure/clojure "1.10.1"]
                 [com.stuartsierra/component "0.4.0"]
                 [com.taoensso/carmine "3.1.0"]
                 [compojure "1.6.1"]
                 [ring/ring-core "1.8.0"]
                 [ring/ring-defaults "0.3.2"]
                 [ring/ring-jetty-adapter "1.8.0"]
                 [ring/ring-json "0.5.0"]]

Now, create a new http.clj file in the src/clojure_for_js_devs directory. We’ll start by making the defrecord of the server.

(ns clojure-for-js-devs.http
 (:require
  [com.stuartsierra.component :as component]
  [compojure.core :refer [GET routes]]
  [compojure.route :as route]
  [ring.adapter.jetty :as jetty]
  [ring.middleware.defaults :as ring-defaults]
  [ring.middleware.json :as ring-json]
  [clojure-for-js-devs.handlers :as http-handlers]))

; this is just a function stub - we'll build this out soon
(defn start-server
 "Start the HTTP server with our routes and middlewares."
 ([host port redis-component]))

(defrecord WebServer [host port]
 component/Lifecycle
 (start [this]
   (if (:server this)
     this
     (let [server (start-server host port (:redis this))]
       (assoc this :server server))))
 (stop [this]
   (when-let [server (:server this)]
     (.stop server))
   (dissoc this :server)))

(defn new-server [host port]
 (->WebServer host port))

The WebServer defrecord is built roughly the same way as the Redis deferecord we created earlier. WebServer implements the Lifecycle protocol and we’ve created a new-server function that makes an instance of the WebServer class. We haven’t yet built out the start-server function, but we will do that shortly.

There’s one important difference, however, between our WebServer and Redis defrecords. Notice how when start-server is called, we pass in (:redis this) as its third parameter. What’s going on here? Where did :redis come from? If this refers to an instance of the WebServer class, then when did a :redis key end up in that data structure?

The :redis key is going to be added into this component by the stuartsierra/component library. Go back to core.clj file and modify the system map so that our WebServer can know about the Redis component.

Modify the imports and main-system function of your core.clj file with the following code:

(defn main-system
 "Creates map of component dependencies with implementation of Lifecycle protocol"
 []
 (component/system-map
  :redis (redis/new-redis "redis://redis:6379")
  :http-server (component/using
                (http/new-server "0.0.0.0" 8080)
                [:redis])))

We’ve added our HTTP server as another component in our system map, and we’ve also made the Redis component a piece of state that our HTTP server will have access to under the :redis key.

Head back over to http.clj so we can create our routes and middleware. To do this, we’re going to use a few libraries:

• Jetty is Java HTTP server. We’re using Jetty via a Ring library that adapts Jetty for Clojure.
• Ring is a collection of libraries that creates our Jetty server and defines the middleware for processing requests and sending responses.
• Compojure is a routing library for Ring.

Now we’re ready to create our routes. At the top of http.clj, but below where we import in external namespaces, add the following snippet.

(defn app-routes
 "Creates Ring handler that defines routes and route handlers"
 [redis-component]
 (routes
  (GET "/hello-world" [] (http-handlers/hello-world-handler))
  (GET "/ping" [] (http-handlers/ping-handler redis-component))
  (GET "/counter" req
    (http-handlers/counter-handler req redis-component))
  (route/not-found "Not found")))

The app-routes function will take in a redis-component as an argument which it will then pass to its route handlers. Go ahead and create the route handlers. Make a new file handlers.clj in your src/clojure_for_js_devs directory, and add the following code.

(ns clojure-for-js-devs.handlers
 (:require [clojure-for-js-devs.redis :as redis]))

(defn hello-world-handler
 "To check that HTTP server is working."
 []
 "howdy!")

(defn ping-handler
 "To check that HTTP server can interface with Redis."
 [redis-component]
 (println "Handling ping request")
 (redis/ping redis-component))

(defn counter-handler
 "Increment count of times that IP address has hit endpoint and return count."
 [req redis-component]
 (let [ip (:remote-addr req)
       counter (redis/getKey redis-component ip)]
   (redis/incr redis-component ip)
   (str "Counter: " counter)))

Now that we have our HTTP handlers, go back to the http.clj file. So far, we’ve defined our routes, but we haven’t yet actually connected these routes to our HTTP server, defined how incoming requests will be processed, or even started our Jetty server. Replace the function stub we made for start-server with the following function.

(defn start-server
 "Start the HTTP server with our routes and middlewares."
 ([host port redis-component]
  (-> (app-routes redis-component)
      ;; Turn map response bodies into JSON with the correct headers.
      (ring-json/wrap-json-response)
      ;; Turn JSON into map
      (ring-json/wrap-json-body {:keywords? true})
      ;; Parse query strings and set default response headers.
      (ring-defaults/wrap-defaults ring-defaults/api-defaults)
      (jetty/run-jetty {:host host
                        :port port
                        :join? false}))))

It is helpful to understand what the -> is doing here. -> is a called threading macro. Remember how in the first post we said that a macro lets you extend the Clojure language to things that otherwise wouldn’t syntactically be possible? Well, that’s what’s happening here. This threading macro works by executing the first function after the -> and then passing the result to the next function in the list. You could accomplish the same thing with a bunch of nested function calls, but this looks prettier, right?

start-server is defining how an incoming request will be processed. It begins by defining the routes, then defines how request is processed, then finally starts the jetty server.

Our microservice is now functional and ready to go! Fire it up:

docker-compose -f docker-compose-services.yml up
lein run

Go to http://localhost:8080/counter in your browser to see if the routes work.

As you refresh /counter, it should increment.

Wrapping up

Congratulations! You’ve just created your first Clojure microservice. You’ve learned the basics of how we structure our application with Leiningen and the stuartsierra/components library. You’ve also learned the key Clojure concepts defrecords and defprotcols. You now know about macros that are central to how these libraries work.

We’re not done yet, though. In the next post, we’re going to build out testing and continuous integration (CI) for our new microservice.

Permalink

deps.edn and monorepos IV

This is part of an ongoing series of blog posts about our ever-evolving use of the Clojure CLI, deps.edn, and Polylith, with our monorepo at World Singles Networks.

Permalink