Roam is like a good swiss army knife, it even has a ClojureScript development environment. I spent the past 14 days getting acquainted with {{roam/render}}. This post serves as a note-to-self, to summarize and organize what I have learned. This post is targeted at developers and Roam hackers. If you are not in one of these camps, you will likely struggle with the content. I am not a web
Don't worry. In this post, I will share some of the basics that I have learned about Clojure and ClojureScript.
With Clojure, you can write simple, functional and minimalistic code. Some of the cool things with Clojure are:
Clojure runs on top of Java, suitable for all kinds of "backend" things. It also runs on top of JavaScript. Yes, JavaScript - making it possible to develop all kinds of apps, tools, products and websites. There's even an option to run in a .NET runtime, but I haven’t tried out that one yet.
Even though the syntax may look unfamiliar, you can already see in this silly example the less amount of keystrokes needed using Clojure, compared to JavaScript.
Everything you write in Clojure is a list, wrapped in parentheses. That may sound weird. I agree, it is weird. But your eyes will get used to it. In short, the format of writing something in Clojure is:
(operator operands) or even (function arguments) Here is an example:
... and how it would be written in JavaScript:
If all this is new to you, you might have this expression in your face right now: 😬
That is perfectly normal. I have looked like that a lot during 2020. When digging deeper, you may need to go through The 12 Stages of Learning Clojure.
With ClojureScript, you develop web apps and write simple React components. You write code in Clojure, compile it to native JavaScript and run it in your favorite browser. The compilation happens all the time and automagically as you type.
Just like with Java or .NET in the backend, ClojureScript has native access to JavaScript. You can use and interact with your favorite npm packages and React components just as you would with JavaScript.
I'm going to fast-forward a bit by adding a ClojureScript library called Reagent. With Reagent, you write React components but with Hiccup instead of JSX. Wait, what?
View components are built up using Clojure data structures. The example above creates html elements with Clojure vectors and keywords. This style is called Hiccup. It will save you a lot of keystrokes and you'll have a lot less of scrolling in your code editor.
I'll fast-forward again by adding a very nice library called re-frame.
From the re-frame docs:
“... It has reactivity, unidirectional data flow, pristinely pure functions, interceptors, coeffects, conveyor belts, algebraic effects, statechart-friendliness and claims an immaculate hammock conception. All while being both simple and easy. ...” 💥🤯💥
With re-frame, data is stored in one place called the "app db". Your components will fire events and subscribe to changes in the app db. The cool thing is that you write code in a sequential way, making it very readable and easy to understand. However, the syntax of the actual events and subscriptions might take a while to understand, at least if you are new to Clojure in general.
Here, we have a Reagent component that adds a button component, only when the app is disconnected. It is subscribing to changes of disconnected in the app db. React will re-render the components when the app state has changed.
Yes, most likely. I know there are Clojure plugins for VS Code, Atom, Sublime Text and intelliJ. I use Emacs, because ... I just like it.
There is something important and special going on with developers and their text editors. I think you will be able to find your favorite dev setup. I also think that you will learn a lot of new programming concepts and styles when reading more about this cool language and its libraries.
Photo by Marcos Paulo Prado on Unsplash
You have probably already heard about the red-green-refactor workflow when practicing Test Driven Development: start with a failing test, make the test pass, refactor and repeat.
I don’t think I have ever been very strict about the TDD workflow. I usually jump back and forth between the code and the unit test, often begin with trying out naming of functions and less-important things like that. I also find refactoring difficult at that stage, probably because the goal is to write code that does something and that does it well enough. I find it much easier to reflect and come up with refactoring ideas once I see a Pull Request diff on GitHub.
Until recently, I have failed to find a workflow that is seamless when writing, running and testing code. There is often context switching involved: executing commands that run tests in a terminal or navigating the test runner of an IDE. Sometimes that means a pause with a delay of seconds, or even minutes. The eyes and the brain have lost attention to the code. The focus and flow is gone. Now where was I?
These days, the programming language that makes me 🤩 is Clojure. It has been quite a journey learning how to use the language and how to use the code editor properly. I guess most of us don’t pick up all the good parts of a language or a tool immediately. It takes a while to get there.
An example: my brain hasn’t had the bandwidth enough to get how to use the interactive REPL until recently. There has been a lot of other things to unpack: language syntax, functional programming concepts and keyboard shortcuts. On a positive note: there are opportunities for daily learning and that’s where I am today.
- Wow, I can evaluate an expression in the code editor and the result pops up right next to it!
Star struck. 🤩
The other day I was frustrated with the slow feedback from a CLI test command running in a terminal window. I guess it has something to do with the Java Virtual Machine starting up every time a command is executed. I’ve tried out tools with file watchers, reacting to changes, that speed things up - but there’s still context switching involved, moving your eyes off the editor & the code.
What would a Clojurist do? I think the answer would be:
- Use the REPL.
With Clojure, using the REPL means writing code in the editor, with autocomplete and everything. Also, evaluating the code while in the editor. There is no need to write a statement in a separate terminal window. Your editor has a connection to the interactive REPL (a running server), that will evaluate your code and return the result back to you. With the interactive REPL, you can quickly try things out, without switching to a different tool or view.
This is what some people call REPL Driven Development. Before I learned about it, I have always thought of a REPL as something that is used in a terminal window, away from the editor and away from the code.
By combining REPL Driven & Test Driven Development we have a setup that solves the issues with context switching and slow test runs. From what I’ve learned and tried out so far, I think it is a very nice workflow. 🤩
With REPL Driven Development, you write code, evaluate it and get instant feedback while keeping your focus on the code. Maybe the quick visual feedback in the editor is good enough to continue working? Great. Or, you write one or two commands wrapped in a runnable Clojure (comment) block just below the functions, to simplify the evaluation.
Everyone needs a coffee break. Commit the code (yes, the commands too) and continue with it later. ☕
Your brain has been charged and you feel refreshed again. When returning to the code, you realize that the commands are probably evolving into something similar to a unit test. Should it be a proper unit test? If so, wrap it in a (deftest) function with some assertions and put it in a separate file.
Here’s a tip: give the unit test the same namespace as the actual code, but add a -test suffix to it. When doing so, your code editor (if it is smart) can run the corresponding tests when in a namespace. If you are an Emacs user, simply press the keys C-c C-t n to run the tests.
Even though you have put the unit tests in a separate file, in a separate folder, there is no need to switch context. Your tests run from where you are and you get instant feedback on the result. Very cool.
I think this is Test Driven Development Deluxe. 🤩
☕
Simpler is better
Ooops, I did it again! In my first blog post I described how cool was my setup with
shadow-cljs, clojurescript and react-native and big my ambitions of improving my knowledge
of this amazing stack was.
Well, I learned something (see this blog post) but I am going to be honest: the setup had its quirks. Org files would be converted to markdown and the markdown would be then exported to HTMl by a clojure library. Plus, the routing logic was using another clojure library which was kind of hard to work with: I spent hours trying to place a link to my resume.
I sat down and I thought: what do I want from my blog? I want it to be simple so I can focus on writing. I was losing enough time with my previous setup to make everything work. Still, it was a very valuable experience. It is still available at this URL.
Since that first blog post in May 2020, I must admit I have become even more of an emacs fanboy.
I am now heavily invested in org mode for all my prose and most of my experimental code
(I'll write about my emacs-jupyter setup one day).
Recently I stumbled upon a really elegant solution that uses the emacs exporting facility to export
org to HTML and collects them in a static site. That's it.
As a developer you need to worry about three things:
publish.el file)You can basically copy the minimal working example from the github project and start writing your org blog posts. You can then refine the default style and templates.
Once the blog is setup, you can just focus on the writing. Oh, and it provides an RSS feed out of the box!!
Starting from the basic example I added a few nice things.
I started with defining a Github Action that on every build will re-generate the website and publish it
to Github Pages. In the Github Action I install emacs and run emacs --script publish.el.
In my publish.el script I check for a GHUB env variable and if present I install the required dependencies.
I also ported the 2 tools I was using in my previous setup, the Hyvor Talk comments and
the clicky.com analytics. To do so I simply had to add HTML to the index.html and post.html templates.
Finally, I edited some CSS to adjust the default size of the header and to add the dates of each blog post to the index.
I just had to edit the blog.html templates, where I can extract file properties of org files with this syntax:
<h1>Posts</h1> <ul class="posts"> {% for post in posts %} <li class="post-title"> <a href="{{ url_for("posts", slug=post.slug) }}">{{ post.title }}</a> <p class="post__meta"> {{ post.date|strftime("%b %d, %Y") }} {{ post.filetags }} </p> <p> {{ post.subtitle }} </p> </li> {% endfor %} </ul>
The default style of the blog is not as nice as the previous one I had but if needed I can
adjust the post__meta class. I plan to start using Tailwind CSS soon.
In conclusion, I had a lot of fun porting my posts over this new setup and in addition I am confident this will further reduce the friction between me and blogging!
In Clojure we can use the merge function to merge multiple maps into a single map. If a key is in multiple maps the value of the key merged last will be used in the resulting map. If we want to influence how the value of a duplicate key is set we can use merge-with. We specify as first argument the function that will be used when the same key is available in multiple maps. The function must accept two arguments, where the the first argument is the value of the key in the first map and the second argument the value of the same key in the following map. The result is assigned to the key in the resulting map. If we pass more than two maps to the merge-with the function will be called multiple times for a key if it is part of more than two maps.
In the following example we use Clojure core functions and a custom function to merge multiples maps, so we can alter the value for duplicate keys:
(ns mrhaki.core.merge-with
(:require [clojure.test :refer [is]]))
;; Merge maps and use the function specified as first argument
;; to calculate the value for keys that are present
;; in multiple maps.
(is (= {:a 60 :b 3 :c 44 :d 100}
(merge-with * {:a 2 :b 3 :c 4} {:a 10 :c 11} {:a 3 :d 100})))
;; Works for all maps and independent of type that is used for keys.
;; We can use any function for merge-with.
(def languages (merge-with (comp vec flatten conj) {"Clojure" [:dynamic :functional]}
{"Java" [:jvm]}
{"Groovy" [:jvm]}
{"Clojure" [:jvm]}
{"Groovy" [:dynamic]}))
(is (= {"Clojure" [:dynamic :functional :jvm]
"Java" [:jvm]
"Groovy" [:jvm :dynamic]}
languages))
;; Sample map with small inventory.
(def inventory {"pencil" {:count 10 :price 0.25}
"pen" {:count 23 :price 0.4}})
;; Sample basket with items.
(def basket {"pencil" {:count 5} "pen" {:count 2}})
;; Function to subtract the :count value for a basket item
;; from the :count value for the same inventory item.
(defn item-sold
[inventory-item basket-item]
(update-in inventory-item [:count] - (:count basket-item)))
(is (= {"pencil" {:count 5 :price 0.25}
"pen" {:count 21 :price 0.4}}
(merge-with item-sold inventory basket)))
Written with Clojure 1.10.1.
The other day I was looking for a way to turn a map into a bunch of vars. I know a macro is what will get the job done but I write macros so rarely that it always takes me a while to figure it out. In the end I ended up with something like this:
(defmacro def-all [m]
(->> (for [[n v] m]
`(def ~(symbol n) ~v))
(into [])))
Using macroexpand you can see that this translates to the a bunch of def calls in a vector:
user=> (macroexpand '(def-all {:a 1 :b 2}))
[(def a 1) (def b 2)]
Like myself, you may wonder why the vector is needed. The issue is that for will return a list and that would result in the macro emitting the following Clojure code:
((def a 1) (def b 2))
After evaluating the two inner def forms, this will result in another function call where the return value of the first def is used as a function. Depending on what you are defining this may fail or lead to unexpected behavior.
After sharing my solution using (into []) in the Clojurians Slack I was made aware that instead if turning the thing into a vector you can also just prepend a do into that list, resulting in code that feels slightly more aligned with my intention:
(defmacro def-all [m]
(->> (for [[n v] m]
`(def ~(symbol n) ~v))
(cons 'do)))
(macroexpand '(def-all {:a 1 :b 2}))
; returns
(do (def a 1) (def b 2))
I realize this is a super basic macro but I can totally see how that might be useful to people starting to write their own macros. If you're looking for a more full-fledged guide, Clojure for the Brave and True got you covered.
Thanks to Justin Smith for sharing his experience on Slack with me so many times.
Description of the problem Today I am going to implement the Stoer-Wagner algorithm that finds...
Slow, steady progress
To continue with the series of experience reports at Magnet, at this time we bring you eKanban project, a digital platform that helps organizations handle their manufacturing process following the Kanban methodology. This project showcases another implementation of our Hydrogen platform, in this case to automate the logistics system at Orkli.
eKanban is a scheduling system for lean and just-in-time manufacturing. All the manufacturing process is monitored starting from the supplier of the raw materials, the preparation and delivery of them, stock availability in the production line and the usage of it. The main goal is to monitor the process and to provide information to both actors in the process, the suppliers and the consumers of every material in use. That is, generating information and optimizing the stock needed in each part of the process for an optimal production.
The first version of eKanban was created as an internal project to fulfill the organization’s needs, using internal resources. Once it was implemented, other companies showed interest in it, and Orkli decided that although the solution worked and it was a great proof-of-concept, they needed help to improve it and make it commercially viable.
At Magnet we use this mantra often: make it work, make it right and make it fast.
In the case of the eKanban project, our mission was to take this proof-of-concept and make it right and fast.
eKanban circuit view with all information and detailsWe started tackling some very specific requirements to improve a partially working in-house system with the goal of evolving it into a market-ready professional solution.
Cloud infrastructure
The previous system was deployed on premises which, for several reasons, did not escalate with business needs. Hence we decided to migrate it to a cloud infrastructure and chose AWS for its scalability, robustness, security, accessibility and easy maintenance.
Our client did not have any previous experience with AWS, but they quickly saw the added value and they did not hesitate to embrace the cloud.
Modeling the domain knowledge
After the infrastructure was set we immersed ourselves in the domain at hand, that is logistics, which is a vast and complex area.
The previous data model was not architectured for scalability and this presented a challenge. For better communication and understanding with the client we wrote a domain dictionary and decided to get rid of the previous data model and start fresh with a new and more efficient one.
Thanks to this we have a more logical organization of the data and a much better performance and agility of the system. It also provided more flexibility and the possibility of new features, which the old model couldn’t handle.
Leveraging Hydrogen platform
Hydrogen accelerated the development of eKanban bringing many technical ingredients we needed: database, authentication, business intelligence, web server, the continuous delivery automation, etc.
There were some features that we were missing so we built additional modules such as the scheduling and the object storage implementation for ftp servers.
Scheduling
We identified that quite a lot of functions should be scheduled and fired in certain moments. Not just internal logics as checking if stock should be re-evaluated or alert emails sent, but also providing the companies the flexibility to decide their own frequency or periodicity of the task.
With this new module, our internal functionality is robust and covers what the system needs. Now, they can offer their clients their own personal customization of scheduling, and it can be done in a totally automatic way, with not coding or technical skills.
Object Storage in FTP servers
Another requirement of the project was to enable integration with different ERPs, as some information needed in the eKanban logic is generated in these systems (order numbers), and also, once processed in the system, it has to be persisted in the ERP (stock).
Hydrogen offers a REST API layer and ideally this ERP integration should be done through this HTTP protocol. But alas, the real world is stubborn and some ERPs are not ready, so we had to explore alternatives. Our final decision was to exchange XML files via a FTP server. The actual object storage module can fetch and store objects indistinctly in AWS S3 service or an FTP server. With a boundary that abstracts totally the storing indistinctly to one or other.
The connection to a FTP server is totally configurable and reading and writing to it is customizable. Once this kind of communication was in place, our client has implemented different tasks on top of it providing the users with further services not present in the previous version of the application.
Business Intelligence
We leverage Grafana as part of Hydrogen when we need to deliver this functionality.
In this case, a further development was done to securitize and automate what each user was allowed to see and how. The same token used to access our backend is then used to fetch graphs with the data owned by the authenticated user.
Additionally, the eKanban team can access Grafana as a platform and use all its features for their own business intelligence purposes.
Web services
The exposed REST API is used by two main actors:
These third parties have integrated the service securely quite easily and transparently.
Front-end
All the user interfaces have been redesigned, as the previous front-end wasn’t scalable and didn’t fit the functional requirements.
The usability has been improved and the user feedback has been very positive. It’s a reactive and rich-feature user interface that makes the user experience much smoother.
Conclusion
It’s not easy to re-build something from scratch, but sometimes you have to consider your first effort as a proof-of-concept and reuse your knowledge to build a more productive solution.
In this case, the legacy system proved the validity of the business value proposition, but it was not performant and the data model, among other things, made it difficult to scale it. Migrating from an on-premises model to a cloud native model was a game changer as well.
Building the system from scratch was a risky decision but served us well. No looking back.
Successful migration of a logistics solution from a legacy on-premise system to the cloud. was originally published in magnet.coop on Medium, where people are continuing the conversation by highlighting and responding to this story.
Data Oriented Programming is an exciting new paradigm that eliminates the usual complexity caused by combining data and code into objects and classes. DOP cleanly separates code and data, which simplifies state management and eases concurrency.We've asked Yehonathan Sharvit, the author of the book Data Oriented Programming to introduce us to the DOP principles and tell us why every developer should learn this programming paradigm. I have been a software developer since 2001. After 10 years of suffering with C++, Java, JavaScript and Ruby, I discovered Clojure in 2012. Since then, I've been having a lot of fun in coding both for software companies and on open source project...
Tags: Clojure, Functional Programming, Java, Javascript, Learning, Object-Oriented Programming, Programming Languages
Continue reading
Watch me program live every morning at 9:30 EST!
Check out past streams on YouTube!
The downward slope of Month Two, is it really here again? Are we really here in the last week of the month?
Before we dive into the meat of this week’s wrapup post, I have an announcement: the development of Month Two’s startup, Pic Story, will be extending two weeks into Month Three. You may recall that I stopped production on Month One’s startup, Plotters, due to the fact that I had hit the end of the month and was nowhere close to finishing. Back then, oh so long ago, I felt the need to commit to my format in order to give myself the positive pressure to keep things small and simple. I’m once again in a position where I’ve bitten off more than I can chew in a single month, but I don’t think it would be right to not finish a second time.
Adhere too strictly to the rules, and they can transform from something that informs your success to something which impedes your success. More on that idea below 👇
I am a little bit disappointed to once more miss my target, but it also offers some exciting opportunities for experimentation! I’ve got a long list of things which I’d like to build, but some of them are really tiny: not enough to fill up a whole month. I figure I can spend the final two weeks of Month Three to build out one of these extra tiny startup products to see what happens with it. Stay tuned.
And now for (what used to be called) the failure of the week feature! This weird world is great at showcasing a success, but behind every success lies ten hard-won lessons, and we don’t get to see very many of those. In each weekly recap, I’m going to put these hard lessons up front to show you that it takes a lot of trial and error to get anything right in the end.
I’ve been trying to find a better word than “failure” for the things I’m experiencing. If you take a sampling of dictionary entries for the word “failure” (so cliché, I know), you see things like:
But none of those definitions is what I’ve been writing about in this recurring segment, is it? I’ve been writing about confounded expectations, unexpected outcomes, hidden truths, secondary implications of the things that I’m setting out to do. Really what I’ve been documenting is the gap between expectation and reality. I’m not really “failing” in the sense that I’m not achieving my purpose, it’s more like a process of feeling out the territory. Learning the moves. Hitting that Goomba as many times as it takes until you learn to jump to avoid it.
So anyway, what is the Goomba of the Week then? This time around, it’s adhering too rigidly to my schedule, and sacrificing the things that help me build a balanced, sustainable tempo of work when I feel like I’m falling behind.
On the one hand, this year is all about experimentation: rapidly iterating on a bunch of different ideas until I key in on one that works. Trying out different ideas, ways of working, variations on themes; anything that brings me closer to my goal of establishing a self-sustaining income counts! On the face of it, that amount of freedom seems amazing, but at least for me, it’s also kinda scary. If you’ve ever done anything creative, you know one of the most difficult parts of the act of creation is simply starting. Taking that big empty page in front of you and filling it in with something, anything, is so hard when you don’t have any boundaries to work within.
We’ve all got inherently limited minds, and they are thirsty for structure. Some of this is attributable to growing up in the wake of the Second Industrial Revolution: the shift starts at 9 AM sharp, you work until your scheduled break, you must produce units of work on a fixed schedule and quota.
But a lot of structure comes from the natural world too: day and night, spring and summer, monsoon and dry season. Hibernating in the winter, migrating in the spring, stockpiling acorns in the fall. These things all follow cycles and patterns, and the big biological clocks housing these modern minds of ours are still synchronized with those cycles, even if we don’t have eyes to see it most of the time.
At any rate, it all speaks to an inherent need to chop the world up into comprehensible pieces with some kind of regularity in form or tempo so we can understand the world around us and make decisions and predictions about it. Whether any of this is “right” or “good” to do, or whether any of these divisions are valid at all, is another question entirely, but we seem to desire it on some level.
Hell, this need for structure and order is what draws me to Clojure and LISP in the first place: the language places very simple but very strict limitations on what can be done. Thou shall not mutate a variable, thou shalt perform sequential transformations on data with a threading macro, thou shalt not worry about types too much. And yet, this rigidity has the confounding effect of giving you more freedom than you would have had otherwise. By restricting your possibilities to a handful of broad but nevertheless restrictive paths, you avoid the completely stupid, completely avoidable, almost inevitable pitfalls that imperative-style coding puts in front of you.
And so, knowing all of this, and appreciating the benefits that structure can bring, we come to the subject of my schedule. I’ve been attempting to craft a weekly schedule which achieves balance. My dream day would go something like this:
Sounds great right? The right amount of structure coupled with plenty of freedom to flex, stretch and explore what’s possible.
But god damn man, when push comes to shove, it all comes back to the bottom line in the end. Every time I hit a ditch in the road, and I’m presented with the choice of getting out of the car and muscling the thing out of the ditch (read: fixing a bug, slogging through an unexpectedly complicated feature) or just sitting down on the side of the road and trusting that help will come soon (read: giving it a rest when things get too hard, taking care of my mind and body), I take the slog.
Every single time.
Time is money, production must stay constant, clear and obvious forward motion at all costs. What, you want to go for a run? That’s insane! Think of all the work you could be getting done instead! Go for a run when you’ve started to earn some money!
I really enjoy watercolor painting videos. Really, anything that shows the process of creation draws me in like a moth to a porch light, but watercolor in particular blows my mind. Harumichi Shibasaki is the kindly Japanese grandpa we all need, and James Gurney is great too. When I was up late on Wednesday night, unable to sleep, mind wracked with thoughts of all of the stuff I wanted to do and how little of it I got done, I watched this. This line in particular is exactly what I needed in that moment:
Most of my paintings go through an “ugly stage,” where it seems impossible to see my way clear to the finish. You may imagine from my calm voice that I know exactly where I’m going, but in truth I feel like a pilot lifting the aircraft through a layer of blind clouds. At this stage I have to fly on instruments with faith that the clear air is waiting for me on the other side.
I started to reevaluate this week in terms of it being my “ugly stage”: the moment where I’ve gotten fairly deep into the startup, and where the complexity really starts to show its head. After graduating from implementing the basics, you start to get deeper into the complexity of what you’re attempting to undertake, and your past decisions start to force your hand in sometimes uncomfortable ways. The idea that you can’t do it or won’t finish starts to creep in, or that it won’t be good enough. If only you put in another half hour here or there, it’d all come together.
And what are my “instruments” in this example? Sleeping well. Eating well. Taking breaks. Taking time to reflect on what’s happening. Acknowledging when something isn’t working and making thoughtful course corrections. Above all: exercise. It’s like knocking out beams in the first floor of your skyscraper when you let exercise lapse. Sure, you could get away with it a few times, but how many exactly? How soon until the whole thing collapses?
When things get weird, disregarding your instruments and shoving the throttle forward is a really easy choice to make. It is so easy to mistrust and disregard the things keeping the plane steady and to try to pull through the cloudy parts. It’s easy to imagine that there’s nothing except for you controling the plane. But all of that eminently ignorable stuff behind the instrument panel is really the stuff that’s doing the flying. The very things which are so easy to write off as a “waste of time” or a “distraction” from the Work are the very things that enable the Work to continue apace.
I need my instruments. However loathe I am to give up my sense of autonomy to them, my feelings of invincibility, the sense that I know what to do during the weird, ugly parts of this flight, I need them.
Clojure comes with many useful functions that are a great learning resource for those who are just starting to explore the language.
Description of the problem Today, I am going to write a clojure function that solves vertex cover...
If we would like to comment out a line of the code we simply use a semicolon at the beginning. To comment out a block of the code we use hash underscore #_ (reader macro).