Metosin’s Favorite Editors

Introduction

At Metosin we have a diverse group of Clojurians. Some of them have strong opinions regarding their favorite IDE, some of them are less passionate. It's a joy to read the Metosin slack regarding these discussions. One day I realized that it might be a nice idea to let our blog readers and library users to have a peak inside our Metosin office in the form of informal interviews regarding various topics. The first topic I chose is: What is your favourite IDE / editor setup, and why. So let's start our walk on the Metosin virtual office and interview Metosinians! When reading further remember that these guys really like to pull one's leg - interpretations are your own responsibility.

Matti Hänninen - "I have always gravitated back to Emacs"

Kari: So Matti, which editors have you been using in the Lisp world?

Matti: For Clojure mostly Emacs. In the early days I used to use some version of SLIME or the Superior Lisp Interaction Mode for Emacs. That was natural as I was using it for Common Lisp too, or primarily, and it happened to have some kind of support for Clojure. From there I moved onto using nrepl.el that later became CIDER which I'm using still. I've tried to use other editors like Vim, Kakoune, and VSCode for programming lisps but in the end I have always gravitated back to Emacs. It just seems to work for me ... at least as far as lisps are concerned.

Kari: How about visual aid while editing, font, ligature, theme? Any favorites in the Emacs world?

Matti: I use JuliaMono throughout my system and I'm pretty happy with it. No ligatures as they are more of a distraction and look really bad when they are off. Likewise heavy syntax coloring is a distraction for me. I use my custom color theme that is very monochromatic and distinguishes just between comments, literals, and the rest of the code; barely.

Kari: And then the standard question: paredit or parinfer?

Matti: Paredit.

Kari: And one extra question for you, Matti. I remember that you are also a keyboard enthusiastic like me. Have you already found a perfect keyboard?

Matti: No but successors tend to be improvements over their predecessors. Currently I use a external 40% Planck keyboard with highly customized firmware as my daily driver.

Kimmo Koskinen - "Mostly Emacs, accompanied by OSX native applications"

 

Kimmo: I use mostly Emacs, but Fork.app for patch selection & commits, push/pull/rebase happen in the terminal and in Emacs. Long time Postico user for PostgreSQL work, Keynote for presentations. During my time at University, I was avid Linux & Emacs fan (Latex/Beamer yay), but for the past ~15 years, I've been a Mac user.

Kari: Have you tried IntelliJ IDEA or have you always been an Emacs user?

Kimmo: Eclipse was the goto-tool during Java era, IntelliJ Idea fly by but didn't really stick with me, Clojure brought me back to Emacs.

Kari: How about the editor support with Emacs+Cider, are you happy about that?

Kimmo: Current state is pretty good, getting a ClojureScript repl running in Cider isn't a hurdle any more.

Kari: How about visual aids like font, highlighting, ligature, theme etc. Are they important to you?

Kimmo: I haven't gone into font tuning yet, but have used the dark deeper-blue theme in Emacs, finding that dark background suits me best.

Kari: And finally: paredit or parinfer?

Kimmo: Paredit, of course (smiling).

Pauli Jaakkola - "I'm a bit clumsy with my tools"

Kari: Did you say that you are a bit clumsy with your tools? Damn man, you are considered as one of the Clojure wizards of this company. I'd really like to hear what you mean with that.

Pauli: Considered by whom (smiling)? Anyway, I just want to do programming (and Computer Science-y things). Editors, builds and devops should just work and get out of the way. Often that seems like a distant dream in these days of Atom, Webpack and Kubernetes (sigh). I use Cursive for Clojure at work because it works out of the box and jumping to documentation and definitions is very convenient. For everything else I use (Neo)vim... and Make (laughing). Stuff that is always available, barely changes and does not make your computer slow. I guess I had my fill of trying out different options and configurations with Linux distro- and DE-hopping. Although I am still on Arch Linux with i3 (Sway at home, yay!) instead of e.g. Ubuntu or Fedora.

Kari: Interesting. I guess various visual aids like font, highlighting, ligature, theme etc are not that important to you?

Pauli: I use Solarized Dark whenever possible. High contrast in general and white backgrounds in particular are painful to look at. The dark mode trend was way overdue; as a Linux user I always felt that everything should be themeable, at the very least to avoid snow blindness. The usual token type -based syntax highlighting only serves to anchor the eyes and does not get any better than rainbow parens. If Lisp is "oatmeal with fingernail clippings" to Larry Wall, now I have a blueberry smoothie with sprinkles. Much more appetizing, but how much more sense does it actually make? I suppose I should try semantic highlighting some time, and learn more Paredit than just slurp, barf and raise. In any case the highlighting of matching parens or variable uses/definitions is more useful than token coloring. In DrRacket it even works with custom macros! I wonder if that would be possible with Clojure (Colin, are you reading this?). And no, I don't bother with fonts or ligatures. Monospace fonts are necessary to follow indentation but unavoidably ugly. Ligatures seem like operator precedence to me; slightly prettier but can't possibly scale. (What is the relative precedence of $, >>= and <|> in Haskell? Beats me. In Lisp we just say screw it, let's not bother with infix operators at all.)

Kari: Ok. So, it's paredit then. Have you ever tried parinfer?

Pauli: No. I probably should, especially since it comes with Cursive now. Or maybe forget about both of them and spend my evenings practicing Vim instead of Bach's Inventions?

Miikka Koskinen - "Still waiting for a perfect editor configuration"

Kari: You gave an interesting comment when I announced this idea that I'd like to write a blog post regarding which IDEs / editors Metosinias use. You said: "Maybe I finally configure my editor configuration".

Miikka: I use Spacemacs (for small Clojure projects and everything else), IntelliJ IDEA (for big Clojure projects and Scala) and VS Code (for JavaScript) at work. Jumping between the three editors means that I haven't learned any of them deeply. Spacemacs stopped getting releases a couple of years ago, although its develop branch is still actively developed. I never got around to switching the develop branch, so now my Spacemacs is outdated and slightly broken. This setup is stable, but otherwise I would not recommend it to anyone. It would be a good idea to focus on a single editor. I hope that the blog post gives me some ideas on which way to go.

Kari: How about visual aids: font, highlighting, ligature, theme? Any preferences?

Miikka: I just use the default settings of the editors.

Kari: And finally: paredit or parinfer?

Miikka: Paredit. Parinfer had not been invented yet when I started learning Clojure, so the choice was simple. By the time it popped up, I had gotten very comfortable with Paredit, so I have never even tried Parinfer.

Tommi Reiman - "I broke my Mac laptop keyboard by..."

Kari: First I need to ask you about that Mac keyboard (laughing).

Tommi: I broke my Mac laptop keyboard by resetting my Integrant system too many times.

Kari: I remember that story. Would you like to share it with our readers?

Tommi: Having the Butterfly keyboard on my Macbook, which really sucks, will send it for repairs on winter holiday (we have that in Finland). I'm using Save File + (integrant) reset workflow zillion times a day and because of that keys command, s and l are totally busted. Having shortcuts for mostly everything.

Kari: So, you are an IntelliJ IDEA / Cursive user. Have you ever used or considered Emacs? Is there something special you like about Cursive?

Tommi: I started with Emacs back in the 90's, used it for everything possible. Switched to Java IDEs in the early 2000 and haven't gone back, too busy re-learning to use it. Cursive is great for Clojure, a bit laggy, but gets the thing done, and I really love the deps, java & testing support.

Kari: How about Integrant, I guess you use it quite a lot as part of your Clojure programming workflow? Would you like to tell something about it?

Tommi: I think Integrant is IoC done (mostly) right. Also, while developing, the repl gets dirty quickly and being able to reset the whole system is a killer feature. And we are doing that in the clouds too: having an Nrepl open in an app deployed in Kubernetes cluster and being able to play with the app from local IDE is kinda mind-bending compared to traditional deploy cycles. Just have to remember to use Integrant's suspend+resume on the nrepl server or you get thrown out on reset.

Kari: How about the font? I remember seeing in some pair programming session that you prefer Ligature symbols e.g. in threading macros?

Tommi: The most common question when tweeting with screenshots is: "what's the font, please?" (smiling). I use Liga Menlo - Menlo transformed via Ligaturizer. When Sam Aaron was at first ClojuTre in 2012, he had an inspiring talk about tuning your tooling, "Build your own lightsaber". We are using our computers a lot and aesthetics matter too: fonts, colors etc.

Kari: Paredit or parinfer?

Tommi: Paredit. (You can see an example of Tommi's editor theme here )

Juho Teperi - "I use Vim, because it is Vim"

 

Juho: I use Vim, because it is Vim. Everything else is shit.

Kari: Heh, a bit harsh talk (coughing). Would you like to clarify that statement a bit?

Juho: No.

Kari: Ok. Paredit or parinfer?

Juho: Not quite either. I use vim-sexp and vim-sexp mappings for regular people. It is a plugin for "precision editing S-expressions", so quite similar to Paredit, but it is built on top of Vim modal editing and text objects and commands, so it works a bit differently. It doesn't affect the insert mode (that's where you can write text) at all. Instead, it provides some new text objects and commands which are used in normal mode (the mode where you don't write text, but just use commands). New text objects, like one current form, which work together with Vim text commands, like built-in delete or change commands, or the command to send the text to REPL. It also provides some new commands like moving parenthesis, and these also work with all Text objects, not just those added by Vim-sexp.

Jarppe Länsiö - "The rest is, as they say, history"

Kari: I remember hearing that it was you who initially suggested that Metosin starts using Clojure?

Jarppe: Yeah, that’s true. We all - me, Tommi and Mikko - had been programming in Java for a long time. In one of my projects at that time we used immutable data structures a lot. At the same time I experimented with Scala and Clojure on my hobby projects. Working with immutable data pipelines felt really natural and fun in Clojure, and I kind of fell in love with it. Then Metosin got a new gig and I proposed we should try Clojure. The rest is, as they say, history.

Kari: A legendary history, indeed. Which IDE / editor did you use in those early days?

Jarppe: I can’t remember what I used in the early days, but pretty soon I started using Eclipse with Counter Clockwise plugin. It was a really good plugin, and since most Java programmers were familiar with Eclipse it provided a gentle path to try Clojure. I remember that I actually made a video training material regarding how to start programming Clojure and use Counter Clockwise plugin.

Kari: How about nowadays? Are you still using Counter Clockwise?

Jarppe: The last 5 years or so I have been using IntelliJ with Cursive. Cursive is just amazing.

Kari: Are visual aids important to you, like font, highlighting, ligature, theme?

Jarppe: Yeah, I’m very used to syntax highlighting and ligatures, so when ever I see code without them, I feel like it takes a little bit more cognitive effort to understand it.

Kari: Paredit or parinfer?

Jarppe: Paredit.

Kalle Lehikoinen - "Emacs, just Emacs"

Kari: Kalle, which IDE / Editor are you using?

Kalle: Emacs.

Kari: Have you always used Emacs?

Kalle: Always used emacs.

Kari: What are the most important reasons for you to use Emacs?

Kalle: I’ve always used it, so it’s what I know.

Kari: Paredit or parinfer?

Kalle: Paredit.

Tuukka Kataja - "I do not have any interest in participating in editor (or language) battles"

Kari: Tuukka, which IDE / editor are you using?

Tuukka: Recently, I have been programming mostly in Python and I use VSCode. I also had little experimentation with Clojure using the Calva extension about half a year ago. Before that I mostly used IntelliJ IDEA + Cursive which was really great for Clojure. I especially liked the refactoring functionalities in Cursive. DataGrip is another tool from JetBrains that I use for database-related tasks (mostly working on queries against Oracle/Postgres).

Kari: Why did you move to VSCode?

Tuukka: I switched to VSCode about a year and a half ago after moving to a Python-project. The main attraction was the large ecosystem of available extensions. Probably I will try to use VSCode + Calva when programming in Clojure again. I have a gut feeling that VSCode is pretty popular nowadays and a good VSCode extension probably helps if someone new to Clojure wants to play with it. One VScode feature that I am now relying on is the SSH remote development which seems to be functioning pretty well with my development VMs.

Kari: Do you use some special mode like emacs or vim?

Tuukka: I have Vim extension enabled on DataGrip and NeoVim integration on VScode. To be honest, I have been pretty lazy learning the Vim tricks until now. Actually, I installed the VScode NeoVim integration to solve the horrible VSCode input lag I had with default settings. Then little by little, I started to learn the Vim stuff. I do not have any interest in participating in editor (or language) battles. Someone might like and get the job done with PHP and Notepad++ and I am quite okay with it.

Kari: Ok. Are visual aids important to you, like font, highlighting, ligature, theme?

Tuukka: Yes I think they are pretty important. As a programmer you read and write lots of code so it makes sense to improve the aesthetics and readability with font selection, theming and so on. Inversely, bad theming e.g. with incomplete language keyword coloring can make things harder to read. Many years ago, I had an Ubuntu dev machine where IntelliJ IDEA font rendering was just horrible. That combined with the default Darcula theme must have been visually the worst experience ever. There is one pitfall however: with beautiful font rendering etc. you can fool yourself that the code you wrote is beautiful. Like with articles written in LaTeX: even mediocre content seems better on pretty LaTeX output (smiling).

Kari: Heh, ok (smiling). Final question: paredit or parinfer?

Tuukka: Paredit. I did try parinfer but gave up since I already had learned the most typical paredit commands to feel comfortable enough with it on daily work.

Jari Vasell - "Good visualization of the data structures in the IDE would really help..."

Kari: So Jari, Which IDE / Editor are you using?

Jari: I'm using IntelliJ. Initially the main reason for choosing it was that also Jarppe was using it, and it must be good (laughing). Eventually I've realized that built-in functionalities are really useful as my main job is not software development, and I might need to do various stuff only seldom, i.e. getting the IDE to help me as much as possible is a joy.

Kari: What are the most important reasons for using IntelliJ?

Jari: As I said earlier, for me the most important thing is time saving stuff, in the sense: how to do this correctly. I think exploring structure in data is something that I've found really cumbersome. This is something that might be related both to Clojure and Python and I have not found a really good way to explore data structures outside of coding helper functions. For example: You have a table with tens of columns and thousands of lines. The data has been originally read from Excel, but then the data has been modified in code. Then you encounter stuff that does not work (error in data?) and you would quickly want to find out where the problem lies. Good visualization of the data structures in the IDE would really help. Writing helpers to debug and print code in the console is the current way (for many developers). But how to do this better? Any comments from my colleagues are welcome.

Anssi Heiska - "An eclipse fan boy"

Kari: So Anssi, pick your poison: Emacs or IntelliJ?

Anssi: I have used IntelliJ IDEA and Sublime Text also. But I must confess that I'm really an Eclipse fan boy.

Kari: Eclipse? I believe you belong to the minority then, with the Vim guys.

Anssi: Eclipse is awesome for Java, but sadly many plugins are nowadays abandoned. On the other hand many things work better in Eclipse than in IntelliJ IDEA.

Kari: Such as?

Anssi: Eclipse hot keys! Oh, I almost forgot: I did my Master's thesis about Eclipse plugin!

Kari: That makes you an Eclipse master! How about fonts, do you have a strong preference regarding fonts?

Anssi: Nope, as long as there are ligatures (laughing)! Ok, I have been using JetBrains Mono but also Fira Code.

Kari: And finally: Paredit or parinfer?

Anssi: I don't have a strong opinion about that.

Miika Pirkkanen - "My IntelliJ and Cursive both are builds from early 2017"

Kari: Miika, IntelliJ or Emacs?

Miika: I use mainly IntelliJ Idea with Cursive. I try to avoid spending my time with configuring and updating things for no reason as with every update things can potentially break (or even worse, change). IntelliJ and Cursive simply work out of the box and do not require endless hours of editing config files just to get syntax highlighting to work. Fragile updates are even greater risk when a piece of software is dependent on another as it is the case with IDEs and their plugins. I mean, if it ain't broken don't fix it. In fact I have "fixed" my current setup only once so my IntelliJ and Cursive both are builds from early 2017. I only dared to update for some cool Cursive feature that I wanted really badly. I can't remember which one but it might have been the "jump to event definition" feature for cljs namespaces. I'm afraid the changelog only goes back to 2019 so there is no way to tell for sure. Some new Cursive features are quite tempting, though, maybe enough so to get me to update my tool chain again.

Kari: Unbelievable! (looking Miika's IntelliJ IDEA About window: "IntelliJ IDEA 2017"). I always use the latest IntelliJ version. Would you like to tell our readers something about your visual aids, theme etc.?

Miika: The theme I use is IntelliJ's Darcula. It looks nice and is easy on the eyes. As for version control features I mainly use git via its command line interface but IntelliJ has a great tool for resolving merge conflicts that I find useful.

Kari: How about the classic Clojure editor question: paredit or parinfer?

Miika: I like paredit as parinfer always seems to mess up my namespace when I give it a try. Things break and I don't like it so I fall back to paredit. Propably an user error but I don't see the benefit.

Ilmo Raunio - "Jack of all editors"

Kari: Ilmo, which editor or IDE are you using?

Ilmo: I use a few different editors: Cursive/IntelliJ, Vim, VS Code, and Sublime Text 2/3 (ST).

Kari: Interesting, you have quite a few editors in your toolbox, I guess that makes you a Jack of all editors (laughing).

Ilmo: Right. I originally started out my career with ST, editing Python code with it and also using it with my Clojure hobby projects. I later exited the Python industry for some time and found out about Cursive/IntelliJ with the permissive non-commercial license. When I started working in a Clojure customer project, I was already using Cursive and continued using it (with a commercial license). Somewhere along there, I started phasing out my ST habits to learn more Vim. Usually because terminals would have some combination of editors to get by and Vi/Vim were one of them. I’ve used Vim for editing IaC files and some occasional Python scripts. VS Code is a recent editor during this year for React/TypeScript frontend work that I’ve been doing in my current customer project.

Kari: What are the most important reasons for using those editors?

Ilmo: Well, for Cursive/IntelliJ the most important reasons are code navigation, refactoring, structural editing, and REPL-integrated development. I also occasionally use the Cursive debugger. Multi-caret support for the occasional “non-refactorable” string replacements is also nice, as is ability to use library code and navigate Java classes.

Kari: How about other editors, Vim, VS Code and ST?

Ilmo: For Vim, portability and the fact that you can find Vim in most *nix machines. Vim is a good general editor. I used VS Code when learning the TypeScript ecosystem - didn’t want too many distractions, and it was generally well understood and supported by the customer project teams. ST has a good multi-caret support, and is also a good general editor.

Kari: If you are using the standalone Vim, do you use also the Vim mode in IntelliJ?

Ilmo: Nope. I try to keep my setups somewhat vanilla because it empowers others to use my setup during pair coding sessions (once we can all huddle together without fear of the ’Rona).

Kari: Are you mostly happy with IntelliJ or is there some feature that you especially miss?

Ilmo: At least more integrated formatter support, shortcut functions that you could apply to code and receive code as output (if this is possible already, then I just haven’t figured it out yet), and reorder namespace imports in lexicographic (ascending) order.

Kari: Since you already know quite a few editors, is there one more you'd like to learn?

Ilmo: Well, Emacs. Maybe also some Python IDE.

Kari: You don't like IntelliJ as a Python IDE? I thought it has a pretty good Python support?

Ilmo: I just haven’t tried hard enough to really form an opinion. I tried importing a Python project once as a module to a Clojure project, but somehow I never got it working. For me, using and editing Python from the command line has always been a thing, but at times I’ve felt the pain of lacking an IDE not least when I had to maintain a largeish Python codebase.

Kari: One last question: paredit or parinfer?

Ilmo: Paredit. I think my editor picked a default for me and it was maybe paredit at the time? I have no strong opinions for either, and I would like to try parinfer during some quieter learning time.

Lassi Immonen - "Vim wannabe but using IntelliJ"

Kari: Lassi, which IDE / Editor are you using?

Lassi: IntelliJ (Ultimate, Rider, Android Studio). IDEA has been my main IDE for quite a many years now.

Kari: Interesting. For what kind of work are you using Rider and Android Studio?

Lassi: I tried configuring plain IDEA for Android development, but found out some quirks and settled to Android Studio, it’s IDEA under the hood anyways, given I have plenty enough memory to spend. Rider is for C# development when working on Unity projects.

Kari: What are the most important reasons to stick with the IntelliJ family?

Lassi: IDEA just (mostly) works and familiarity gives speed. There is a big array of extra functionality through plugins, and the code completion, code navigation, and refactoring experience is superb.

Kari: Is there some specific keymap like Emacs or Vim mode you use with IDEA?

Lassi: I’m doing casual quick editing with Vim and I do like it as an editor. But with IntelliJ I'm not using the Vim mode. I have tried it a couple of times but seems too big a jump to get it really working, though entertain the idea from time to time on delving into it. I kind of secretly harbour the idea that I would become so fluent in Vim that I would really do most of my coding with it. I practice some Vim trickery from time to time, but then the reality kicks in and I’m back in my safe haven with IDEA (laughing).

Kari: Ok. If you really consider start using Vim as your Clojure editor I recommend you contact Juho - he's pretty fluent with Vim and uses it as his main Clojure environment. So, the last question: Paredit or parinfer?

Lassi: Parinfer! I love it!

Miika Pihlaja - "I think in Typora"

Kari: Miika, your favorite editor or IDE?

Miika: I started writing Clojure with Emacs, must have been around 2011. I’m not actually sure which one was my main editor back then Vim or Emacs, but somehow I ended up using Emacs with Vim keybindings, the Evil-mode. It’s also unclear why I swapped to IntelliJ/Cursive. Probably had something to do with the big hairball Emacs configuration running totally out of control and supposedly easier debugging experience on the Cursive side that I actually never really remember using. At some point I tried to use IntelliJ IDEA with Vim mode too - didn’t like it, and surely enough, don’t remember exactly why. That’s a lot of unknowns.

Kari: But you are still using IntelliJ IDEA? Which mode are you then using?

Miika: Yeah, I’m using Cursive with pretty much vanilla keybindings, usually a bit older vintage, maybe a year or so, so things are not broken. Only specialty is some templates for datomic queries and maybe other project specific stuff. I’m certainly less invested in configuring editor than I used to be. Maybe it has to do with the fact that I use much more time on reading code, in Excel and in the Markdown editor writing stuff up (for that I use Typora - not only use, I think in Typora) than writing code. Ah, that reminds me, Cursive has very nice find usages for Clojure functions. And even for Re-frame subs and handlers too. I think that was one of the reasons I got stuck using Cursive, I just couldn’t get that working nicely in Emacs/Cider back in the day. I’m sure it was/is possible, but there we go.

Kari: Final question: Paredit or parinfer?

Miika: To be completely honest, I had to google up what's the difference. Paredit.

Teemu Kaukoranta - "It's hard to justify using Emacs, so I use Cursive"

Kari: Teemu, you are the newest member in the Metosin family. What is your favorite Clojure editor or IDE?

Teemu: I use IntelliJ IDEA with Cursive, mainly because it's easy to get working and it comes with nice features out-of-the-box. I use the different refactoring tools a lot, e.g. renaming vars and keywords(!). I've thought about learning Emacs many times, and even started down that road a couple of times, but it just feels like I would need to invest so much time into it until it reaches a similar level of comfort and power, that it's hard to justify. I'm ready to believe that in the end I could get more out of Emacs than out of my current setup, but on the other hand IDEA+Cursive probably does offer some features that are not available, or just not as good, in Emacs.

Kari: How about theme, font, and ligature symbols, any favorites?

Teemu: I have to say, I've never understood the point of ligatures (smiling). Honestly I think they just make the code harder to read. In e.g. Javascript the difference between == and === is pretty important, and it's really easy to distinguish != from == thanks to the exclamation mark. I enjoy tuning the aesthetics of my editor too, but to me ligatures have an adverse effect on readability. Maybe you get used to them after a while though. I've never gotten into fonts. I like Solarized Dark, though I really only use it in my terminal. I use the Dracula theme in IDEA, because the last time I tried using alternative themes, not all UI elements got the new theme.

Kari: Paredit or parinfer?

Teemu: I use paredit (or well, maybe a subset of it). I learned to use it quite quickly after starting to use Clojure five years ago. Parinfer wasn't a thing back then, I think? Not sure which one I would recommend to new clojurists right now.

Kari Marttila - "Clojurian with a multiple personality disorder"

Kari: So, you are the last one?

Kari: Yep.

Kari: I understood you are an IntelliJ / Cursive user?

Kari: Yep. I started my career programming C, C++ and Java with Emacs. Later on I switched to Eclipse. Then IntelliJ IDEA with Emacs mode. I really like IntelliJ IDEA / Cursive but last winter I also made an experiment: can I configure Emacs to have as much look-and-feel as my IntelliJ IDEA / Cursive setup. I was pretty successful - all my favourite hotkeys now work exactly the same way in both editors, and both editors have the Leuven theme which is kind of nice to my eyes.

Kari: Paredit or parinfer?

Kari: Paredit, no question about it. I have configured nice keymappings for slurping and barfing - couldn't live without it.

Kari: Anything else?

Kari: I really like Cursive but I guess I could live with Emacs also. But I'd like to mention that there are two things I won't change for the rest of my career.

Kari: ... and they are?

Kari: Paredit and my Dygma Raise keyboard.

Kari: Ok. One final question. Is it really true what they say about you that you discuss with yourself in the Metosin slack?

Kari: Where did you hear that? Do they think that I'm some sort of a Clojurian with a multiple personality disorder?

Permalink

Fancy form posts in Biff

One of Biff's core features is authorization rules (patterned after Firebase Security Rules) which allow you to send transactions to the database[1] without going through a custom endpoint. For each type of document in your database, you define rules which specify the shape of that document and who is allowed to write to it. Then you can submit arbitrary transactions from the front end, and Biff will make sure they satisfy the rules.

Right now, Biff only includes this feature in its SPA project template. You write ClojureScript code which sends transactions as EDN over a websocket. However, since October, Findka is no longer a SPA. I pivoted and built a new, simpler website from scratch, and a SPA would've been overkill. So now I'm using regular form posts for write operations, not websocket events. Despite that, it would still be nice to use authorization rules and a single endpoint to handle the posts instead of having a proliferation of endpoints.

For reference, here's how you would normally send a Biff transaction from the front end:

(defn set-display-name [user-id display-name]
  (send-websocket-event
    [:biff/tx
     {[:users {:user/id user-id}]
      {:db/update true
       :display-name display-name}}]))

The ident, [:users {:user/id user-id}], means that we're writing a document of type :users and the document ID is {:user/id user-id}. We set the :display-name key on that document, and :db/update true means we merge that into the existing document instead of overwriting it completely. See the docs for more info.

Even though Biff doesn't come with an endpoint for receiving a transaction like this via form post, it's pretty easy to set one up:

(defn form-data->tx [params] ...)

(defn form-data->redirect [params] ...)

(defn submit-tx
  [{:keys [biff/node biff/base-url params]
    :as sys}]
  (let [biff-tx (form-data->tx params)
        crux-tx (biff.crux/authorize-tx
                  (assoc sys :tx biff-tx))]
    (if (biff.util/anomaly? crux-tx)
      (pprint [:transaction-rejected tx])
      (crux/await-tx node
        (crux/submit-tx node crux-tx))))
  {:status 302
   :headers/Location
   (str base-url (form-data->redirect params))})

(def routes
  [["/api/tx" {:middleware [wrap-anti-forgery]
               :post submit-tx
               :name ::submit-tx}]])

First, submit-tx takes the form parameters and converts them to Biff's transaction format. biff.crux/authorize-tx checks if the transaction conforms to your authorization rules and returns a Crux transaction if so. After you submit that transaction, you'll need to redirect the user back to whatever page they were on.

form-data->tx is where things get messy, so let's get form-data->redirect out of the way first. Here's a form from Findka's main page:

[:form {:method "post"
        :action "/api/tx"}
 [:input {:type "hidden"
          :name "redirect"
          :value "home"}]
 ; ...
 ]

My form-data->redirect looks kind of like this:

(defn form-data->redirect [{:keys [redirect]}]
  (case redirect
    "home" "/home"
    ...))

(Be sure to read the relevant OWASP page). I also have some logic for setting query parameters, which are used to display messages like "URL submitted successfully", etc.

Before we go on to form-data->tx, let me show you the complete form:

(let [uid ...
      n-links ...
      n-days ...]
  [:form {:method "post"
          :action "/api/tx"}
   ; anti-forgery token goes here
   (for [[k v] {"redirect" "home"
                "docs[0][table]" "users"
                "docs[0][id]" (pr-str {:user/id uid})
                "docs[0][update]" "true"}]
     [:input {:type "hidden"
              :name k
              :value v}])
   [:div "We'll send you "
    [:select {:name "docs[0][n-links]"}
     (for [i (range 1 11)]
       [:option {:value i
                 :selected (when (= i n-links)
                             "selected")}
        i])]
    " links every "
    [:select {:name "docs[0][n-days]"}
     (for [i (range 1 8)]
       [:option {:value i
                 :selected (when (= i n-days)
                             "selected")}
        i])]
    " day(s)."]
   [:button {:type "submit"} "Update"]])

So we need to map this form data to Biff's transaction format. Again, a Biff transaction is a collection of documents (and their idents). This form submits a transaction for only one document, so we can store it under "docs[0]".

And now we come to the messy part: we must convert the form fields' values from strings to their respective data types. I specify a coercion function for each key. For example, "docs[0][table]" is coerced with keyword and "docs[0][id]" is coerced with clojure.edn/read-string. I have a k->coerce-fn map which I use for the rest of the fields. Whenever I add a new form, I make sure to add new coercion functions as needed.

(def k->coerce-fn
  {:n-links #(Long/parseLong %)
   :n-days #(Long/parseLong %)
   :user edn/read-string
   ;...
   })

See the appendix for the implementation of form-data->tx.

This solution is good enough for my purposes, but it doesn't feel great. Hard-coded fields (like hidden form fields or select fields) are fine, since you can always just set the value to an EDN string. Although I'm using Long/parseLong for :n-links and :n-days above, I guess I could use clojure.edn/read-string instead, for consistency's sake. But text fields need to be left as-is, and check boxes need special treatment:

:some-checkbox-key #(or (= % "on") (= (last %) "on"))

Maybe that's not so horrible. But it might be nice to add some metadata via hidden fields that specify the types of any non-EDN fields, so you wouldn't have to maintain a global k->coerce-fn map.

I'll probably add this into Biff at some point. I like to test out new features in Findka for a while first. Let me know if you've got any suggestions.

Appendix

(defn form-data->tx [{:keys [params/docs]}]
  (for [{:keys [table id] :as doc} (vals docs)]
    [(cond-> [(keyword table)]
       id (conj (edn/read-string id)))
     (when-not (:delete doc)
       (->> (dissoc doc :table :id)
         (map (fn [[k v]]
                [(case k
                   :update :db/update
                   :merge :db/merge
                   k)
                 ((get k->coerce-fn k identity) v)]))
         (into {})))]))

Notes

[1] They also let you query the database from the front end, but that's not relevant to this post.

Permalink

Middle Business Analyst

We are looking for a Middle Business Analyst to join our team. The product is an AWS hosted multi-module payment and analytical platform for the healthcare services, written in Clojure/Golang/Python language stack. The product encompasses a few applications for customer journeys (web, mobile), data science/data analytics platform, multiple integrations with federal and governmental resources, and complex micro-service architecture. The product’s domain is Healthcare/Fintech, hence all the compliances and accent on security and high-performance.

Requirements:

• Understanding of SDLC 
• At least basics in business analysis and requirements management 
• Experience with writing requirements, e.g. stories, legends, high-level business requirements 
• Experience with selecting vendors and/or with researching market for a certain business need 
• Ability to work in complex and/or stressful situations 
• Advanced communication skills 
• English-Advanced level 

Nice to have:

• Experience with design tools, e.g. Figma as a smart user 
• Technical or fintech/healthcare background 

Responsibilities:

• Own the sprint backlog readiness 
• Work on the Product roadmap 
• Work with US Product team on epics and stories 
• When needed, research the market for ready solutions or for ideas for the product 
• Track and refine the product backlog 
• Participate in design discussions 

We offer friendly working conditions with competitive compensation and benefits including:

• Comfortable working environment
• Friendly team and management
• Competitive salary
• Free English classes
• Regular performance-based compensation review
• Flexible working hours
• 100% paid vacation, 4 weeks per year
• 100% paid sick-leaves
• Corporate and team building events

Apply Now 

Permalink

Full Stack Engineer - Remote

Permalink

Planning my career development for 2021

If you are interested in reading this article in Spanish 🇪🇸, check out my blog:
The Developer's Dungeon

Hey guys, 2021 is finally here!. It has been a hard year but now so it is time to plan everything again, to write down on paper what I want to learn or do this year.

If you been following my blog posts for some time you will know that I am a big fan of setting goals, planning, tracking, and of course DOING.

Last year I focused on too many things and while I accomplished quite a few of the things I proposed to myself, I also noticed the pressure of not reaching what I was expecting was pushing me back. This year I am gonna go a little lighter, in the number of things but not on the effort, let's call it just having a "better" direction.

Let's start already, shall we?

1. Be comfortable writing at least in one functional programming language

Last year I spent quite a lot messing around, going from book to book, looking up languages, talking to people, doing small tests on different environments. For this year I have made the decision to focus on learning Haskell, at least first. This is gonna be my initial curriculum:

• Read Haskell Book.
• Read a Type of programming.
• Read Learn you a Haskell for great good.
• Read Category Theory for programmers.
• Do at least one big personal project with Haskell.
• Explore other languages like Elm, PureScript, F#, Clojure.
• Read Functional programming for the object-oriented programmer.
• Re-Read Structure and interpretation of computer programs.

By separating a big goal into smaller concise tasks I don't need to worry about what to do next, I just follow the plan, if something is not going right I solve the issue and continue.

2. Get better at software architecture and computer science

Last year I didn't focus on this at all, last year I had this idea that I wanted to learn more frontend, so I switched positions at my job and became a frontend developer. That taught me that I not really that interested in the frontend after all haha.

• Read The imposter’s handbook series.
• Read and study Clean Architecture.
• Read and study Domain Driven Design: Tackling Complexity in the heart of software.
• Read and study Implementing Domain-Driven Design.
• Take on a project at work where I can design a system from scratch.
• Create a personal project using the concepts learned, while diagraming the architecture.
• Participate in architecture discussions at my new position to learn about distributed systems.

I started the year with a new position, a new job as a Senior Software Developer in a place with some complex architectural decisions and a lot of cools systems to play with. I am gonna use this opportunity to learn as much as I can from my new team members and at the same time try to boost my own knowledge on the subject.

3. Get better at writing

Last year I wrote a lot of blog posts, even a few that got me on the top of the list at awesome communities like CodeNewbie, but not nearly as much as I would like.

I started strong but in the end, the search for a new job, and my other goals got in the way. This year I am aiming to have fewer things to focus on, but do them with maximum effort!.

• Plan certain dates during the week where I am gonna write.
• Have a list of topics I can write about so I can just pick one from the pile when the time comes.
• Write at least 2 blog posts a week.
• Write about what I am learning to solidify knowledge.

4. Improve my involvement with the community

This year I joined a community called FrontendCafe, awesome people from Argentina and other countries of Latin America (and people from Spain too 😄 ). This year I want to continue meeting developers all over the world but I want to give special attention to people from the Spanish speaking community, specifically those who don't have the means to learn new things or that face some of the problems I faced when I was starting in this industry.

• Give more mentorships at FrontendCafe and outside the Spanish speaking community.
• Spend more time helping new users and using my role as a Staff member in FrontendCafe.
• Go to meetups (if they come back).
• Give a talk at a local meetup (if they come back).

As you may notice some of the things I already said in the planning of 2020 and that is perfectly fine. I didn't reach the level I want to achieve in many topics so this year I am gonna focus even more on those.
Other topics have disappeared completely, that is fine too. After 2020 I learned a few lessons about myself, about what I like, and in which topics I want to invest more of my time on.

I hope you like my planning for 2021, if so please share it and let me know below in the comments. Do you have a similar plan? I would like to read it, the comments is also the perfect place for that.

See you soon 😄

Permalink

Becoming A Software Developer

I get asked a lot for advice on becoming a software developer. I didn't study anything related to software in university, but now it's what I do for a living. I also spent about 3.5 years mentoring with Lighthouse Labs, a local web development bootcamp, and watched (and would like to believe at least in some cases, helped) hundreds of people of every age from wildly diverse backgrounds get jobs as software developers.

This is my advice for people with no technical experience who want to land their first job writing software. It's based on my experience as both a person who did it myself and as a mentor to other people who did it. There's no single right way to go about this, and no matter how you do it's a lot of work. But these are some things that have worked for other people, myself included. These suggestions are arranged roughly in order, but of course many overlap and kind of happen at the same time.

Know what you're getting yourself into

Software engineering is nothing like I expected it to be. All the hard parts involve dealing with people. Part of my original motivation for choosing a career in software was thinking I wouldn't have to deal much with people. I wasn't very good at it and didn't enjoy it, but my fantasy of just sitting in front of my computer all day and watching my bank account fill up was totally delusional.

The reality of writing code for a living, in my experience, includes dealing with a lot of office politics, entitled middle managers, stressed out co-workers, and yes, condescencion and occasional harassment (especially if you're not male). I find the culture of the tech industry generally toxic and abusive and it has very nearly broken my spirit on multiple occasions.

You won't be alone dealing with this. Your co-workers and countless people across the industry share the experience. And it is ultimately bearable.

I don't say this to discourage anyone from getting into software, but just to highlight that the stuff you hear from recruiters is bullshit. It's not all free trips and fancy perks and big paycheques. It is those things, sure, but the reality is much less glamorous than they make it sound.

Being a software developer means getting paid to solve other people's problems. The rest is fluff. This makes it a solid career path, though, because other people have a lot of problems.

It is still an empowering and lucrative career. Despite my generally negative view of the industry, I still highly recommend and help people into a career in software regularly. Most individual software developers, like most people, are genuine and kind. And I can't speak to problems in other industries first hand, but I gather most high-paying, male-dominated industries are pretty similar anyway.

Meet lots of people

I was referred to three of the four programming jobs I've had so far through friends. I made friends with those people who were already programmers at meetups. I didn't ask them to get me a job, but a lot of companies are pretty much always looking for new software engineers, so when your friend hears you're looking for that kind of job, they can bring it up at work. I still did interviews, but it makes an enormous difference when someone the boss already knows and trusts can vouch for you.

A word on meeting strangers. I, like many people who are into nerd things like software, am the most introverted person I know. I found hanging out in groups of new people excruciating at first, but honestly learning how to not be awkward around strangers is just as important as learning how to code if you want to do well in your career. I just went to the meetups and stood off to the side, awkwardly making eye contact with a few random people. Eventually a friendly extrovert would come up and initiate a conversation, and before I knew it I could hold my own in a crowd.

Check out meetups for programming languages you're interested in, or if there aren't those kinds of groups where you live, join online communities. Most programming languages have a slack or discord server where people hang out, and in my experience people are usually very welcoming and kind to newbies (at least in the Ruby, Elixir, and Clojure communities; all of which I highly recommend).

Learn how to code, in public

Without any formal education in software development, you have to offer something else to prove you actually know how to write code. I did this by building a bunch of crappy side projects. The very first thing I ever did was replicate the Google landing page with HTML and CSS. The first actual app I made was a blackjack game with Sinatra. I also made Twitter and Reddit clones with Rails, and little browser games like Pacman with different JavaScript frameworks. Some of that code still exists in my GitHub graveyard if you want to see what I mean by crappy. Other common toy projects include link shorteners, photo galleries, e-commerce stores, time trackers, or clones of whatever software you normally use, like spreadsheets, workflow management, calendars, etc. The possibilities are endless.

Look for tutorials online that walk through how to build the kinds of things you're interested in. If you don't know where to start, The Odin Project and Free Code Camp are two free programs that offer very realistic learning roadmaps. I did parts of both of those but never finished either. I also did several free courses on Coursera and edX, which are both amazing but you have to be careful not to end up in a rabbit hole. Those two mostly have actual university courses, which are super interesting, but not very relevant to things you'll do in a real job.

If your goal is to get from nothing to making money as a developer as fast as possible I recommend sticking to a more targeted learning roadmap like the two above. There are also countless paid ones which vary wildly in price and quality. Be weary about shelling out huge amounts of money for bootcamps. I did pay for a few months of Launch School (which at the time was called "Tea Leaf Academy"), and it was the best career move I ever made. I highly recommend them because they're in a sweet spot of offering a robust and relevant curriculum without costing a fortune. It's not free ($200/month), but a few hundred dollars is a very small investment in the scheme of your career. Apparently they also offer a deferred payment option now, too.

The career-oriented bootcamps will also teach you loads of useful practical things they don't teach in the free university courses, like version control (git), testing, and how to collaborate with other developers on one project.

It took me 6 months of studying, meeting people, and building crappy software full time before I got my first internship. 95% of Lighthouse Labs graduates are employed as software developers in that timeframe. Launch School delivers impressive, concrete results for many of their students, too. I never finished most of the courses or programs I started. I funded that time off by living like a peasant on money I made running one of those student painting businesses the previous summer. Learning to code can feel like a slog at times, but just stick with it and you're very likely to be among the vast majority who succeed at this self-educated developer thing. It is a totally achievable and realistic goal and you're in good company.

Other than doing free online courses and building apps from tutorials, other ways I learned in public during this time included blogging about my learning journey, making myself a portfolio website to showcase my little projects, and volunteering with some local tech communities. Do what works for you, but focus on ending up with at least a few medium sized projects you can show someone to prove you know how to make software that works.

Learn best practices

A surprising number of experienced developers still write really horrible code. Spend some time learning about software design and software engineering best practices to avoid becoming one of them. How you do that depends a lot on how you learn. (BTW, you should figure that out before you embark upon a journey of self-re-education for a career change). I absorb a lot from reading and prefer to learn by studying theory over trial and error, so reading and watching talks works really well for me. YMMV and you do ultimately just need experience to really get this stuff, but you don't have to start from scratch.

Read "The Pragmatic Programmer" by Dave Thomas and Andrew Hunt, and anything by Sandi Metz. Watch Rich Hickey's talks, especially "Simple Made Easy", and Sarah Mei's "Livable Code". Other classics include Eric Evans' "Domain Driven Design" and Martin Fowler's "Refactoring". There are countless others and everyone has their opinion about what should be included in the software engineering canon, but these are my recommendations. They'll provide a solid foundation and ample fodder for further reading and watching.

Pour your heart out

Once you can write code that mostly works, start looking for a job. Put the word out that you're seeking your first role writing software. Hopefully one of your programmer friends will know about an open position, but you can still apply to random jobs on the internet, too. This has a very low success rate, but it's possible. I got one of my four jobs by firing my resume into the abyss. I included a very long and heartfelt cover letter about how I loved programming so much and really wanted to work for this specific company (which I did), and how I learn fast and really believed I could do the job if they'd give me a chance.

It worked. The hiring manager said my letter caught his attention. I had one year of experience at that time and had never worked remotely before, but I got the job.

Don't be afraid to be authentic. Don't lie and say you know more than you do, but don't sell yourself short, either. Bootstrapping a career in software is no small feat and there are many teams out there who would love to hire the kinds of self-starting fast learners who do it.

Don't be too picky (at first!)

The reality is that your first job or two will probably be pretty crappy and underpaid. At this point you're trying to convince someone to take a chance on you, and it's a trade off between your dignity and your future career. In the very beginning, you should prioritize gaining experience over almost everything else. Obviously never put up with anything you're uncomfortable with, but if you can stick it out it's worth setting the bar pretty low just to get your first few months of real job experience, IMHO.

I lucked out and worked with the most amazing team in the world for my first job as a developer, but I discovered from the students I mentored that that is very much not the norm.

There are pros and cons to every kind of company, so don't worry too much about finding the perfect fit right away. Focus on gaining experience at first so you can have the freedom to focus on finding the perfect fit later. I've worked with a few different kinds of companies and there are pros and cons no matter where you go.

Big corporations usually have a lot of experienced developers and lethargic processes, so there's more opportunity to take your time and get help from people more advanced than you. The pressure is often pretty low because people have such low expectations for new developers, which honestly is fair. You have a lot of room to screw up because you're such a tiny cog in a massive machine. The downsides depend on your personality. For me it was that I find most large tech companies to be evil and didn't like the soul-crushing feeling of helping to build something I hated.

Startups usually have fewer people available to be mentors, but more opportunity to take on disproportionate responsibility for your level, which can be an amazing learning experience in a different way. You also have a lot of room to screw up, but it's more because your screw ups just blend in with the overall culture of moving fast and breaking things. I now believe that's a ridiculous way to run a business, but when there's widespread acceptance of breaking even important things, you have a lot freedom to experiment and learn by trial and error.

Be a good student

Lastly, once you finally land your first job, be a good student. Treat your first year or two on the job like paid education, not your chance to go in and show everyone how it's done. You know way less than you think, and you have no idea what it takes to build functioning software systems, let alone to deploy or maintain them. Always be humble and eager to learn. Soak up every minute of attention you get from more experienced developers around you and learn from them. Not everything they say is wise or true, but pay attention and copy the things that work whilst ignoring the things that clearly don't.

Learn how to ask good questions. Don't be afraid to ask for help, but before you do, at least try to figure out the problem yourself. Write down what you tried, and explain why you're surprised your attempt didn't work.

There are certain things you can only really learn from experience on a real world project, so see what opportunities there are to learn those things, for example:

• securing and backing up actual important customer data
• deploying a huge app with no downtime
• refactoring large, messy, legacy codebases
• adding new features to large, messy, legacy codebases
• fixing obscure bugs with no instructions to reproduce them

These are just a few examples of things you won't run into on your little demo projects. Pay attention to the problems you're dealing with at work that you don't run into on side projects and learn how to fix those. That's your pathway to levelling up. Most importantly, never stop learning.

That's all

Software development is a great career. It's allowed me to be financially independent and stable, travel the world, meet cool people, and lots more. My path won't work for everyone. There are as many different ways to become a software developer as there are people, but I've seen countless aspiring software engineers go through their own journeys and many of them touch on these points, so see how they fit in to your own path. And let me know how it goes 🙂

Permalink

Becoming A Software Developer

I get asked a lot for advice on becoming a software developer. I didn't study anything related to software in university, but now it's what I do for a living. I also spent about 3.5 years mentoring with Lighthouse Labs, a local web development bootcamp, and watched (and would like to believe at least in some cases, helped) hundreds of people of every age from wildly diverse backgrounds get jobs as software developers.

This is my advice for people with no technical experience who want to land their first job writing software. It's based on my experience as both a person who did it myself and as a mentor to other people who did it. There's no single right way to go about this, and no matter how you do it's a lot of work. But these are some things that have worked for other people, myself included. These suggestions are arranged roughly in order, but of course many overlap and kind of happen at the same time.

Know what you're getting yourself into

Software engineering is nothing like I expected it to be. All the hard parts involve dealing with people. Part of my original motivation for choosing a career in software was thinking I wouldn't have to deal much with people. I wasn't very good at it and didn't enjoy it, but my fantasy of just sitting in front of my computer all day and watching my bank account fill up was totally delusional.

The reality of writing code for a living, in my experience, includes dealing with a lot of office politics, entitled middle managers, stressed out co-workers, and yes, condescencion and occasional harassment (especially if you're not male). I find the culture of the tech industry generally toxic and abusive and it has very nearly broken my spirit on multiple occasions.

You won't be alone dealing with this. Your co-workers and countless people across the industry share the experience. And it is ultimately bearable.

I don't say this to discourage anyone from getting into software, but just to highlight that the stuff you hear from recruiters is bullshit. It's not all free trips and fancy perks and big paycheques. It is those things, sure, but the reality is much less glamorous than they make it sound.

Being a software developer means getting paid to solve other people's problems. The rest is fluff. This makes it a solid career path, though, because other people have a lot of problems.

It is still an empowering and lucrative career. Despite my generally negative view of the industry, I still highly recommend and help people into a career in software regularly. Most individual software developers, like most people, are genuine and kind. And I can't speak to problems in other industries first hand, but I gather most high-paying, male-dominated industries are pretty similar anyway.

Meet lots of people

I was referred to three of the four programming jobs I've had so far through friends. I made friends with those people who were already programmers at meetups. I didn't ask them to get me a job, but a lot of companies are pretty much always looking for new software engineers, so when your friend hears you're looking for that kind of job, they can bring it up at work. I still did interviews, but it makes an enormous difference when someone the boss already knows and trusts can vouch for you.

A word on meeting strangers. I, like many people who are into nerd things like software, am the most introverted person I know. I found hanging out in groups of new people excruciating at first, but honestly learning how to not be awkward around strangers is just as important as learning how to code if you want to do well in your career. I just went to the meetups and stood off to the side, awkwardly making eye contact with a few random people. Eventually a friendly extrovert would come up and initiate a conversation, and before I knew it I could hold my own in a crowd.

Check out meetups for programming languages you're interested in, or if there aren't those kinds of groups where you live, join online communities. Most programming languages have a slack or discord server where people hang out, and in my experience people are usually very welcoming and kind to newbies (at least in the Ruby, Elixir, and Clojure communities; all of which I highly recommend).

Learn how to code, in public

Without any formal education in software development, you have to offer something else to prove you actually know how to write code. I did this by building a bunch of crappy side projects. The very first thing I ever did was replicate the Google landing page with HTML and CSS. The first actual app I made was a blackjack game with Sinatra. I also made Twitter and Reddit clones with Rails, and little browser games like Pacman with different JavaScript frameworks. Some of that code still exists in my GitHub graveyard if you want to see what I mean by crappy. Other common toy projects include link shorteners, photo galleries, e-commerce stores, time trackers, or clones of whatever software you normally use, like spreadsheets, workflow management, calendars, etc. The possibilities are endless.

Look for tutorials online that walk through how to build the kinds of things you're interested in. If you don't know where to start, The Odin Project and Free Code Camp are two free programs that offer very realistic learning roadmaps. I did parts of both of those but never finished either. I also did several free courses on Coursera and edX, which are both amazing but you have to be careful not to end up in a rabbit hole. Those two mostly have actual university courses, which are super interesting, but not very relevant to things you'll do in a real job.

If your goal is to get from nothing to making money as a developer as fast as possible I recommend sticking to a more targeted learning roadmap like the two above. There are also countless paid ones which vary wildly in price and quality. Be weary about shelling out huge amounts of money for bootcamps. I did pay for a few months of Launch School (which at the time was called "Tea Leaf Academy"), and it was the best career move I ever made. I highly recommend them because they're in a sweet spot of offering a robust and relevant curriculum without costing a fortune. It's not free ($200/month), but a few hundred dollars is a very small investment in the scheme of your career. Apparently they also offer a deferred payment option now, too.

The career-oriented bootcamps will also teach you loads of useful practical things they don't teach in the free university courses, like version control (git), testing, and how to collaborate with other developers on one project.

It took me 6 months of studying, meeting people, and building crappy software full time before I got my first internship. 95% of Lighthouse Labs graduates are employed as software developers in that timeframe. Launch School delivers impressive, concrete results for many of their students, too. I never finished most of the courses or programs I started. I funded that time off by living like a peasant on money I made running one of those student painting businesses the previous summer. Learning to code can feel like a slog at times, but just stick with it and you're very likely to be among the vast majority who succeed at this self-educated developer thing. It is a totally achievable and realistic goal and you're in good company.

Other than doing free online courses and building apps from tutorials, other ways I learned in public during this time included blogging about my learning journey, making myself a portfolio website to showcase my little projects, and volunteering with some local tech communities. Do what works for you, but focus on ending up with at least a few medium sized projects you can show someone to prove you know how to make software that works.

Learn best practices

A surprising number of experienced developers still write really horrible code. Spend some time learning about software design and software engineering best practices to avoid becoming one of them. How you do that depends a lot on how you learn. (BTW, you should figure that out before you embark upon a journey of self-re-education for a career change). I absorb a lot from reading and prefer to learn by studying theory over trial and error, so reading and watching talks works really well for me. YMMV and you do ultimately just need experience to really get this stuff, but you don't have to start from scratch.

Read "The Pragmatic Programmer" by Dave Thomas and Andrew Hunt, and anything by Sandi Metz. Watch Rich Hickey's talks, especially "Simple Made Easy", and Sarah Mei's "Livable Code". Other classics include Eric Evans' "Domain Driven Design" and Martin Fowler's "Refactoring". There are countless others and everyone has their opinion about what should be included in the software engineering canon, but these are my recommendations. They'll provide a solid foundation and ample fodder for further reading and watching.

Pour your heart out

Once you can write code that mostly works, start looking for a job. Put the word out that you're seeking your first role writing software. Hopefully one of your programmer friends will know about an open position, but you can still apply to random jobs on the internet, too. This has a very low success rate, but it's possible. I got one of my four jobs by firing my resume into the abyss. I included a very long and heartfelt cover letter about how I loved programming so much and really wanted to work for this specific company (which I did), and how I learn fast and really believed I could do the job if they'd give me a chance.

It worked. The hiring manager said my letter caught his attention. I had one year of experience at that time and had never worked remotely before, but I got the job.

Don't be afraid to be authentic. Don't lie and say you know more than you do, but don't sell yourself short, either. Bootstrapping a career in software is no small feat and there are many teams out there who would love to hire the kinds of self-starting fast learners who do it.

Don't be too picky (at first!)

The reality is that your first job or two will probably be pretty crappy and underpaid. At this point you're trying to convince someone to take a chance on you, and it's a trade off between your dignity and your future career. In the very beginning, you should prioritize gaining experience over almost everything else. Obviously never put up with anything you're uncomfortable with, but if you can stick it out it's worth setting the bar pretty low just to get your first few months of real job experience, IMHO.

I lucked out and worked with the most amazing team in the world for my first job as a developer, but I discovered from the students I mentored that that is very much not the norm.

There are pros and cons to every kind of company, so don't worry too much about finding the perfect fit right away. Focus on gaining experience at first so you can have the freedom to focus on finding the perfect fit later. I've worked with a few different kinds of companies and there are pros and cons no matter where you go.

Big corporations usually have a lot of experienced developers and lethargic processes, so there's more opportunity to take your time and get help from people more advanced than you. The pressure is often pretty low because people have such low expectations for new developers, which honestly is fair. You have a lot of room to screw up because you're such a tiny cog in a massive machine. The downsides depend on your personality. For me it was that I find most large tech companies to be evil and didn't like the soul-crushing feeling of helping to build something I hated.

Startups usually have fewer people available to be mentors, but more opportunity to take on disproportionate responsibility for your level, which can be an amazing learning experience in a different way. You also have a lot of room to screw up, but it's more because your screw ups just blend in with the overall culture of moving fast and breaking things. I now believe that's a ridiculous way to run a business, but when there's widespread acceptance of breaking even important things, you have a lot freedom to experiment and learn by trial and error.

Be a good student

Lastly, once you finally land your first job, be a good student. Treat your first year or two on the job like paid education, not your chance to go in and show everyone how it's done. You know way less than you think, and you have no idea what it takes to build functioning software systems, let alone to deploy or maintain them. Always be humble and eager to learn. Soak up every minute of attention you get from more experienced developers around you and learn from them. Not everything they say is wise or true, but pay attention and copy the things that work whilst ignoring the things that clearly don't.

Learn how to ask good questions. Don't be afraid to ask for help, but before you do, at least try to figure out the problem yourself. Write down what you tried, and explain why you're surprised your attempt didn't work.

There are certain things you can only really learn from experience on a real world project, so see what opportunities there are to learn those things, for example:

• securing and backing up actual important customer data
• deploying a huge app with no downtime
• refactoring large, messy, legacy codebases
• adding new features to large, messy, legacy codebases
• fixing obscure bugs with no instructions to reproduce them

These are just a few examples of things you won't run into on your little demo projects. Pay attention to the problems you're dealing with at work that you don't run into on side projects and learn how to fix those. That's your pathway to levelling up. Most importantly, never stop learning.

That's all

Software development is a great career. It's allowed me to be financially independent and stable, travel the world, meet cool people, and lots more. My path won't work for everyone. There are as many different ways to become a software developer as there are people, but I've seen countless aspiring software engineers go through their own journeys and many of them touch on these points, so see how they fit in to your own path. And let me know how it goes 🙂

Permalink

World Statistics Exercise

World Statistics App.

Introduction

After implementing my web-store exercise using six languages (Clojure, Python, Javascript, Java, Go, and Kotlin) and with 5 data stores (CSV, DynamoDB, Azure Tables, PostgreSQL, and Datomic, read more about in my blog posts, e.g. Five Languages - Five Stories ) I was pretty much fed up with that exercise. I wanted to do something totally different. But what? I spent quite a lot of time figuring out what to do. I was dreaming to do something with Finnish Open data that could benefit the general public. I was browsing the Finnish Open data but couldn’t figure out anything useful that would contain a lot of data points. Then I decided to take a look at the World Bank Open data. Just out of curiosity I downloaded various metrics between the years 2002-2017, some 150.000 data points altogether. Then I just started to experiment with the data using the Clojure REPL - a great tool for studying and experimenting with data. Little by little a new exercise was formed in my head: Implement a full-stack app using Clojure and ClojureScript to filter and enrich the data and visualize the data sets using a frontend. This blog post depicts the story of this exercise.

The application source can be found in my Github clojure repo in directory statistics/worldstat. The application is live in Heroku (as long as the Heroku free tier dyno hours have not run out). NOTE: If there are still dyno hours and no-one has used the app for some time you may need to wait for a while for Heroku to start the app. Since the app is running in the free tier there might be other aspects as well that may impact using the app (max connections etc.)

Technology Overview

I used Clojure, and some Clojure libraries (like reitit) to implement the backend. The backend reads the data from CSV files that I downloaded from the Worldbank site, and manipulates the data so that we have the data and meta-data nicely as Clojure data structures stored as part of the Integrant state. I used kixi as the statistics library.

I used ClojureScript, Reagent, and Re-Frame to implement the frontend. For frontend development I used shadow-cljs. For CSS I used the excellent and light CSS library Bulma. For data visualization, I used Vega Lite and as a Clojure wrapper for Vega, I used Metasoarous’ Oz library. The world-110m.json world map is from Vega.

Backend - Creating the Data

In main.clj I read the Integrant configuration: the data files (the CSV files that I downloaded from the Worldbank, and the toposon file) and inject them to init.clj:

(defmethod ig/init-key :backend/data [_ {:keys [data-files topojson-file]}]
  (init/get-data data-files topojson-file))

In init.clj I parse the data from the CSV files and convert it to edn. In that file, I do some processing and finally return the data structure back to main.clj which stores the data as part of the Integrant state:

(defn get-data [data-files topojson-file]
  (log/debug "ENTER get-data")
  (let [topojson-country-codes (read-topojson-country-codes topojson-file)
        ids (into {} (map (juxt :acr :id) topojson-country-codes))
        data-points (get-data-points data-files ids)
        {:keys [countries series years]} (get-metadata data-points)]
    {:raw-points data-points
     :points (remove-non-country-values data-points non-country-codes)
     :countries countries
     :series series
     :years years
     :country-codes (country-codes-to-names countries)
     :series-codes (series-codes-to-names series)
     :non-country-codes non-country-codes
     :country-ids ids
     :topojson-country-codes topojson-country-codes
     }))

An important part of this processing is to store the topojson country ids as part of the data points so that we can later use them as hooks to actual countries in the topojson map.

I downloaded some 150.000 data points for this exercise (:raw-points). I removed various data points (:points) that were associated with continents, income groups, etc. Finally, I had some 130.000 data points that were associated with actual countries found in the topojson map. It’s easy to study the data using the Clojure REPL:

(count (:raw-points (user/data)))
=> 153152
(count (:points (user/data)))
=> 129824

… but more about that in the next chapter.

An individual data point is a Clojure map. Let’s get one data point from Finland and check it in more detail:

(first (transduce (ws-filter/filter-by :country_code :FIN) conj (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS))))
=>
{:country_name "Finland",
 :country_code :FIN,
 :series-name "Hospital beds (per 1,000 people)",
 :series-code :SH.MED.BEDS.ZS,
 :year 2002,
 :value 7.4,
 :country-id 246}

Backend - Using Clojure REPL to Experiment with the Data

Using a real REPL is such a joy with Clojure. Let’s use it to experiment with the data:

(set! *print-length* 100)
=> 100 ; So that we don't accidentally list 150.000 data points in repl...
;; Require some namespaces.
(require '[worldstat.backend.data.world :as ws-world])
=> nil
(require '[worldstat.common.data.filter :as ws-filter])
=> nil
(require '[kixi.stats.core :as kixi])
=> nil
;; Check how many points there are...
(count (:raw-points (user/data)))
=> 153152
(count (:points (user/data)))
=> 129824
;; What keys do we have in our data?
(keys (user/data))
=>
(:country-codes
 :raw-points
 :series
 :non-country-codes
 :country-ids
 :points
 :series-codes
 :countries
 :topojson-country-codes
 :years)
;; Let's sort countries...
(sort-by :country_name (:countries (user/data)))
=>
({:country_name "Afghanistan", :country_code :AFG, :country-id 4}
 {:country_name "Albania", :country_code :ALB, :country-id 8}
 {:country_name "Algeria", :country_code :DZA, :country-id 12}
 {:country_name "American Samoa", :country_code :ASM, :country-id 16}
 ...)
;; Get hospital beds, filter data points belonging to Finland, and get first data point. 
(first (transduce (ws-filter/filter-by :country_code :FIN) conj (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS))))
=>
{:country_name "Finland",
 :country_code :FIN,
 :series-name "Hospital beds (per 1,000 people)",
 :series-code :SH.MED.BEDS.ZS,
 :year 2002,
 :value 7.4,
 :country-id 246}
;; Let's check some basic statistics, mean and standard deviation for certain data. 
(transduce (map :value) kixi/mean (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS)))
=> 3.668379451410944
(transduce (map :value) kixi/standard-deviation (:points (ws-world/get-world-data (user/env) :SH.MED.BEDS.ZS)))
=> 2.601440577329428

Repl is truly such a powerful tool in Lisp. I always keep a repl scratch file in all my Clojure projects and I do experimentation like above in those scratch files - it’s nice to have those experimentations later on as a historical record of how I implemented functions.

Backend - API

I used the excellent Metosin reitit library for routing: routes.clj:

    ["/data" {:swagger {:tags ["data"]}}
     ["/metric/:metric" {:get {:summary "metric"
                               :parameters {:path [:map [:metric string?]]}
                               :responses {200 {:description "data"}}
                               :handler (fn [req]
                                          (let [metric (get-in req [:parameters :path :metric])]
                                            (-> (world/get-world-data env (keyword metric))
                                                r/ok)))}}]
     ["/countries" {:get {:summary "countries"
                          :responses {200 {:description "data"}}
                          :handler (fn [req]
                                 (-> (world/get-countries env)
                                     r/ok))}}]
     ["/years" {:get {:summary "years"
                      :responses {200 {:description "data"}}
                      :handler (fn [req]
                                 (-> (world/get-years env)
                                     r/ok))}}]
     ["/metric-names" {:get {:summary "metric-names"
                             :responses {200 {:description "data"}}
                             :handler (fn [req]
                                        (-> (world/get-metric-names env)
                                            r/ok))}}]]]])

So, you can query the actual data points by giving a metric (:series-code) and backend returns the data points for that metric. Basically, I just implemented a minimum API that is needed by the frontend to do the visualization.

Reitit library provides swagger documentation out of the box, so you can experiment with the API easily also with the browser:

API Swagger documentation.

Next, let’s move to the frontend.

Frontend - Reagent and Re-frame

I used ClojureScript, Reagent, and Re-Frame to implement the frontend. For frontend development I used shadow-cljs. Reagent provides a nice interface to React, and Re-frame is a ClojureScript framework, that uses Reagent and provides simple state management to build single-page applications using Clojurescript. All this tooling works pretty seamlessly with the Clojure backend repl so that when I refresh the Integrant state (with one hotkey, of course) the frontend also refreshes. One can get a Clojurescript repl and connect to the frontend application running inside the browser if you need it. Usually, you don’t, since there are nice tools and tricks to examine the frontend state.

Clojurescript frontend tools.

On the right side of the browser, you see the standard Chrome developer tools window. There you can examine the html structure, see console log, etc, nothing new for any frontend developer.

At the bottom is the Metosin reagent-dev-tools. You can subscribe all re-frame application data there and browse the data. I also implemented a simple debug panel in which I show some most important application data that I used when implementing the frontend. Btw. I really recommend using the Metosin reagent-dev-tools. E.g. I’m in a customer project in which we are implementing a rather complex Clojure fullstack application and there is really complex business logic - we have various fixtures in the reagent-dev-tools which makes it a breeze to initialize the frontend and the whole application to a certain state in the middle of the business logic.

All this tooling works really nicely. I’m an old backend programmer myself, but using these tools it was quite effortless to implement this fullstack application. You don’t have to be a Javascript + React guru to implement a frontend like the one in this exercise.

CSS - Bulma

I used Bulma as a CSS framework. Bulma is really great - I barely added any custom css in this application. Let’s see part of the frontend using the hiccup notation:

[:div.row
 [:div.columns
  [:div.column.is-9
   [oz/vega-lite (ws-data/world-schema points selected-year selected-metric-name min max) (ws-util/vega-debug)]]
  [:div.column.auto
   (if country-stats
     [:div.rows
      [:div.row
       [:p.is-size-3 selected-country-name]]
      [:div.row
       [:p.is-size-5 "Min: " (ws-util/one-decimal (:min country-stats))]
       [:p.is-size-5 "Max: " (ws-util/one-decimal (:max country-stats))]
       [:p.is-size-5 "Mean: " (ws-util/one-decimal (:mean country-stats))]
       [:p.is-size-5 "Std.dev: " (ws-util/one-decimal (:standard-deviation country-stats))]]
      [:div.row
       [oz/vega-lite (ws-data/country-year-diagram points selected-metric-name selected-country-code min max) (ws-util/vega-debug)]]]
     [:p.is-size-4 "Click a country!"])]]]
[:div.row
 [:p "(Missing data for country is shown with color gray)"]]
[:div.row
 [:p "Read more about this app in my blog "
  [:a {:href "https://www.karimarttila.fi/clojure/2021/01/19/world-statistics-exercise.html"} "www.karimarttila.fi"]]]

You can use Bulma styles seamlessly with hiccup, e.g. :div.columns tells the browser to render the div using Bulma columns etc. Check more in the Bulma documentation.

The dropdown box for selecting the metric and the slider for selecting the year to visualize in the map are also Bulma components. You can customize the Bulma components quite nicely using Clojurescript, example showing the year slider:

(defn slider [id initial step min max type]
  (let [slider-value (r/atom initial)]
    [:div.slider-content
     [:input.slider.is-fullwidth {:id id :step step :min min :max max :value @slider-value :type type
                                  :on-change (fn [event]
                                               (.preventDefault event)
                                               (let [new-value (.-value (.getElementById js/document id))]
                                                 (reset! slider-value new-value)
                                                 (re-frame/dispatch [::ws-state/select-year (js/parseInt new-value)])))}]]))

So, basically, I just take the current value of the slider and store it in the re-frame database: (re-frame/dispatch [::ws-state/select-year (js/parseInt new-value)]) .

Frontend - Creating the Map

For data visualization, I used Vega Lite, and as a Clojure wrapper for Vega I used Metasoarous’ Oz library. The world-110m.json world map is from Vega.

I think I spent most of the time developing this application by tuning the world map. But it was really fun to experiment how to visualize data and with how to interact with the map. The result of this experiment can be seen in file data.cljs, I highlight here the most important part of the map configuration:

(defn world-schema [points year metric-name min max]
  {:schema "https://vega.github.io/schema/vega/v5.json"
   :autosize "none"
   :width 900
   :height 500
   :scales [{:name "color",
             :type "quantize",
...             
   :marks [{:type "shape"
            :from {:data "graticule"}
            :encode {:update {:strokeWidth {:value 1}
...
           {:type "shape"
            :from {:data "world"}
            :encode {:enter {:tooltip {:signal "datum.value === null ? {'country': datum.country_name, 'value': 'Missing data'} : {'country': datum.country_name, 'value': format(datum.value, '.2f')}"}
                             :href {:field "url" :type "nominal"}}
                     ;; If missing data show as grey.
                     :update {:fill [{:test "datum.value === null" :value "gray"}
                                     {:scale "color" "field" "value"}]}
                     :hover {:fill {:value "red"}}}
            :transform [{:type "geoshape"
...
   :data [{:name "countries"
           :values (transduce (ws-filter/filter-by-year year) conj points)}
          {:name "world"
           :url "data/world-110m.json"
           :format {:type "topojson" :feature "countries"}
           :transform [{:type "lookup"
                        :from "countries"
                        :key "country-id"
                        :fields ["id"]
                        :values ["value" "country_name" "country_code"]}
                       {:type "formula"
                        :from "countries"
                        :expr "'#/country/' + datum.country_code" :as "url"}]}
          {:name "graticule"
           :transform [{:type "graticule"}]}]})

So, the Vega interface provides quite extensive ways to inject data to the Vega component and how the component should transform, interpret and visualize the data.

Let’s spend some time with this configuration since it is the actual beef of this app.

First I inject to the map configuration the data (points, year, metric-name, min and max of the points): (defn world-schema [points year metric-name min max]. Then I transform the data points using a join (lookup) between the topojson map data and my data points using id of the topojson data and country-id of my data points as joining the data. In this join, I inject the corresponding country_name and country_code to the new data set so that I can use the country_name to provide the country name when the user hovers the mouse over a certain country. I use the country_code for providing a hook when the user mouse clicks a certain country: :expr "'#/country/' + datum.country_code" :as "url"}]} and :href {:field "url" :type "nominal"} . That’s basically what you have to do - and the Vega library does the rest. Then I just show the map component in the application home page, main.cljs (injecting the re-frame subscribed data):

          {selected-metric-code :code selected-metric-name :name} @(re-frame/subscribe [::ws-state/current-metric])
          selected-year @(re-frame/subscribe [::ws-state/current-year])
          {selected-country-code :code selected-country-name :name} @(re-frame/subscribe [::ws-state/current-country])
          {:keys [points min max mean standard-deviation]} @(re-frame/subscribe [::ws-state/world-data selected-metric-code])
          metric-names @(re-frame/subscribe [::ws-state/metric-names])
          _ (if-not points (re-frame/dispatch [::ws-state/get-world-data selected-metric-code]))
          country-stats (if selected-country-code (ws-data/get-stats points selected-country-code))]
...
          [:div.column.is-9
           [oz/vega-lite (ws-data/world-schema points selected-year selected-metric-name min max) (ws-util/vega-debug)]]

Using Vega Editor

Since the map component is a bit complicated and for a Vega newcomer the transform operations are a bit hairy Vega provides excellent development tools. See the three dots icon here:

Going to Vega Editor.

Just click it and choose the Open in Vega Editor and the browser opens this for you:

Vega Editor.

You can experiment with the graphics first using the Vega editor, see the data, and check how various tunings in parameters look like in the map - and once you are happy just copy-paste the configuration back to your clojurescript source file.

Frontend - Navigation etc.

There is very little navigation in the application. I used Reitit also to do the frontend navigation:

(def routes-dev
  ["/"
   [""
    {:name ::ws-state/home
     :view home-page
     :link-text "Home"
     :controllers
     [{:start (fn [& params] (ws-log/clog (str "Entering home page, params: " params)))
       :stop (fn [& params] (ws-log/clog (str "Leaving home page, params: " params)))}]}]
   ["country/:country-code"
    {:name ::ws-state/country
     :parameters {:path {:country-code string?}}
     :view home-page
     :link-text "country"
     :controllers
     [{:start (fn [& params] (ws-log/clog (str "Entering country, params: " params)))
       :stop (fn [& params] (ws-log/clog (str "Leaving country, params: " params)))}]}]])

Basically, the only navigation is provided when the user clicks a country - we show the year statistics of the selected metric in the line diagram. I actually spend some time with this functionality. I couldn’t figure out how to register the mouse button click event in the vega map - but I found in the documentation a way to inject a href as part of the map, so I used it: :expr "'#/country/' + datum.country_code" :as "url"}]} and :href {:field "url" :type "nominal"} - then in the frontend routing I just provide a route for this uri: ["country/:country-code", and in state.cljs:

(re-frame/reg-event-db
  ::navigated
  (fn [db [_ new-match]]
    (let [old-match (:current-route db)
          new-path (:path new-match)
          country-code (get-in new-match [:parameters :path :country-code])
          country-name (get-country-name country-code)
          controllers (rfc/apply-controllers (:controllers old-match) new-match)]
      (ws-log/clog (str "::state/navigated, new-path: " new-path))
      (ws-log/clog (str "::state/navigated, country-code: " country-code))
      (cond-> (assoc db :current-route (assoc new-match :controllers controllers))
              country-code (assoc :current-country {:code country-code :name country-name})))))

So, here we just get the country-code from the uri, find the matching country-name for it and store them into the re-frame database for further use.

The fullstack app is now ready. Let’s next wrap the app into a deployment unit.

Docker Container and Heroku Deployment

I considered deploying the app to AWS but since I’m pretty stingy I didn’t want to spend my personal money on that. Then I realized that I can use Heroku Free Tier to deploy the app to the cloud (at least for the time the Heroku free dyno hours run out, or the World Bank wants to start sponsoring the app :-) ).

First I needed to create a docker container. Instructions for creating and running the Docker container (basically just testing that it works, Heroku creates the image using the heroku cli, more about it later):

• just uberjar
• docker build -f Dockerfile . -t worldstat:latest
• docker run -e PORT=9999 -it -p9999:9999 worldstat:latest

The Dockerfile is pretty minimalistic. I just create a non-root user to run the app (Heroku requires this), I copy the app and the data to the image, and provide the entrypoint to the application: run-app.sh. Notice the pretty harsh memory limits in the app (Heroku gives just max 512MB memory in the free tier).

NOTE: The application is running in the Heroku free tier and Heroku removes the application if it is not used for a certain period of time - in this case just wait patiently a few seconds and you should see the app unless the dyno hours are completely spent.

The Heroku deployment process for Docker images is pretty nicely documented here. Let’s give short instructions if you want to try it yourself:

• First build the uberjar: just uberjar
• Then basically just follow the instructions in the Heroku documentation.
• Create a Heroku account
• heroku create
• heroku container:login
• heroku container:push web -a <YOUR-HEROKU-APP-NAME>
• heroku container:release web -a <YOUR-HEROKU-APP-NAME>

I then changed the application name to some more descriptive: km-worldstat .

Conclusions

This exercise was really interesting. I learned quite a lot how to experiment data using REPL in the backend. I also learned quite a lot how to create nice visualizations using the Vega library. I also learned that using Clojure and Clojurescript you can create quite powerful fullstack applications without the need being a Javascript / React guru.

The writer is working at Metosin using Clojure in cloud projects. If you are interested in starting a Clojure project or getting Clojure training in Finland you can contact me by sending an email to my Metosin email address or contact me via LinkedIn.

Kari Marttila

• Kari Marttila’s Home Page in LinkedIn: https://www.linkedin.com/in/karimarttila/

Permalink

Predictive maintenance platform for Gearboxes

We often hear the term Industry 4.0 as a necessary process for a more automated, efficient and sustainable future. In this post I would like to present a real example of an Industry 4.0 project we are working on.

At Magnet we help traditional industries digitize their activities so they can gain insights from data and automation. Recently we have built a gearbox monitoring platform called SmartG that is able to manage large amounts of data in real-time and my idea in this post is to summarize some key technical features that I find interesting.

The functional scope of SmartG covers monitoring, smart analytics, customized reports, alerts and warnings. SmartG basically gets data from the physical world via sensors which are sent to the software platform and is capable of handling large amounts of data to produce insights.

In summary, SmartG brings business value to many shareholders: high availability, better asset utilization, higher product quality and so on. My background is on computer programming so I will focus more on software-related aspects rather than mechanical ones, but if you are interested in reading more you can find more information about this project here and here.

Data transmission

We have chosen a publish/subscribe protocol called MQTT for bidirectional communication and for faster data transmission.

The MQTT server of our choice has been RabbitMQ for being lightweight and deployable on-premise and cloud, which is a critical requirement for this project.

The hardware device is sending us three JSON files, with different frequencies (1 second and 10 second) and 1000 records in each message.

Time-series database

We at Magnet love PostgreSQL, it’s our preferred database, so we felt we struck it lucky when we found Timescale, built on PostgreSQL and focused on time-series data; just what we needed. As I said, Timescale is PostgreSQL for time-series, so we have the best of both worlds: relational and time-series databases for the price of one. It’s super fast, scales horizontally and we can continue to write SQL queries! What’s not to like?

On premise vs cloud deployment

SmartG has been deployed in a company in the automotive sector and the final client’s preference was to deploy the solution on their on-premise environment.

Our Hydrogen platform, which is the underlying technology stack behind SmartG, is ready for the challenge: we have created the Test environment on the Cloud (AWS) to accelerate development process, and the production environment on-premise, as requested by the client.

Dynamic dashboards

To fully leverage the value of data, clients need to experiment with it, they need to play with data creating customized panels by themselves without the help of programmers. Each user has different roles and interests so we need to be able to personalize the dashboard for them. In order to do this, we have seamlessly integrated Grafana and Highcharts in SmartG.

Reactive web user interfaces

The look and feel of the user interface is fast and interactive thanks to the underlying React-based Clojurescript library called re-frame.

Re-frame offers great productivity to produce sophisticated webapps with less lines of code and simpler models. Many companies have been using it successfully since 2015 and it allows us to use the same programming language (Clojure) in both the server and the client.

According to Hermann, Pentek, Otto (2016), there are 4 design principles and goals that are integral to Industry 4.0, namely Interconnection, Information transparency, Technical assistance and Decentralized decisions.

These goals are tackled with different technological approaches in each project, depending on clients’ needs and resources. In fact, we have used different tools in other Industry 4.0 projects we are building, to achieve similar concepts.

After an intense 6-month work and collaboration, SmartG is up and running with an on-premise deployment; another small step for the digitalization of traditional manufacturing and industry.

Predictive maintenance platform for Gearboxes was originally published in magnet.coop on Medium, where people are continuing the conversation by highlighting and responding to this story.

Permalink

Carve that Clojure codebase

Tag: Clojure Visit https://juxt.pro for more.

Permalink

Computing Then and Now

In this episode, we read excerpts of Maurice Wilke's 1967 ACM Turing Award lecture titled 'Computing Then and Now'.

The post Computing Then and Now appeared first on LispCast.

Permalink

Month One Week Two

Watch me program live every morning at 9:30 EST!

Check out past streams on YouTube!

Well, it was a tough second week, not gonna sugarcoat it. This was really the week where I swam out from the shoals of Basic Beach and took my first couple of strokes and kicks in the Ocean of the Unknown. Set up a Twitch page? Start streaming? Start a blog? Set up a basic HTTP server? Configure a database? From a certain angle, these are indeed challenging things to do, and these are all things that have given me a lot of trouble at various stages of my career. But most of it was nevertheless known to me, even though I’m doing it all in a brand new language and in front of an audience, so I could feel relatively confident as I did it.

This week we started getting into some new technologies and hit our first big technical roadblocks, which is a great segue into our headlining topic! A failure of the week. This weird world is great at showcasing a success, but behind every success lies ten failures. In each weekly recap, I’m going to put my failures up front to show you all that stuff is hard, and it takes a lot of trial and error to get it right in the end.

Failure

Chief among my failures this week was setting up the connection between the game frontend and the game backend (if you’re not familiar with the distinction, here’s a decent Quora answer). Seeing as this is a multiplayer game, the communication between players needs to be instantaneous, and that means websockets. I’d heard of these mysterious things before, but I’d never used them. I did some background reading, looked up Clojure websocket libraries, chose a library (Sente if you’re curious), and started in on integrating the library into my existing code.

This, of course, completely flopped. I got majorly stuck on the implementation on day one and didn’t get anywhere with it. So on day two I decided that I had better find a pre-built starter that I could use, seeing as stringing up something from scratch was fairly involved and above my head. I did find a starter kit, and did get it running, and the instant I tried to do something custom with it, it all fell apart on me. Somehow I managed to not only break my super simple test code, but also the pre-existing code as well. Which is fairly embarrassing. More embarrassing still in that the starter kit had been purposely built out to be as simple as possible, and I still couldn’t figure it out.

And so after two precious days, 10% of my allotted development time for the month, I ended up with no working code, maybe a couple new insights about how websockets work (along with a ton of new confusion), a nasty bundle of nerves. What am I doing? I purposely picked this game for month one because I thought it was going to be easier than my other ideas, and I’m already stuck? Bad news.

Redemption: Let It Sit

So I was feeling pretty raw on Wednesday after a Monday and Tuesday full of roadblocks. At a certain point in my life I probably would have jumped back into it – jaw set, lip stiff – and wrestled the problem to the ground, but I decided to switch gears and work on something else instead. I’ve yet to take this course for real, but the teaser video on focused and diffuse modes of learning really sold me on it. The idea is that your brain does a lot of work in the background to solve a problem or learn something new, and this is completely different from how you learn when you’re focusing intentionally on a problem.

I remember the exact moment I learned how sleeping on a problem can give you new insights into it. I was really into Donkey Kong Country as a kid, and I got stuck for days on this puzzle. You can only slide down on the purple ropes. You can hold the up button on the controller to stay in the same spot, but you can never climb up. So when the game presents you with two purple ropes, a pit of death below you, and seemingly no way to go up, little 9-year-old me was totally at a loss for how to progress. My grandma, after listening to my 9-year-old lamentations and taking them grandma-seriously, said that sometimes you just need to sleep on a problem and the solution comes to you out of nowhere. And that night, I literally dreamed about the solution: you jump back and forth between the ropes instead of climbing.

So, finding myself stuck on a purple websocket rope, I jumped over to a blue templating rope. The game needs a UI no matter what, so I may as well give myself a few days of wins by coding up some screens with Hiccup and Tailwind, and it went great! I was firmly back in the domain of what I know well, and I was getting some important UI work done. I haven’t dived back into websockets just yet, but I’ll bet you $10 that I’ll have a new and better perspective on the problem when I do.

It’s a damn shame that the world, with its schedules, sprint metrics and deadlines, doesn’t give you more opportunities to sleep on a hard problem. The scientist who predicted the Higgs boson particle said in an interview that he wouldn’t be productive enough to succeed in today’s academic system. Einstein would spend hours on a row boat in the middle of a lake and playing the violin, and this is when he’d have his breakthrough ideas. There’s certainly a balance to be struck between putting in the time on a hard problem and letting it sit, the focused and the diffused. I think a lot about where that balance lies.

Laying Foundation Blocks

I’m also grappling and coming to terms with the fact that while this is indeed the first startup, and while I’m shooting for a fully functional MVP product to release at the end of week four, this first startup is going to be so much more work just for the simple fact that I’m starting from square one with everything. Streaming, the Clojure language itself, developing the app, figuring out how to host a production app, figuring out how to host 12 production apps without completely breaking my bank or sinking all my time into server maintenance, balancing promotional activities with building, on and on and on. There is so much foundation that goes into all of this. Way more than my rose-tinted mind was able or willing to see before I started.

Makes me think that I would have been better off with just a little bit more preparation. Just a few more days where I spent some time laying more ground work so I could hit the ground running just a little bit faster. But what better teacher than the real thing, right? What better way to actually discover what those foundation blocks need to be than by just diving into it unprepared anyway.

Several lifetimes ago, I walked the Camino de Santiago, an ancient 500-mile pilgrimage across northern Spain, East to West. When it comes to preparing for the Camino, sure, you could train for it, but why not make the first couple days of the pilgrimage your training instead? The average distance you need to walk to do it in a month is 20 kilometers per day, every single day, and that is a hell of a lot of walking in the hot Spanish sun. How could you even train to walk 20 km a day? Wouldn’t you need to walk 20 km a day for like a week? That’s not just something you casually pop off after work, that’s five hours of walking at an average pace without breaks. And if you’re gonna be doing that much training walking, why not make the first week of the real thing the training session?

And hey, it’s early days right? I don’t want to jinx myself, but I think it’s healthy to view this first pass with low expectations. It’s a proof of concept, a test run, a shakedown cruise. What better time to make your mistakes, lay your foundation, and learn your lessons? And besides, I’ll have 11 more shots after this if the first one doesn’t land.

Community

One thing I’ve been very thankful for is the tiny little community I’ve built already. I’ve done the solo developer thing a couple times in my career, and it’s really no fun to work by yourself in a vacuum without anybody else around. It breeds paranoia. Much better to have some people around, even if I only get to see shadows of them in the form of stream chat messages and video view counts.

Last week I got one of the things I was really hoping to have, which was Clojurists in my chat helping me out with the code. I was writing a string-truncating function for the plot summaries – basically something to turn a big long string into a shorter string with a “…” at the end – and a viewer jumped right in with some comments when they saw I could make it better. You can check out our back-and-forth at this timestamp in the YouTube recording! Here’s the code we wrote together:

;; My first function, very much a product of my "Javascript" brain:
(defn truncate-plot [len plot]
  (str (join "" (take len (str/split plot #""))) "..."))

;; Second function, after working with person in chat a bit:
(defn truncate-plot-2 [len plot]
  (join "" (concat (apply str (take len plot)) "...")))

;; Final, refined version after more chat help:
(defn truncate-plot-3 [len plot]
  (apply str (concat (take len plot) "...")))

Building that function cooperatively with a viewer was a really special moment for me. If you don’t know the language, it might not look like much, but that last function is beautiful. It’s like poetry: a fairly involved micro-universe of meaning distilled into basic units and expressed in a basic form. So thanks dontlikecats, whoever you are. I hope to meet more like you along the way!

- Andy

Permalink

Clojure Developer

Permalink

A mental model for thinking about Clojure's Transducers

IntroductionClojure's documentation describes Transducers as composable algorithmic transformations. They are transformations which are independent of input or output and they compose without awareness of input or creation of aggregate values.When talking about transducers, we should look at the original thing that gave rise to this abstraction. The reductionA reducing function is essentially any function that you can pass to a reduce call. This function is supposed to take the result so far, generally known as the accumulator and the new value and generate a new result. So a reducing function isaccumulated-result, input -> accumulated-result While a transducer is the more general ...

Tags: Clojure, Functional Programming, Transducers

Permalink

Restore GPT/EFI partition table as well as grub on Linux

Programming Clojure, ClojureScript and Lisp.

GithubLinkedinTwitterKeybase

Site Map

Feeds

RSSAtom

GithubLinkedinTwitterKeybase

Site Map

Feeds

RSSAtom

Restore GPT/EFI partition table as well as grub on Linux

The Linux operating system, especially in the early days, was considered unforgiving: when you start fiddling with something there is a non-negligible chance that you will need a backup so that you can recover from your mistakes.

In this post, I wanted to share a bunch of commands that I found useful while moving my Linux box to encrypted partitions.

I am going to assume familiarity with the (linux) command line and the concepts of a partition table and a boot loader.

With great power comes great precaution

Changes to hard drive configuration cannot be performed while the system is running - you will need a bootable USB flash drive containing any of these Live Linux Distributions.

It is a wise choice to pick the one that matches what you have installed on disk if you are recovering from a previous mistake. For the purpose of this post, I will use Manjaro Live because I use Manjaro i3 Community Edition.

For writing the downloaded ISO to a USB drive I use isousb. Arch (on which Manjaro is based) gives you access to a plethora of great little hacks tools. Simple and effective.

Another thing you want to do before running any of the commands is to take a full-disk snapshot ¹ but I am not going to go deep into that.

Backing up a GPT partition table

After booting up Manjaro Live, I can use sgdisk ² for storing the GUID Partition Table (GPT) in a safe place:

sudo sgdisk --backup=/path/to/usb/gpt_$(date --iso-8601=date) /dev/sda

You can use the gdisk program family only with GPT partition tables and not all tables, especially legacy ones, are GPT ³

Restoring the partition table

Let's say one day while fooling around with gparted you inadvertently click on the "Create Partition Table..." menu button. Warnings are on the screen, but you persist in your mischief by applying the changes.

It goes without saying that even a seasoned Linux user makes mistakes. Fortunately, we have a backup that we can restore from:

sudo sgdisk --load-backup=/path/to/usb/gpt_20210101 /dev/sda

Easy-peasy reboot and...boom It does not boot.

The partition table is only half of the story, if you are using GPT it is likely your firmware is using the Unified Extensible Firmware Interface - UEFI for short.

Restoring UEFI requires access to the file system ⁴ - we need to chroot.

Chroot

The Arch wiki page has the best definition here:

A chroot is an operation that changes the apparent root directory for the current running process and their children.

We will mount all the relevant partitions to a directory (usually /mnt) and then execute commands from it. This step depends on how the partitions were organized in the first place (gparted can shed some light on this).

I have mainly seen two setups:

• /boot on one partition and root on another

  /           # /dev/sdX
  └── boot    # /dev/sdY and EFI System Partition
  

• only /boot/efi on a separate partition and root (including /boot on another)

  /           # /dev/sdX
  └── boot
      └── efi # /dev/sdY and EFI System Partition
  

The former is what I have got so these are the commands I needed to run:

mkdir -p /mnt
sudo mount /dev/sdX /mnt
sudo mount /dev/sdY /mnt/boot

cd /mnt
mount -t proc /proc proc/
mount -t sysfs /sys sys/
mount -o bind /dev dev/
mount -o bind /sys/firmware/efi/efivars sys/firmware/efi/efivars/

When your live distro contains helpers like arch-chroot or manjaro-chroot you can simply:

sudo mount /dev/sdX /mnt
mkdir -p /mnt/boot
sudo mount /dev/sdY /mnt/boot

sudo manjaro-chroot /mnt /bin/bash # I prefer bash over sh

Chroot and LUKS

An additional series of commands are necessary if your partitions are encrypted. Without going too much into details again, LUKS is the most common way to encrypt your disk on Linux.

If that is you, do the following:

sudo cryptsetup luksOpen /dev/sdX cryptoroot
sudo mount /dev/mapper/cryptoroot /mnt
sudo cryptsetup luksOpen /dev/sdY cryptoboot
sudo mount /dev/mapper/cryptoboot /mnt/boot

sudo manjaro-chroot /mnt /bin/bash

Restoring the EFI

Finally, we can fix our booting issues. After chroot-ing (you'll usually notice the different terminal prompt), execute:

sudo grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=Grub --recheck
sudo update-grub # this is optional but recommended

A gotcha to be aware of is that --efi-directory specifies where grub's EFI will be created, not the location of /boot/efi .

Consequently the resulting folder structure for a setup where /boot also serves an EFI System Partition (the first above) is:

/boot/
├── EFI
│   └── Grub # this matches the --bootloader-id
├── grub
└── ...

In the other scenario you will end up with something inside /boot/efi/EFI instead.

Finding safety in the dragon's nest

While technically you should be good to boot now, sometimes this still does not happen mainly because the computer firmware's boot manager does not know of the existence of EFI/Grub/grubx64.efi .

Once again, that is entirely dependent on the machine but on a Dell you will have to enter the BIOS setup (F12) and figure your way around "Boot Sequence" and "Add Boot Option".

This last piece of information also concludes the post, happy linuxing!

---

1. my favourite tool for that is Clonezilla.

   ↩

2. for way more detail check this very instructive post.

   ↩

3. check MBR vs GPT - Things to Know When Partitioning.

   ↩

4. I am not entirely sure about the correctness of this sentence but the commands work. Checking UEFI Booting also seems to confirm.

   ↩

Copyright © 2015-2021 Andrea Richiardi

GithubLinkedinTwitterKeybase

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 4.0 International license.

Powered by Perun and Fomantic UI.

Site MapRSSAtom

Permalink