Can Developers Use Zettelkasten?

In Flexiana, as a Clojure software company, developers use different methods to organize their knowledge. Zettelkasten is one of those methods. Zettelkasten is a particular way that you take notes, and then tag those notes to help with making connections. And from there, you can generate new ideas quickly. 

Nowadays we have access to so much information but how do we make sense of all this information? How do you link an idea you’ve found from a book to an insight you got from watching a documentary? That’s the whole idea behind the Zettelkasten Method. 

New knowledge is added to your existing knowledge, and this boosts your understanding of both. And this helps you generate new ideas. You can use Zettelkasten to make connections between the information you follow. 

What’s it about?

So, Zettelkasten literally means a box of notes. Niklas Luhmann used this method to publish at maximum efficiency. He published over 40 books and more than 600 papers in his career. It can show how powerful this method can be.  But how can you apply the Zettelkasten Method to your daily routine to become a creative machine?

You will need two things

  1. A way to take notes: you can use a pen and paper, or digital note-taking tools, like Notion
  2. A program with bidirectional linking: it can connect two ideas back and forth between each other, like Remnote

Think of yourself as an idea collector. When you are running some errands, reading a book, or attending a meeting, for example, new ideas come to your mind. Just write these new ideas down! When you write it down, make sure to include the source of where it came from.

For example, you are browsing the internet. You see an interesting quote by a famous person. You just write it in your Notes app including the name of that person. Throughout the day if you come up with new ideas you will write them down. Luhmann suggested avoiding copying and pasting. Instead, use your own words for a more conducive idea generation, but it might not work for everyone. Just try different ways yourself.

At the end of the day go back and look at all the notes you have collected. Explore your ideas and put tags on them, like hashtagging. 

Import all your notes into a bidirectional linking program, and put tags on them. There is no one way to tag. It should be customized and personalized to you. Some people might categorize them, some might try to relate them to other ideas from the past. It is better if your tags are descriptive and not general, but still, it can be any way you want.

Now it’s time to figure out which one of these notes you want to keep. You don’t have to keep all of your notes, just keep the best ones. You can decide by comparing your new daily notes with the ones you already have. Just go through your tags and see the ideas under those tags. Try to see the connection you make between your ideas under each tag. You can also use the source of your ideas to get a better understanding. This also can help you with generating new ideas. 

After adding some more ideas, you can write a blog post or make a speech, or create a new product using these ideas and tags.

  1. Take notes on the go
  2. Tag your notes
  3. Which daily notes are worth keeping?
  4. Start working on your ideas

Using the Zettelkasten Method, you will have a more systematic way to store, organize and connect information, and generate a lot of new ideas quickly. 

There are only two rules to follow:

  1. Every idea that you write down should be clear and understandable.
  2. Every concept that is written should be linked to a card that has its explanation. The explanation card should have a link back to that concept card.

Zettelkasten and developers

Researchers and students mostly use Zettelkasten, but anyone can use it, as its principles are adjustable.

If you are a developer and you want to learn a new language, scale up your skills or work on a new project, you can use Zettelkasten.

With the rules mentioned above, you can interconnect all your knowledge of development skills. With Zettelkasten you can create two different cards for the syntax of a language and its explanation, to make it easier to understand. This can also apply to methods by breaking them down to their fundamentals and writing them in one card. You can later link the fundamentals to the cards that later explain them. Breaking down to the most basic level can lead to variables and data types.

You can also connect problems, syntax, and functions to different solutions. This way you will have a more broad understanding. Having all the information on cards can give you a total image which can result in helping you with understanding, remembering, and visualizing your knowledge.

Also, when doing research on performing a function, you will get different results from forums, sites, and other materials. Using Zettelkasten, you can put the bits of your findings together to help out. 

You can also include snippets of your codes. Write the snippets on cards. Link them to the concepts you want, including where those snippets have been used. Then, link the snippets to the related ones. Later you can add more cards and snippets to this stack.

You can be creative and match Zettelkasten with GTD. The first is for knowledge management and GTD is for task management. 

There are different applications that you can use to set up your Zettelkasten system. You can also be creative and use command lines to have your own very system like Bob from PyBites.

All in all, Zettelkasten can be used for developers when they want to know:

  • How to do something
  • How a specific thing is implemented in a framework, method, language, etc.
  • The definition of different concepts, etc.
  • Anything knowledge-based.

Later on, you might be able to share your system with your colleagues or anyone. Just don’t forget to update your Zettelkasten system.

Using different methods our Clojure developers make the world go round. If you like our senior developers to collaborate with you, have a remote coffee with us.

The post Can Developers Use Zettelkasten? appeared first on Flexiana.

Permalink

Using nREPL as System Interface

Clojure REPL is a powerful tool for developing programs interactively. Connecting the editor to the REPL allows us to get instant feedback on the code we're writing and have confidence that it works as intended as we're developing the application. However, the REPL isn't inherently limited to application development. It provides us with an interface to the language, and the language in turn be used as an interface to the host system. Let's take a look at how we can use Babashka along with Osquery to inspect the state of the host.

Osquery is a handy tool that allows using SQL commands in order to leverage a relational data-model to describe a device. Different aspects of the system are mapped to relational tables using the following schema. The tables give us access to files, ports, mounts, and many other aspects of the system. One aspect of Osqeury that's particularly useful to us is that it's able to return results in JSON format that we can parse into EDN and work with as structured data in the REPL.

To see how this works we'll start the nREPL server by running bb --nrepl-server. The REPL will start on port 1667 by default, we can also set a custom port by providing port number as the second argument to bb. Once the REPL is running we can connect any nREPL compatible editor such as Calva or Emacs.

Let's create a file called osquery.clj and open it in the editor and add some code to drive Osquery. First thing we'll need to do is to require the namespaces for interacting with the shell and parsing JSON:

(require '[clojure.java.shell :refer [sh]]
         '[cheshire.core :as json])

Next, we'll define the osquery function that will take a SQL query as text, execute osqueryi command and return its result as EDN:

(defn osquery [query]
  (let [{:keys [exit out err]} (sh "osqueryi" "--json" query)]
    (if (zero? exit)
      (json/decode out true)
      (throw (Exception. err)))))

And we're now ready to try to query some information about the system. Let's run a query to see all the routes where destination is ::1

(osquery "select * from routes where destination = '::1'")

We should get back a list of routes that looks something like the following:

({:hopcount "0",
  :interface "lo0",
  :mtu "16384",
  :type "local",
  :source "",
  :gateway "::1",
  :netmask "128",
  :flags "2098181",
  :destination "::1",
  :metric "0"})

The result is just a plain Clojure data structure we can trivially manipulate using full power of Clojure.

We can even go a step further using HoneySQL library that will allow us to make structured queries. We'll need to require deps and pull in the library as follows:

(require '[babashka.deps :as deps])

(deps/add-deps '{:deps {com.github.seancorfield/honeysql {:mvn/version "2.2.861"}}})

(require '[honey.sql :as hsql])

Finally, we'll update our osquery function as follows:

(defn osquery [query]
  (let [{:keys [exit out err]} (apply sh "osqueryi" "--json" (hsql/format query {:inline true}))]
    (if (zero? exit)
      (json/decode out true) 
      (throw (Exception. err)))))

With the changes above we can now write our queries in EDN:

(osquery {:select [:*] :from [:routes] :where [:= :destination "::1"]})

I hope this little example illustrates how the REPL can be used as a powerful OS interaction tool as well as a programming tool and inspires you to use the REPL in new and exciting ways. Babashka in particular is a great tool for such REPL driven interaction due to fast startup and wide range of useful libraries that let us access databases, HTTP servers, and other resources. This makes Babashka an excellent tool for doing devops.

Permalink

Advance Your Team’s Agility with Clojure Stack

Do you not know how to organize your work such that it allows you to be productive and effective while retaining your motivation and passion? If you are concerned about disappointing your customers, deviating from your strategic course, or falling short of your goals because your staff isn’t flexible enough. You’ve come to the right place, as we’ll be sharing the knowledge of our professionals with you. The topics highlighted include the bus factor, the project’s benefits, and disadvantages, using the Clojure stack on our experience, and the language’s speed, agility, and simplicity of use.

Today’s mission is to comprehend software product delivery. The aim is an agile team, and we’ll highlight all the different variations of going ahead since that’s been mainstreaming software delivery for a long time.

First, you need to know a term as the bus factor that’s what we will come across while explaining one of the main challenges in keeping the team agile, flexible, and high performing. The bus factor is the number of team members who might be hit by a bus before the project is substantially jeopardized. The lower the number, the higher the risks.

What should be done to raise the bus factor?

  • Writing documentation. It is one of the things how to deal with the bus factor on many projects.
  • Review code of each other. A code review is required, and you should attempt to adhere to it and ensure that it is not only a project checkbox activity.
  • Pair programming. Try to stick to pair programming that gives you better communication and understand where the like gaps in communication are or what the kind of document should be.
  • Do daily standups or meetings. Attempt to keep on the same page with meetings and not overwhelm people.
  • But there is one more important item, and you need to focus more on it: interchangeability. It’s incredibly helpful and vital to have an interchangeable team to improve communication, offer clarity that documentation or meetings may not, and make sure everyone is on the same code page.

The next helpful item that helps in the guidance of business is how to not bench people. There are different ways to avoid punching people and we kind of rely on a very large team. That has very granular duties and everyone should kind of learn a lot of high-level information about the business by performing granule duties and so on. Of course, it is tried the option to hire more staff on the project, but the healthy bench is around like having around 5-8% of the employees on it. It always costs extra money.

Also, the bench is the extra costs that are not always healthy to the project kind of effectiveness for the company or the product effectiveness for the company. A need to have a table load. There is always a need to have a stable load to manage the healthy bench that would help in the bus factory situations, and it brings higher risks in the situations that are the extra product or extra projects that would happen the certain group of people or company that operates in a certain environment.

Communication and interchangeability are the primary objectives of overcoming obstacles such as recruiting additional personnel, benching workers, and accumulating hazards. Consequently, we determined that the most effective and least expensive approach would involve people and a whole stack.

Let’s move on to the positives and drawbacks of using Clojure stack for projects.

Benefits of using Clojure stack

  • Proper handling of Bottle Necks
  • Reusable Codebase (allows you to jump between different platforms in projects)
  • Faster Delivering
  • Stay Agile and Flexible

Disadvantages of using Clojure stack

  • High entry-level (having a basis in functional engineering, you must be a lisp-oriented engineer or join the stack)
  • Not that many developers on the market
  • The ecosystem doesn’t have critical mass yet
  • Not a perfect pick for small scripts

Our model and experience with Clojure

Using a group of Clojure and Clojure Script full-stack engineers on our squad’s projects decreases the bus factor and increases project agility.

Set of Clojure/Clojure Script Full-Stack Developers. Try utilizing the same stack for all purposes by leveraging a common set of libraries, a great Clojure feature. It varies depending on whether it’s used with Java or JavaScript and on which platform, but you don’t have to rebuild the vehicle every time.

The same stack for every purpose and shareable libraries would be these three essential items that we have underlined here. Apart from that, it’s very cool by the very nature of the language Clojure and Clojure Script combination.

Clojure stack improves quality, speed, and communication. A minor team of full-stack replaceable engineers can offer more and quicker, which is why Clojure is great for startups. Engineers that specialize in Clojure are often the ones who have worked in this field with other stacks. Usually, Clojure engineers are more experienced than newcomers to the area. Clojure developers are a close-knit group because they speak the higher-level language and have a piece of better knowledge. This leads to solid cooperation and if required, solo effort.

Example of using Clojure stack in our project

We were tasked with constructing a communication platform for our clients, which included the front and back ends and several integrations. We were working on multi-channel messaging for large recipient groups. This implies high load, considering performance and spikes, and a host of other concerns that will bring up JVM and Clojure as the stack that may be used for highly loaded systems.

The Agiliway team was tasked with developing an intuitive UI wizard for setting up communication campaigns. Also, were supposed to deliver the dynamic front end templates that would allow for an easy setup for our target customer group, and the system’s SMS/E-emails were supposed to be highly secured and compliant with various standards for transferring PII/PHI data, as the system was designed for communicating with various recipients of health care or fintech-related information.

The development of a secure platform employing the Clojure stack received much attention. From the time it was created until the time it was released, the platform was built by a team of developers in around four months.

It’s estimated that three constant engineers and a fourth shifted between various duties. Consequently, you also have this vital possibility. A developer may easily switch between minor projects. Because the stack is the same everywhere he works, he thoroughly understands the project from which he was taken. Since they can read the same code, they don’t detect differences and understand it completely. Changes or fixes are possible. Since they utilize a different language or technology, they don’t need to ask you. It’s the same for everyone and decreased throughout our talk.

To achieve the top, the most important tool is self-development; you do not need to confine yourself to a certain industry or application. Contact us if you need Clojure/Clojure Script courses or Clojure-based solutions, and our experts will consult with you on your desire. Don’t give up if you want to learn Clojure, it’s simple to learn but difficult to master. Gain agility with the Clojure stack!

READ ALSO: COMPLEX CRM SOLUTION FOR CLIENT MANAGEMENT AND DATA ANALYTICS

The post Advance Your Team’s Agility with Clojure Stack first appeared on Agiliway.

Permalink

Testing babashka scripts

For testing babashka scripts, you can write your own test runner from scratch, which is easy enough:

(ns my-test.runner
  (:require
   [clojure.test :as t]))

(def test-namespaces '[my-test])

(defn -main [& _]
  (doseq [test-ns test-namespaces] (require test-ns))
  (let [{:keys [fail error]}
        (apply t/run-tests test-namespaces)]
    (when (and fail error (pos? (+ fail error)))
      (System/exit 1))))

and then run it with bb -m my-test.runner.

Not too bad, but still, it's work and boilerplate and even more so when you want to make a runner with CLI argument parsing to support running a subset of your tests. Since a year or so, you can use the cognitect-labs/test-runner with babashka. But this required a fork of tools namespace to be on your babashka classpath (using :deps in your bb.edn file).

No more! Since babashka version 1.0.166 you can use org.clojure/tools-namespace unmodified. The fix for this was to add the clojure.tools.reader namespace with the read function in babashka as a built-in. Babashka doesn't support the whole clojure.tools.reader namespace yet, but this is a good start to make it compatible with tools.namespace and now also the cognitect test runner.

To use it with babashka, add the following to your bb.edn.

{:tasks
 {test:bb {:extra-paths ["test"]
           :extra-deps {io.github.cognitect-labs/test-runner
                        {:git/tag "v0.5.1" :git/sha "dfb30dd"}}
           :task (exec 'cognitect.test-runner.api/test)
           :exec-args {:dirs ["test"]}
           :org.babashka/cli {:coerce {:nses [:symbol]
                                       :vars [:symbol]}}}}}

The exec function call, :exec-args and :org.babashka/cli coercion is there so we can call a Clojure function from the command line. See Babashka tasks meets babashka CLI for more details.

Now create a test file in test/my_test.clj:

(ns my-test
  (:require [clojure.test :refer [deftest is testing]]))

(deftest my-first-test
  (testing "equality works"
    (is (= 1 1))))

(deftest my-second-test
  (testing "equality still works"
    (is (= 2 2))))

And run the tests:

$ bb test:bb

Running tests in #{"test"}

Testing my-test

Ran 2 tests containing 2 assertions.
0 failures, 0 errors.

To run a single test you can specify the name of a var:

$ bb test:bb --vars my-test/my-second-test

Running tests in #{"test"}

Testing my-test

Ran 1 tests containing 1 assertions.
0 failures, 0 errors.

$ bb test:bb --vars my-test/my-first-test my-test/my-second-test

Running tests in #{"test"}

Testing my-test

Ran 2 tests containing 2 assertions.
0 failures, 0 errors.

Perhaps this will come in handy for Advent of Code 2022!

Permalink

Configuration

Which library do you use for project configuration? There are so many to choose from. Take a look at The Clojure Toolbox under “Configuration”. Choosing the right one mostly comes down to that you want out of the configuration library. Here are my requirements: It should not maintain it’s own state. So environ is out. It should allow the project to control configuration sources (e.g. from environmental variables, java properties, command line arguments, file (edn?

Permalink

Founding engineer Arc Studio – GitHub for Hollywood at Arc Studio

Founding engineer Arc Studio – GitHub for Hollywood at Arc Studio


  • We're building a real-time collaborative screenwriting app. North-star vision is to be come a collaboration-first Google Docs (think GitHub for normal text)
  • Lots of interesting challenges: We're building documents with real-time editing but also branching and merging (like git)
  • Traction in B2C (paying customers)
  • Arcane (the Netflix show) used us in their writers' room (!!)
  • competitive pay + good equity an option
  • we're fully remote
  • we're looking for somebody who thrives in a fast-paced, start-up-y environment and is happy to go the extra mile as needed
  • good frontend/browser knowledge required, backend/Postgres/AWS knowledge a plus

Please apply here: https://tally.so/r/npbYay Or email michi at arcstudiopro dot com

Permalink

Biff tutorial: build a chat app step-by-step

Last night I finished writing the last section of Biff's new official tutorial. This project was funded by a grant from Clojurists Together. A huge thank you to them and all their donors, as well as to JUXT and others for their ongoing sponsorship of Biff.

The new tutorial takes you through the process of building an app called eelchat.

eelchat is like a barebones version of Slack, Discord, and the like. From the introduction:

In eelchat, users can create communities, each of which has a collection of channels for text chat. Channels can only be created and deleted by the user who created the community (i.e. the admin). The admin can also add RSS subscriptions to channels, so new posts will be displayed.

In building eelchat, you'll get a tour of all of Biff's main parts, including:

  • Creating and deploying a new project (I like to get new apps in production right away)
  • Rendering pages with Rum and Tailwind
  • Modeling your app's domain with Malli schemas
  • Doing CRUD with XTDB and htmx
  • Pushing updates to the client with transaction listeners, htmx, and hyperscript
  • Handling background jobs with scheduled tasks and in-memory queues

All of the code is available on GitHub, with a separate commit for each section. Links to the commits are included throughout the tutorial. And at the end there are suggestions for additional features you can add to eelchat, so you can get a chance to spread your wings. 

A few people over in #biff on Clojurians Slack have already started working through the tutorial as I've been releasing the sections over the past weeks, and I haven't been alerted to any glaring faults yet. I did do some refactoring last night though, and I don't have it in me to go through the entire tutorial myself just yet, so let me know if you run into any snags.

I am planning to update the code formatting. Most of the code blocks were pasted verbatim from git diff, which is a quick way to write a tutorial though perhaps not the pinnacle of UX. You'll have to delete a bunch of +s at least. (Maybe I'll finish that tonight and render this paragraph obsolete... but if you're reading this, that didn't happen.)

Now that the tutorial is finished, my next order of business is to get Platypub to a state where I can run a publicly available instance of it. When that's finished, Biff will have not just a fabulous tutorial, but also a killer app that anyone can try out easily. Part of my grand scheme is still to have Platypub become a hook that brings more people into the Clojure ecosystem—e.g. custom themes in Platypub are just Babashka scripts.

Concurrently, I'll continue to write more documentation. I'd like to add a page with links to resources for learning Clojure, and I've got a handful of how-to articles I'd like to write (such as "how to do pagination with htmx," "how to receive email," and "how to scale out"). In general my plan is to spend my time roughly 50/50 between writing open-source apps with Biff and writing documentation. (I'd also like to start doing meetups again, but I'm going to keep those on hiatus at least until the end of the year.)

If there's anything you'd like to see in Biff world, let me know. And remember...

Permalink

Clojure Deref (Nov 23, 2022)

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem. (@ClojureDeref RSS)

Highlights

In this week of thanks, thank you to the Clojure community! It is a pleasure to present your work every week here and always a surprise how much the community does every single week.

Podcasts and videos

Blogs and articles

Libraries and tools

New releases and tools this week:

  • dev.datafy 0.1 - Dev-time datafy and nav

  • pedestal 0.5.11-beta-1 - The Pedestal Server-side Libraries

  • antq 2.2.962 - Point out your outdated dependencies

  • edamame 1.0.16 - Configurable EDN/Clojure parser with location metadata

  • nbb 1.0.136 - Scripting in Clojure on Node.js using SCI

  • hickory 0.7.2 - HTML as data

  • antq 2.2.962 - Point out your outdated dependencies

  • fulcro-rad 1.3.9 - Fulcro Rapid Application Development

  • fulcro-rad-semantic-ui 1.2.18 - Semantic UI Rendering Plugin for RAD

  • calva 2.0.320 - Clojure & ClojureScript Interactive Programming for VS Code

  • joyride 0.0.23 - Making VS Code Hackable since 2022

  • scittle 0.4.11 - Execute Clojure(Script) directly from browser script tags via SCI

  • select-tree 0.1.0 - Clojure library to recursively select subtrees of collections

Permalink

in which legibility comes at a price

It's been a weird few weeks for decentralized social media. The collapse of Twitter at the hands of an incompetent billionaire has led to a massive wave of users on the Fediverse1, from under a million to over seven million now. And at the time of this writing Twitter hasn't even finished collapsing yet; the site is still hobbling along, albeit with gaping holes. Though by the time I've finished writing this post, who knows. A lot can happen.

a graph of new users per day overlaid with various events leading to Twitter's demise; each disaster causes a spike.

As you might expect, the huge influx has caused some technical problems to the Fediverse. The mega-instances like mastodon.social saw slow page load times and hours of lag between something posted and it propagating out to other servers. More reasonably-sized servers have still seen some lesser propagation lag, but the main effect of this influx has been nontechnical. New users have had a hard time finding well-run servers that are accepting new members, and there has been a lot of complaining that the system is hard to understand or navigate due to its distributed nature.

whenever twitter makes a bunch of its users angry we get a new wave of activity on the [Fediverse] and all the new mastodon users engage in the primary hobby of a new mastodon user, which is listing all of the reasons why mastodon doesn't work. it's real fun

- @rootsworks@mastodon.social

Some of this is simply a necessary difference between a single unified corporate system to a distributed network. In the 1990s we saw a similar type of confusion as people left silos like America OnLine (AOL) and CompuServ to explore the early days of the World Wide Web. It wasn't as polished, and there was no central view of the entire network. You had to do more exploring to find independent communities that ran their own sites. People who remember making this shift know now that the effort was worth it, but back then you couldn't fault people for wondering why things couldn't just be easy like it used to be.

But I believe there's more to the situation with the Fediverse today than simply the differences that arise out of technical necessity.

I recently finished reading the book Seeing Like a State by James Scott. There's a lot to unpack in this book2, but I want to talk about the way it defines the term legibility. Legibility describes the degree to which a place is understandable and navigable to an outsider.

a street near the market in Mae Sot, Thailand

My favorite example of legibility is street addresses. When I lived in Thailand I was initially very surprised to learn that even long-time residents had a very rough grasp of the names of most of the streets in the city. People navigated by landmarks instead. As a newcomer to the city it was very disorienting; I had to set aside my ideas I brought in about how to navigate and learn to read the city like a local. In the end after living there two years I knew the names of four or five of the major streets, but when someone wanted to know where my house was, I didn't give them a street address; I told them it was a block east of the bus terminal.

This attitude towards navigating looks different depending on whether you're an outsider or a resident. To a resident it's no big deal, but to someone who's new to the city it can be frustrating and bewildering. In some cases (for instance, cities that rely on tourism) there are incentives for the locals to increase legibility, but other factors can incentivize people to reduce their legibility.

Legibility is particularly interesting when it lies at the crux of a power imbalance, and the book describes many such situations. As pre-modern states in Europe consolidated their power, they looked for ways to collect taxes and conscript soldiers more effectively. Customary naming made this very difficult.

Only wealthy aristocrats tended to have fixed surnames…Imagine the dilemma of a tithe or capitation-tax collector [in England] faced with a male population, 90% of whom bore just six Christian names (John, William, Thomas, Robert, Richard, and Henry).

Seeing Like a State by James Scott

In order to consolidate their power over their citizens, states rolled out policies to increase the legibility of the population they ruled over; they started to require that everyone take a surname in order to uniquely identify them to the state in censuses and tax documents. As you can imagine, this didn't go over well; the historical record documents intense resistance, in some cases even open rebellion. The people themselves had no need for surnames; they got along fine with a given name supplemented when necessary with contextual specifiers. "John" might be "John the Baker" in some contexts and "Short John" in others and "John Underhill" to people in the next town. This system worked fine for everyone involved, except those wishing to exert their authority from the outside.3

If this sounds familiar to you, it might be because you were paying attention to social media controversies of the last decade! Prominent advertising companies like Facebook and Google attempted to roll out tremendously unpopular policies requiring users to identify themselves using their legally-documented name4. Though these policies were always accompanied by some flimsy justification about "user safety", the real goal was to make their users more legible, primarily for advertising purposes, but also to law enforcement and other authorities.

Well, it turns out just like villagers in premodern France, lots of users don't want to make themselves more legible! Even setting aside the impossibility of a company like Facebook accurately identifying the difference between a pseudonym and a legal name, there are a lot of great reasons to not want to be easy to find. Avoiding targeted advertising is just the tip of the iceberg; ask any member of a marginalized group and they'll tell you that being easy to find can have a high cost when it exposes you to targeted abuse.

Twitter features full-text search of everything posted. This is fantastic for legibility; if you're curious about a topic, you can always see who's talking about it. One of the ways this gets used on Twitter is to pick fights. Trolls find a hot topic and barge uninvited into conversations to yell at people who disagree with their position. On the Fediverse, search is opt-in instead of opt-out. Normal text doesn't get indexed, but hashtags do. If you want your post to become searchable under certain terms, a # is all it takes, but the randos won't show up unless you go out of your way to invite them.

In some ways, people today coming to the Fediverse from corporate social media remind me of how I felt when I first moved to Thailand—lost and bewildered by the lack of street signs. They've spent a lot of time on the highways of platforms that prioritize "engagement" at all cost, and it's taking a while for it to sink in that not all the Internet works that way. They've never considered that illegibility can be a defense mechanism..

Become illegible.5

[1] The Fediverse is a network of interconnected social media servers that exchange posts. Most of its usage is based on a similar model to Twitter, but there are some systems on it that focus on photos or videos. Because the most common Fediverse server is called Mastodon, many people use the word Mastodon to talk about the network, similar to how in the 1990s people said "Internet Explorer" when they meant the web.

[2] While the book is insightful, it is pretty long, and the two case studies in the middle drag on a bit. I strongly recommend reading the first three chapters and the last two chapters, but you can still get 90% of the insight by skipping the two very detailed case studies in the middle.

[3] The scary thing about this is how absolute the normalization of surnames became over time despite the intense resistance. In the end, people accepted the greater control, and even tho surnames were rare in many societies only a few hundred years ago, nowadays they are thought of by most people as "natural".

[4] These policies often misleadingly referred to the legal name as the "real name", as if legal process had some magical access to reality itself which could not be achieved using the customary naming patterns people have been using for millennia.

[5] I have to admit to stretching this a bit for a punchy closing line, but of course it's not as simple as "legibility bad"; obviously there are many cases in which you want legibility. I'm trying to convey the fact that legibility does have downsides, and if you've never even considered this, then it's impossible for you to have a productive conversation about social software in today's context.

Permalink

The last Python Architecture you will ever need?

Should features live in several independent Microservices, or is a Monolith the way to go? What kind of setup is the best one for the code in a repo? Organizing is difficult.

"... I'll just put it in a utils folder for now ..."

There's a thing called Polylith. I've written about different aspects of it before. Polylith is an architecture (and a tool) focusing on the Developer and Deployment experience. It is developed by Joakim Tengstrand, Furkan Bayraktar and James Trunk.

The Polylith Architecture is a fresh take on how to share code, and offers a nice and simple solution to the Monolith vs Microservice tradeoffs. In addition to that, it is a good fit for functional programming. Some time ago, I got an idea: how about bringing a couple of the good things from there to here? So I developed something that I believe could be useful for many Python teams out there.

I already released a preview in early 2022, it was missing some essential features and the great developer experience wasn't really there yet. Also, I had little knowledge about how to package Python apps & libraries, but have learned a lot since then.

Today, I believe the Python tools for the Polylith Architecture is ready to use in the Real World. You will find version 1 on PyPI.

A couple of Poetry plugins

I have developed it as two different Poetry plugins. One of them - the Multiproject plugin - adds support for workspaces to the popular Poetry tool, by adding a new command called build-project.

The second one - Polylith plugin - adds useful tooling support for the Polylith Architecture itself. With the tooling, you can add components & projects in a simple way and also keep track of what's happening in the workspace.

Have a look at the repo and the docs.

An Architecture well suited for Monorepos

Polylith is using a components-first architecture. The components are building blocks, very much like LEGO. The code is separated from the infrastructure and the building of artifacts. This may sound complicated, but it isn't.

In a way, it is about:
  1. thinking about code as LEGO bricks, that can be combined into features.
  2. making it easy to reuse code across apps, tools, libraries, serverless functions and services.
  3. Keeping it simple.
Have a look at these introductory videos, describing Polylith in Python and the tooling support:


Python with the Polylith Architecture



The Poetry Polylith Plugin


Give it a try! I would love to hear you feedback.



Top photo by frank mckenna on Unsplash

Permalink

The last Python Architecture you will ever need?

Should features live in several independent Microservices, or is a Monolith the way to go? What kind of setup is the best one for the code in a repo? Organizing is difficult.

"... I'll just put it in a utils folder for now ..."

There's a thing called Polylith. I've written about different aspects of it before. Polylith is an architecture (and a tool) focusing on the Developer and Deployment experience. It is developed by Joakim Tengstrand, Furkan Bayraktar and James Trunk.

The Polylith Architecture is a fresh take on how to share code, and offers a nice and simple solution to the Monolith vs Microservice tradeoffs. In addition to that, it is a good fit for functional programming. Some time ago, I got an idea: how about bringing a couple of the good things from there to here? So I developed something that I believe could be useful for many Python teams out there.

I already released a preview in early 2022, that was missing some essential features and the great developer experience wasn't really there - yet. Also, I had little knowledge about how to package Python apps & libraries, but have learned a lot since then.

Today, I believe the Python tools for the Polylith Architecture is ready to use in the Real World. You will find version 1 on PyPI.

A couple of Poetry plugins

I have developed it as two different Poetry plugins. One of them - the Multiproject plugin - adds support for workspaces to the poular Poetry tool, by adding a new command called build-project.

The second one - Polylith plugin - adds useful tooling support for the Polylith Architecture itself. With the tooling, you can add components & projects in a simple way and also keep track of what's happening in the workspace.

Have a look at the repo and the docs.

An Architecture well suited for Monorepos

Polylith is using a components-first architecture. The components are building blocks, very much like LEGO. The code is separated from the infrastructure and the building of artifacts. This may sound complicated, but it isn't.

In a way, it is about:
  1. thinking about code as LEGO bricks, that can be combined into features.
  2. making it easy to reuse code across apps, tools, libraries, serverless functions and services.
  3. Keeping it simple.
Have a look at these introductory videos, describing Polylith in Python and the tooling support:


Python with the Polylith Architecture



The Poetry Polylith Plugin


Give it a try! I would love to hear you feedback.



Top photo by frank mckenna on Unsplash

Permalink

Python: To OOP or to FP?

That is the question

Photo by Susan Holt Simpson on Unsplash

Programmers can never agree on anything, but by far one of the biggest arguments that constantly plagues the interwebs is the battle between object-oriented programming (OOP) and functional programming (FP).

As a reminder, OOP revolves around wrapping all your business logic and data in classes, which can then create objects that share the same functionality. It also includes concepts such as inheritance and polymorphism, which make it easier to have classes with similar, but slightly altered functionality.

The language usually used to demonstrate OOP is Java. In Java, everything must be wrapped in a class, including your program’s main execution loop.

Functional programming, on the other hand, is more concerned with — you guessed it — functions. In functional programming, data is often piped from function to function, each performing an operation on the data. Functions are often designed to produce the exact same output if given the same input.

The most popular functional programming languages are Clojure, Elixir and Haskell.

But what about Python?

Python is an interesting case. It shares a lot of the common features of object-oriented languages, allowing you to create classes and inherit from superclasses, but it also has functionality that you’d normally see in functional languages. You’re allowed to define functions in the main body of the program, and functions are also first-class citizens, meaning you can pass them about as objects.

The truth is, Python is very flexible. If you’re coming from Java and want to write everything in a purely object-oriented style, you’ll be able to accomplish most of the things you want. If you’re a previous Clojure developer, you won’t have too much trouble replicating FP patterns in Python either.

However, the beauty of Python is that you’re not restricted to either way of doing things. You can use features of both paradigms to create readable, extensible code that will keep your codebase maintainable, even as it grows.

Below are three examples of the same (very simple) program written in OOP, FP and a more Pythonic mixture of both. I’ll highlight the strengths and weaknesses of each one, which should give you a good base when architecting your next Python project.

The program

The program used to demonstrate is very simple — it creates two animals (a dog and a fish) and has them perform some very simple actions. In this example, the actions just log to stdout, but they could obviously do a lot more.

OOP example

from abc import ABC, abstractmethod


class Logger(ABC):
    @abstractmethod
    def log(self, message: str):
        ...


class MyLogger(Logger):
    def __init__(self, name: str):
        self.name = name

    def log(self, message: str):
        print(f'{self.name}: {message}')


class Animal:
    def __init__(self, name: str, logger: Logger):
        self.name = name
        self.logger = logger

    def speak(self):
        self.logger.log('Speaking')
        ...


class Dog(Animal):
    def speak(self):
        self.logger.log('Woof!')
        ...

    def run(self):
        self.logger.log('Running')
        ...


class Fish(Animal):
    ...


class App:
    @staticmethod
    def run():
        fido = Dog(name='Fido', logger=MyLogger('Fido'))
        goldie = Fish(name='Goldie', logger=MyLogger('Goldie'))

        fido.speak()
        fido.run()

        goldie.speak()


if __name__ == '__main__':
    App.run()

# Fido: Woof!
# Fido: Running
# Goldie: Speaking

As you can see, the code creates a MyLogger class for logging events to stdout , an Animal base class and then Dog and Fish classes for more specific animals.

To follow the OOP paradigm a little more closely, it also defines an App class with a single method run that runs the program.

The nice thing about OOP and inheritance is that we didn’t have to define a speak method on the Fish class and it’ll still be able to speak.

However, if we wanted to have more animals that could run, we’d have to introduce a RunningAnimal class between Animal and Dog that defines the run method, and potentially a similar SwimmingAnimal class for Fish , but then our hierarchies start getting more and more complicated.

Also, the MyLogger and App classes are pretty much useless here. Each does only one thing and actually makes the code slightly less readable. These would be better pulled out into a log and a main (or run) function.

We’ve also had to create a Logger abstract base class purely so that the code can be properly type hinted and allow users of our API to pass in other loggers if they want to log to somewhere other than stdout, or if they wanted to log with a different format.

FP example

Just a head’s up — I’m less familiar with FP than OOP, so this might not be the most FP-like way to implement this behaviour, but it’s what I’m going with.

import functools
from typing import Callable

Logger = Callable[[str], None]


def log(message: str, name: str):
    print(f'{name}: {message}')


def bark(
    name: str,
    log_fn: Logger,
) -> (str, Logger):
    log_fn('Woof!')
    return name, log_fn


def run(
    name: str,
    log_fn: Logger,
) -> (str, Logger):
    log_fn('Running')
    return name, log_fn


def speak(
    name: str,
    log_fn: Logger,
) -> (str, Logger):
    log_fn('Speaking')
    return name, log_fn


def main():
    run(
        *bark(
            'Fido',
            functools.partial(log, name='Fido'),
        ),
    )

    speak(
        'Goldie',
        functools.partial(log, name='Goldie'),
    )


if __name__ == '__main__':
    main()

# Fido: Woof!
# Fido: Running
# Goldie: Speaking

Off the bat, we can see that our Logger class has just become a handy type alias for Callable[[str], None]. We’ve also defined a log function to deal with our printing. Instead of defining classes for our animals, we’ve simply defined functions that take in the name of an animal and a Logger function.

You’ll notice that the run, speak, and bark functions also all return their name and logging function arguments to allow them to be composed together into pipelines as we’ve done for run and bark for Fido.

We’ve also moved our logic into a main function, removing the need to define an entire class just to run our program.

To get around the fact that our log function doesn’t match the Logger type, we’re usingfunctools.partial to create a partial function that does match. This allows us to replace our logger with anything we like, as long as we can use a partial function to reduce it so that it matches our Logger type.

However, since we’re not encapsulating the data in anything, if we wanted to add more attributes to our animals, we’d probably have to start using dict objects to represent them and pass those around, but then there’s always a worry that the dictionary is created incorrectly, and thus is missing a key that’s relied on in one of our functions.

To get around that, we’d need to create initialiser functions for our animals, at which point the code gets messier and messier again.

A little bit of both

So, what would happen if we were to combine a bit of OOP with a bit of FP? I’m going to bring in a few more Pythonic bits and pieces to pull away from the traditional OOP and FP paradigms, and hopefully make the code a little cleaner and easier to read.

from dataclasses import dataclass
from functools import partial
from typing import Callable

Logger = Callable[[str], None]


def log(message: str, name: str):
    print(f'{name}: {message}')


@dataclass
class Animal:
    name: str
    log: Logger

    def speak(self):
        self.log('Speaking')


@dataclass
class Dog(Animal):
    breed: str = 'Labrador'

    def speak(self):
        self.log('Woof!')

    def run(self):
        self.log('Running')


@dataclass
class Fish(Animal):
    ...


def main():
    fido = Dog('Fido', partial(log, name='Fido'))
    goldie = Fish('Goldie', partial(log, name='Goldie'))

    fido.speak()
    fido.run()

    goldie.speak()


if __name__ == '__main__':
    main()

# Fido: Woof!
# Fido: Running
# Goldie: Speaking

In this example, I’m using Pythons dataclasses module to avoid having to write constructors for my classes. This not only reduces some of the code I need to write but also makes it a lot easier to add new attributes down the line if I need to.

Similar to the OOP example, we have an Animal base class with Dog and Fish subclasses. However, like in the FP example, I’m using the Logger type alias and functools.partial to create the loggers for the animals. This is made easier by Python’s support for functions as first-class citizens.

Also, the main function is just a function. I’ll never understand why Java is how Java is.

Mixing OOP and FP in production

Okay, so I’ll admit that this example was incredibly basic, and while it served as a good starting point for this discussion, I’d now like to give you an example of how these concepts are used in production, and I’m going to use two of my favourite Python libraries: FastAPI and Pydantic. FastAPI is a lightweight API framework for Python, and Pydantic is a data validation and settings management library.

I’m not going to go into these libraries in detail, but Pydantic effectively allows you to define data structures using Python classes, and then validate incoming data and access it via object attributes. This means you don’t run into the problems that stem from working with dictionaries, and you always know that your data is in the format you’d expect.

FastAPI allows you to define your API routes as functions, wrapping each one with a decorator (which is a very FP-like concept) to encapsulate your logic.

Below is an example of how this might be used. Again, it’s a simple example, but it’s fairly representative of what you might see in production.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()


class Baz(BaseModel):
    qux: int


class Foo(BaseModel):
    bar: str
    baz: Baz


@app.get('/foo')
async def get_foo(name: str, age: int) -> Foo:
    ...  # Some logic here
    return Foo(
        bar=name,
        baz=Baz(qux=age),
    )

# GET /foo?name=John&age=42
# {
#   "bar": "John",
#   "baz": {
#     "qux": 42
#   }
# }

As you can see, FastAPI uses Pydantic’s ability to convert nested objects to JSON to create a JSON response for our endpoint. The app.get decorator has also registered our get_foo function with the app object, allowing us to make GET requests to the /foo endpoint.

I hope you’ve found this article helpful. I’d love to hear what you think, and which paradigm you lean towards when writing Python.

Obviously, this isn’t the only way to combine FP and OOP in Python, and there are plenty of design patterns that can be implemented and improved upon using this sort of combination.

I’ll be writing about those in the future, and by following me on Medium, you won’t miss out. I also tweet about Python and my current projects on Twitter, and (more recently) post about them on Mastodon too.

I’m sure I’ll see you soon!

-Isaac

Python: To OOP or to FP? was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Permalink

Learning a new language - part 1 - functional

Now that I've done posting about why I'm retiring we can get back to our usual blogging.

I haven't really done anything technical over the past couple of year. The most programming I did in the past 18 months was a halfhearted effort at the Advent of Code 2021 event last December. I thought I'd see about picking up a new language to change that. In doing so, I was reminded of some of my resistances to learning a new language and how I've seen similar from self taught students coming into my classes.

Of course my situation is different - I'm doing this wholly by choice and once a student is in a class they've got to use the prescribed tool.

The contenders on the functional side were Haskell and OCaml and on the other side, mostly Rust.

Today, we'll talk about the functional side. I decided to look at OCaml. I had briefly looked at it well over a decade ago but barely scratched the surface. Haskell I played with a little more deeply but not much. I decided on OCaml because it was supposed to be more real world and, to be honest, I get turned off by some of the holier than though purely functional talk that sometimes occurs.

Funny thing is that I've had a really hard time getting going - so much so that I decided that for now I'd explore Rust.

So, what's going on?

OCaml looks to have some nice features. It's functional and I'm a fan there and it's also supposed to be pragmatic - you can get real world stuff done with it. It's also statically typed which overall I think is a good thing and it has cool features like pattern matching.

For pattern matching, I'm not talking about things like regex or matching characters in a string but rather something like a really cool and powerful case or switch statement. Even that description, though is not doing it justice.

I think my problem is that I already am comfortable and enjoy programming in Clojure. Clojure is functional and pragmatic but while Clojure doesn't have pattern matching built in but there is a library that basically accomplishes the same thing. As to the type system, Clojure is dynamically typed and while that would be an issue if I were working on large real projects and/or on a team, I'm just writing little toys by myself. If I want stronger typing, there are ways of doing that in Clojure. True, it's not required but working by myself, I have that option, should I choose. It's not the same - I know but as I said, I'm not doing anything big, important or with other people right now.

So, basically, every time I try to dive in to OCaml, I end up feeling that "I can already do that and do it better in Clojure." I'm guessing I'd feel the reverse had I started on the OCaml side.

This has happened to me before. I was an early Python adopter feeling that it was a huge improvement over Perl. A while later, Ruby came out and shortly after that, Ruby on Rails became a thing. Every time I looked into Ruby I had to stop - I could already do and do better in Python. By then, there were enough tools to compete with Rails even though they weren't as mainstream and overall I preferred Python. I never did end up learning Ruby.

I felt a similar sluggishness when I went from C++ to Java - something I had to do because of the AP Exam change but I pushed through since I had to teach it and ultimately, I've learned to appreciate the good and curse the bad of both languages.

I've seen similar blocks or resistance from who've had prior programming experiences over the years. From students who seemed to be taught more "the language" rather than CS using the language. I know that's a nebulous thing to say but I think some of you will catch my drift. It's subtleties like saying "okay, we're repeating this process, here's how we do it in language such and such" vs "here's how we repeat this process period" and show the language construct.

My former math chairman at Stuy said he once, back in the 80s asked Marvin Minsky what they should be teaching for CS in High School. Minsky said, and I paraphrase, "nothing, you'll only mess them up. At most teach them to type." Of course that was a different time and I strongly disagree with that statement but I do think we have to look at how students entering our classes are or are not prepared and adjust accordingly.

So, in the end, it's back to Clojure for me on the functional side and I'll try to do an OCaml dive sometime in the future.

Fortunately, I'm having an easier time getting in to Rust. More about that next time.

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.