FRONTEND WEB DEV

In the ever-evolving landscape of Web Development, the Frontend arena in 2024 has seen a remarkable fusion of creativity, functionality, and technological innovation. Frontend web development, once primarily focused on crafting visually appealing user interfaces, has transcended into a domain where seamless user experiences and cutting-edge technologies converge to redefine digital interaction.

At the forefront of this evolution is the pervasive adoption of Progressive Web Apps (PWAs) and Single Page Applications (SPAs), revolutionizing how users interact with web content. PWAs, with their native app-like experiences, have gained significant traction, offering features such as offline capabilities, push notifications, and fast load times. SPAs, powered by frameworks like React, Angular, and Vue.js, enable dynamic content rendering without the need for page reloads, ensuring smoother and more responsive user interactions.

The frontend development landscape in 2024 is characterized by a polyglot environment, where JavaScript continues to reign supreme but is complemented by a diverse ecosystem of languages and tools. TypeScript, with its static typing and enhanced developer productivity, has emerged as a preferred choice for large-scale frontend projects, offering improved code quality and maintainability. Additionally, languages like Dart and ReasonML, along with compile-to-JavaScript languages such as ClojureScript and Elm, have found niche applications, catering to specific development requirements and preferences.

As the demand for richer multimedia experiences grows, WebAssembly (Wasm) has emerged as a game-changer in frontend development, enabling high-performance, near-native execution of code within the browser. Leveraging Wasm, developers can seamlessly integrate existing codebases written in languages like C, C++, and Rust into web applications, unlocking new possibilities for computationally intensive tasks such as gaming, multimedia processing, and data visualization.

In tandem with technological advancements, frontend development in 2024 places a heightened emphasis on accessibility, inclusivity, and user-centric design. With an increased awareness of digital accessibility standards and guidelines, developers strive to create web experiences that are perceivable, operable, understandable, and robust for users of all abilities. Furthermore, the growing recognition of the importance of inclusive design ensures that web applications are designed and developed with diverse user needs and preferences in mind, fostering a more equitable and user-friendly digital landscape.

The rise of low-code and no-code development platforms has democratized frontend development, empowering individuals with diverse skill sets to create web applications without extensive programming knowledge. These platforms, equipped with intuitive drag-and-drop interfaces and visual development tools, streamline the development process and lower the barrier to entry for aspiring creators, fueling innovation and creativity in the frontend space.

Looking ahead, the frontend web development landscape is poised for further evolution, driven by advancements in emerging technologies such as augmented reality (AR), virtual reality (VR), and the Internet of Things (IoT). As these technologies become increasingly integrated into web experiences, frontend developers will continue to play a pivotal role in shaping the digital future, delivering immersive, accessible, and impactful experiences that transcend traditional boundaries and redefine the possibilities of the web.

HOW TO START ?

Starting a career in frontend web development in 2024 requires a solid foundation in key technologies and a proactive approach to learning and skill development. Here's a roadmap to kickstart your journey:

  1. Learn HTML, CSS, and JavaScript: These are the foundational technologies of web development. HTML is used for structuring web pages, CSS for styling and layout, and JavaScript for adding interactivity and functionality to web applications. Begin by mastering the basics of each language, then delve deeper into more advanced concepts and techniques.

  2. Choose a JavaScript Framework or Library: In 2024, popular JavaScript frameworks and libraries include React, Angular, and Vue.js. Select one to focus on based on its popularity, job market demand, and your personal preferences. Each framework has its own ecosystem of tools and resources, so familiarize yourself with its documentation, best practices, and community support.

  3. Learn Version Control with Git: Git is an essential tool for collaborative development and version control. Learn how to use Git for tracking changes, managing branches, and collaborating with other developers. Platforms like GitHub and GitLab provide hosting services for Git repositories and offer additional collaboration features.

  4. Understand Responsive Web Design: With the proliferation of devices with varying screen sizes and resolutions, it's essential to learn responsive web design techniques. Master CSS frameworks like Bootstrap or Tailwind CSS, which provide pre-designed components and utilities for building responsive layouts efficiently.

  5. Explore CSS Preprocessors and Postprocessors: Tools like Sass, Less, and PostCSS enhance CSS functionality by introducing features like variables, mixins, and automatic vendor prefixing. Familiarize yourself with these preprocessors and postprocessors to streamline your CSS workflow and write more maintainable stylesheets.

  6. Learn Build Tools and Task Runners: Build tools like Webpack, Parcel, and Rollup automate tasks such as bundling, minification, and transpilation, improving the efficiency and performance of your frontend projects. Understand how to configure and customize these tools to optimize your development workflow.

  7. Study Frontend Testing: Gain proficiency in frontend testing methodologies and frameworks such as Jest, Jasmine, or Mocha. Learn how to write unit tests, integration tests, and end-to-end tests to ensure the reliability and quality of your code.

  8. Explore Progressive Web Apps (PWAs) and Single Page Applications (SPAs): PWAs and SPAs are becoming increasingly prevalent in web development. Understand the principles behind PWAs, including service workers, app manifest files, and offline caching. Learn how to build SPAs using your chosen JavaScript framework or library to create fast, dynamic, and responsive web applications.

  9. Stay Updated on Web Development Trends and Best Practices: The frontend web development landscape evolves rapidly, with new technologies, tools, and techniques emerging regularly. Stay informed about industry trends, attend webinars, read blogs, and participate in online communities to expand your knowledge and stay ahead of the curve.

  10. Build Projects and Create a Portfolio: Apply your skills by building real-world projects and showcasing them in a portfolio. Choose projects that demonstrate your proficiency in frontend development and highlight your creativity, problem-solving abilities, and attention to detail. Share your portfolio with potential employers to showcase your expertise and land your first frontend web development job.

Remember that learning frontend web development is an ongoing process, and success in this field requires continuous learning, experimentation, and adaptation to new technologies and trends. Stay curious, practice regularly, and embrace challenges as opportunities for growth and development.

Permalink

Clojure Deref (May 3, 2024)

Welcome to the Clojure Deref! This is a weekly link/news roundup for the Clojure ecosystem (feed: RSS). Thanks to Anton Fonarev for link aggregation.

Podcasts and videos

Blogs, articles, and projects

Libraries and Tools

New releases and tools this week:

Permalink

Ep 114: Brand New, Again

Each week, we discuss a different topic about Clojure and functional programming.

If you have a question or topic you'd like us to discuss, tweet @clojuredesign, send an email to feedback@clojuredesign.club, or join the #clojuredesign-podcast channel on the Clojurians Slack.

This week, the topic is: "what's old is new again". We find ourselves staring at code for the first time—even though we wrote some of it!

Our discussion includes:

  • Christoph re-opened his own code base after 14 months!
  • Nate joined a software team with a large, legacy code base.
  • Why does the problem of "fresh eyes" matter so much?
  • How is my past self going to treat me?
  • Are the past teammates going to help me?
  • Why you shouldn't rely on memory.
  • What's involved in really understanding a code base in order to make a change?
  • The frustration of being unproductive.
  • How is an app put together?
  • What's the flow of information?
  • How do you really know if your code is comprehensible?

Selected quotes

It's always fun to start something new. You don't have to worry about all those other things from the past!

I was a little nervous about that other guy who made all the code: me fourteen months ago! I wasn't sure how good of a teammate he was going to be.

The word "legacy" is like a big bad word, but I feel like it should be a badge of honor! It's software that is running right now and paying your salary! You should have a reverence for code that exists and is running.

At some point in time, you or a new team member, is diving into a code base with fresh eyes. It brings up so many important issues that we face as developers.

We spend so much time reading code and forming mental models about what is going on.

A fundamental challenge in software development is understanding, comprehending and reasoning about the code base.

Comprehending and reasoning about the code is one of the primary drivers behind the "why" of a lot of the so-called "best practices" of the industry. Why do you write tests? Why do you write documentation? Why do you try to have a good design in your application?

There's this constant learning that we have to do, and so try to make that easier.

He moved on to better projects in the sky. We've lost him to a better project in the Cloud. He moved to a better project upstate!

It's easy to say: "We have great documentation! Our code is super readable! Decoupled? Absolutely! Our pure data models? Totally comprehensible!" It's easy to say that, but you really find out if those things are true when somebody new joins the team or when you have to revisit the code after a long time.

Always trying to teach someone else about your code. There's always some future person.

What can we do now as we're setting up the situation for new people in the future?

Permalink

Humble Chronicles: The Inescapable Objects

In HumbleUI, there is a full-fledged OOP system that powers lower-level component instances. Sacrilegious, I know, in Clojure we are not supposed to talk about it. But...

Look. Components (we call them Nodes in Humble UI because they serve the same purpose as DOM nodes) have state. Plain and simple. No way around it. So we need something stateful to store them.

They also have behaviors. Again, pretty unavoidable. State and behavior work together.

Still not a case for OOP yet: could’ve been maps and functions. One can just

(def node []
  {:state   (volatile! state)
   :measure (fn [...] ...)
   :draw    (fn [...] ...)})

But there’s more to consider.

Code reuse

Many nodes share the same pattern: e.g. a wrapper is a node that “wraps” another node. padding is a wrapper:

[ui/padding {:padding 10}
 [ui/button "Click me"]]

So is center:

[ui/center
 [ui/button "Click me"]]

So is rect (it draws a rectangle behind its child):

[ui/rect {:paint ...}
 [ui/button "Click me"]]

The first two are different in how they position their child but identical in drawing and event handling. The third one has a different paint function, but the layout and event handling are the same.

I want to write AWrapperNode once and let the rest of the nodes reuse that.

Now — you might think — still not a case for OOP. Just extract a bunch of functions and then pick and choose!

;; shared library code
(defn wrapper-measure [...] ...)

(defn wrapper-draw [...] ...)

;; a node
(defn padding [...]
  {:measure (fn [...]
              <custom measure fn>)
   :draw    wrapper-draw}) ;; reused

This has an added benefit of free choice: you can mix and match implementations from different parents, e.g. measure from wrapper and draw from container.

Partial code replacement

Some functions call other functions! What a surprise.

One direction is easy. E.g. Rect node can first draw itself and then call a parent. We solve this by wrapping one function into another:

(defn rect [opts child]
  {:draw (fn [...]
           (canvas/draw-rect ...)
           ;; reuse by wrapping
           (wrapper-draw ...))})

But now I want to do it the other way: the parent defines wrapping behavior and the child only replaces one part of it.

E.g., for Wrapper nodes we always want to save and restore the canvas state around the drawing, but the drawing itself can be redefined by children:

(defn wrapper-draw [callback]
  (fn [...]
    (let [layer (canvas/save canvas)]
      (callback ...)
      (canvas/restore canvas layer))))

(defn rect [opts child]
  {:draw (wrapper-draw ;; reuse by inverse wrapping
           (fn [...]
             (canvas/draw-rect ...)
             ((:draw child) child ...)}))})

I am not sure about you, but to me, it starts to feel a little too high-ordery.

Another option would be to pass “this” around and make shared functions lookup implementations in it:

(defn wrapper-draw [this ...]
  (let [layer (canvas/save canvas)]
    ((:draw-impl this) ...) ;; lookup in a child
    (canvas/restore canvas layer))))

(defn rect [opts child]
  {:draw      wrapper-draw   ;; reused
   :draw-impl (fn [this ...] ;; except for this part
                (canvas/draw-rect ...)
                ((:draw child) child ...)}))

Starts to feel like OOP, doesn’t it?

Future-proofing

Final problem: I want Humble UI users to write their own nodes. This is not the default interface, mind you, but if somebody wants/needs to go low-level, why not? I want them to have all the tools that I have.

The problem is, what if in the future I add another method? E.g. when it all started, I only had:

  • -measure
  • -draw
  • -event

Eventually, I added -context, -iterate, and -*-impl versions of these. Nobody guarantees I won’t need another one in the future.

Now, with the map approach, the problem is that there will be none. A node is written as:

{:draw    ...
 :measure ...
 :event   ...}

will not suddenly have a context method when I add one.

That’s what OOP solves! If I control the root implementation and add more stuff to it, everybody will get it no matter when they write their nodes.

How does it look

We still have normal protocols:

(defprotocol IComponent
  (-context              [_ ctx])
  (-measure      ^IPoint [_ ctx ^IPoint cs])
  (-measure-impl ^IPoint [_ ctx ^IPoint cs])
  (-draw                 [_ ctx ^IRect rect canvas])
  (-draw-impl            [_ ctx ^IRect rect canvas])
  (-event                [_ ctx event])
  (-event-impl           [_ ctx event])
  (-iterate              [_ ctx cb])
  (-child-elements       [_ ctx new-el])
  (-reconcile            [_ ctx new-el])
  (-reconcile-impl       [_ ctx new-el])
  (-should-reconcile?    [_ ctx new-el])
  (-unmount              [_])
  (-unmount-impl         [_]))

Then we have base (abstract) classes:

(core/defparent ANode
  [^:mut element
   ^:mut mounted?
   ^:mut rect
   ^:mut key
   ^:mut dirty?]
  
  protocols/IComponent
  (-context [_ ctx]
    ctx)

  (-measure [this ctx cs]
    (binding [ui/*node* this
              ui/*ctx*  ctx]
      (ui/maybe-render this ctx)
      (protocols/-measure-impl this ctx cs)))

  ...)

Note that parents can also have fields! Admit it: We all came to Clojure to write better Java.

Then we have intermediate abstract classes that, on one hand, reuse parent behavior, but also redefine it where needed. E.g.

(core/defparent AWrapperNode [^:mut child] :extends ANode
  protocols/IComponent
  (-measure-impl [this ctx cs]
    (when-some [ctx' (protocols/-context this ctx)]
      (measure (:child this) ctx' cs)))

  (-draw-impl [this ctx rect canvas]
    (when-some [ctx' (protocols/-context this ctx)]
      (draw-child (:child this) ctx' rect canvas)))
  
  (-event-impl [this ctx event]
    (event-child (:child this) ctx event))
  
  ...)

Finally, leaves are almost normal deftypes but they pull basic implementations from their parents.

(core/deftype+ Padding [] :extends AWrapperNode
  protocols/IComponent
  (-measure-impl [_ ctx cs] ...)
  (-draw-impl [_ ctx rect canvas] ...))

Underneath, there’s almost no magic. Parent implementations are just copied into children, fields are concatenated to child’s fields, etc.

Again, this is not the interface that the end-user will use. End-user will write components like this:

(ui/defcomp button [opts child]
  [clickable opts
   [clip-rrect {:radii [4]}
    [rect {:paint button-bg)}
     [padding {:padding 10}
      [center
       [label child]]]]]])

But underneath all these rect/padding/center/label will eventually be instantiated into nodes. Heck, even your button will become FnNode. But you are not required to know this.

Also, a reminder: all these solutions, just like Humble UI itself, are a work in progress at the moment. No promises it’ll stay that way.

Conclusion

I’ve heard a rumor that OOP was originally invented for UIs specifically. Mutable objects with mostly shared but sometimes different behaviors were a perfect match for the object paradigm.

Well, now I know: even today, no matter how you start, eventually you will arrive at the same conclusion.

I hope you find this interesting. If you have a better idea — let me know.

Permalink

Humble Chronicles: Shape of the Component

Last time I ran a huge experiment trying to figure out how components should work in Humble UI. Since then, I’ve been trying to bring it to the main.

This was trickier than I anticipated — even with a working prototype, there are still lots of decisions to make, and each one takes time.

I discussed some ideas in Humble Chronicles: Managing State with VDOM, but this is what we ultimately arrived at.

The simplest component:

(ui/defcomp my-comp []
  [ui/label "Hello, world!"])

Note the use of square brackets [], it’s important. We are not creating nodes directly, we return a “description” of UI that will later be analyzed and instantiated for us by Humble UI.

Later if you want to use your component, you do the same:

(ui/defcomp other-comp []
  [my-comp])

You can pass arguments to it:

(ui/defcomp my-comp [text text2 text3]
  [ui/label (str text ", " text2 ", " text3)])

To use local state, return a function. In that case, the body itself will become the “setup” phase, and the returned function will become the “render” phase. Setup is called once, render is called many times:

(ui/defcomp my-comp [text]
  ;; setup
  (let [*cnt (signal/signal 0)]
    (fn [text]
      ;; render
      [ui/label (str text ": " @*cnt)])))

As you can see, we have our own signals implementation. They seem to fit very well with the rest of the VDOM paradigm.

Finally, the fullest form is a map with the :render key:

(ui/defcomp my-comp [text]
  (let [timer (timer/schedule #(println 123) 1000)]
    {:after-unmount
     (fn []
       (timer/cancel timer)) 
     :render
     (fn [text]
       [ui/label text])}))

Again, the body of the component itself becomes “setup”, and :render becomes “render”. As you can see, the map form is useful for specifying lifecycle callbacks.

Code reuse

React has a notion of “hooks”: small reusable bits of code that have access to all the same state and lifecycle machinery that components have.

For example, a timer always needs to be cancelled in unmount, but I don’t want to write after-unmount every time I want to use a timer. I want to use a timer and have its lifecycle to be registered automatically.

Our alternative is with macro:

(defn use-timer []
  (let [*state (signal/signal 0)
        timer  (timer/schedule #(println @*state) 1000)
        cancel (fn []
                 (timer/cancel timer))]
    {:value         *state
     :after-unmount cancel}))

(ui/defcomp ui []
  (ui/with [*timer (use-timer)]
    (fn []
      [ui/label "Timer: " @*timer])))

Under the hood, with just takes a return map of its body and adds stuff it needs to it. Simple, no magic, no special “hooks rules”.

Same as with hooks, with can be used inside with recursively. It just works.

Thanks Kevin Lynagh for the idea.

Shared state

One of the goals of Humble UI was to make component reuse trivial. Web, for example, has hundreds of properties to customize a button, and still, it’s often not enough.

I lack the resources to make hundreds of properties, so I wanted to take another route: make components out of simple reusable parts, and let end users recombine them.

So a button becomes clickable (behavior) and button-look (visual). Want a custom button? Implement your own look, and use the same behavior. Want to reuse the look in another component (e.g. a toggle button?). Write your own behavior, and reuse the visuals.

The look itself consists of simple parts that can be reused and recombined:

(ui/defcomp button-look [child]
  [clip-rrect {:radii [4]}
   [rect {:paint button-bg)}
    [padding {:padding 10}
     [center
      [label child]]]]])

And then the button becomes:

(ui/defcomp button [opts child]
  [ui/clickable opts
   [ui/button-look child]])

(this and a previous one are simplified for clarity)

Now, the problem. The button is, of course, interactive. It reacts to being hovered, pressed, etc. But the state that represents it lives in clickable (the behavior). How to share?

The first idea was to use signals. Like this:

(ui/defcomp button [opts child]
  (let [*state (signal/signal nil)]
    (fn [opts child]
      [ui/clickable {:*state *state}
       [ui/button-look @*state child]])))

Which does work, of course, but a little too verbose. It also forces you to define state outside, while logically clickable should be responsible for it.

So the current solution is this:

(ui/defcomp button [opts child]
  [ui/clickable opts
   (fn [state]
     [ui/button-look state child])])

Which is a bit tighter and doesn’t expose the state unnecessarily. The look component is also straightforward: it accepts the state as an argument, without any magic, so it can be reused anywhere.

Where to try

Current development happens in the “vdom” branch. Components migrate slowly, but steadily, to the new model.

Current screenshot for history:

Soon we will all live in a Virtual DOM world, I hope.

Permalink

PG2 release 0.1.12

PG2 version 0.1.12 is out. Aside from internal refactoring, there are several features I’d like to highlight.

First, the library is still getting faster. The latest benchmarks prove 15-30% performance gain when consuming SELECT results. Actual numbers depend on the nature of a query. Simple queries with 1-2 columns work faster than before:

Metrics:

Platform JDBC.Next PG2 0.1.1 PG2 0.1.2 PG2 0.1.12
core i5 2 GHz Quad-Core 16G 127.677926 43.026307 44.36297 21.941113
core i9 2,4 GHz 8-Core 32G 83.932103 39.551719 27.672117 20.957904
arm m1 10 cores 32G 49.340986 18.517718 14.670815 9.468902

Although queries with many columns are less sensitive to the new parsing algorithm, they’re still fast. Here is a chart for a complex query:

Metrics:

Platform JDBC.Next PG2 0.1.1 PG2 0.1.2 PG2 0.1.12
core i5 2 GHz Quad-Core 16G 579.59995 273.866142 80.352246 55.835803
core i9 2,4 GHz 8-Core 32G 447.326861 270.262248 54.34384 42.815123
arm m1 10 cores 32G 206.371502 117.241426 30.444798 29.92079

PG2 has become a bit faster with HTTP. The chart below measures a number of RPS of a Jetty server that fetches random data from a database and responds with JSON:

The tests were made using ab as follows:

ab -n 1000 -c 16 -l http://127.0.0.1:18080/

Timings:

Platform JDBC.Next PG2 0.1.1 PG2 0.1.2 PG2 0.1.12
core i5 2 GHz Quad-Core 16G 555.55 968.51 915.93 890.62
core i9 2,4 GHz 8-Core 32G 1304 1909.19 1688.72 1794.75
arm m1 10 cores 32G 1902.31 2999.06 3026 3363.36

The second feature is the :read-only? connection flag. When it’s set to true, the connection is run in READ ONLY mode, and every transaction opens being READ ONLY as well. This is useful for reading from replicas. A small example where an attempt to delete something leads to a negative response:

(pg/with-connection [conn {... :read-only? true}]
  (pg/query conn "delete from students"))

;; PGErrorResponse: cannot execute DROP TABLE in a read-only transaction

When set globally for connection, the flag overrides the same flag passed into the with-tx macro. Below, the transaction is READ ONLY anyway because the config flag is prioritized.

(pg/with-connection [conn {... :read-only? true}]
  (pg/with-tx [conn {:read-only? false}] ;; override won't do
    (pg/query conn "create table foo(id serial)")))

;; PGErrorResponse: cannot execute CREATE TABLE in a read-only transaction

Finally, there is a new reducer called “column” which fetches a single column as a vector. We often select IDs only, but the result is a vector of maps with a single :id field:

[{:id 100} {:id 101}, ...]

To get the ids, either you pass the result into (map :id ...) or, which is better, specify the :column key as follows:

(pg/with-connection [conn {...}]
  (pg/query conn
            "select id from users"
            {:column :id}))

;; [100, 101, 102, ...]

Internally, the reducer fetches the field from each row on the fly as they come from the network. It’s more effective than passing the result into map as it takes only one passage.

For more details, you’re welcome to the readme file of the repo.

Permalink

OSS Updates March and April 2024

This is a summary of the open source work I've spent my time on throughout March and April, 2024. Overall it was a really insightful couple of months for me, with lots of productive discussions and meetings happening among key contributors to Clojure's data science ecosystem and great progress toward some of our most ambitious goals.

Sponsors

This work is made possible by the generous ongoing support of my sponsors. I appreciate all of the support the community has given to my work and would like to give a special thanks to Clojurists Together and Nubank for providing me with lucrative enough grants that I can reduce my client work significantly and afford to spend more time on these projects.

If you find my work valuable, please share it with others and consider supporting it financially. There are details about how to do that on my GitHub sponsors page. On to the updates!

Grammar of graphics in Clojure

With help from Daniel Slutsky and others in the community, I started some concrete work on implementing a grammar of graphics in Clojure. I'm convinced this is the correct long-term solution for dataviz in Clojure, but it is a big project that will take time, including a lot of hammock time. It's still useful to play around with proofs of concept whilst thinking through problems, though, and in the interest of transparency I'm making all of those experiments public.

The discussions around this development are all also happening in public. There were two visual tools meetups focused on this over the last two months (link 1, link 2). And at the London Clojurians talk I just gave today I demonstrated an example of one proposed implementation of a grammar-of-graphics-like API on top of hanami implemented by Daniel.

There are more meetups planned for the coming months and work in this area for the foreseeable future will look like researching and understanding the fundamentals of the grammar of graphics in order to design a simple implementation in Clojure.

Clojure's ML and statistics tools

I spent a lot of time these last couple of months documenting and testing out Clojure's current ML tools, leading to many great conversations and one blog post that generated many more interesting discussions. The takeaway is that the tools themselves in this area are all quite mature and stable, but there are still ongoing discussions around how to best accommodate the different ways that people want to work with them. The overall goal in this area of my work is to stabilize the solutions so we can start advocating for specific ways of using them.

Below are some key takeaways from my research into all this stuff. Note none of these are my decisions to make alone, but represent my current opinions and what I will be advocating for within the community:

  • Smile will be slowly sunsetted from the ecosystem. The switch to GPL licensing was made in bad faith and many of the common models don't work on Apple chips. Given the abundance of suitable alternatives, the easiest option is to move away from depending on it.
  • A greater distinction between statistical modelling and machine learning workflows will be helpful. Right now there are many uses of the various models that are available in Clojure, and the wrappers and tools surrounding them are usually designed with a specific type of user in mind. For example machine learning people almost always have separate training and testing datasets, whereas statisticians "train" their models on an entire dataset. The highest-level APIs for these different usages (among others) look quite different, and we would benefit from having APIs that are ergonomic and familiar to our target users of various backgrounds.
  • We should agree on standards for accomplishing certain very common and basic tasks and propose a recommended usage for users. For example, there are almost a dozen ways to do linear regression in Clojure and it's not obvious which is "the best" way to someone not deeply familiar with the ecosystem.
  • Everything should work with tablecloth datasets and expect them as inputs. This is mostly the case already, but there is still some progress to be made.

Foundations of Clojure's data science stack

I continue to work on guides and tutorials for the parts of Clojure's data science stack that I feel are ready for prime time, mainly tablecloth and all of the amazing underlying libraries it leverages. Every once in a while this turns up surprises, for example this month I was surprised at how column header processing is handled for nippy files specifically. I also fixed one bug in tablecloth itself, which I discovered in the process of writing a tutorial earlier in March. I have a pile of in-progress guides focusing on some more in-depth topics from developing the London Clojurians talk that I'm going to tidy up and publish in the coming months.

The overarching goal in this area is to create a unified data science stack with libraries for processing, modelling, and visualization that all interoperate seamlessly and work with tablecloth datasets, like the tidyverse in R. Part of achieving that is making sure that tablecloth is rock solid, which just takes a lot of poking and prodding.

London Clojurians talk

This talk was a big inspiration for diving deep into Clojure's data science ecosystem. I experimented with a ton of different datasets for the workshop and discovered tons of potential areas for future development. Trying to put together a polished data workflow really exposed many of the key areas I think we should be focusing on and gave me a lot of inspiration for future work. I spent a ton of time exploring all of the possible ways to demonstrate a broad sample of data science tools and learned a lot along the way.

The resources from the talk are all available in this repo and the video will be posted soon.

Summary of future work

I mentioned a few areas of focus above, below is a summary of the ongoing work as I see it. A framework for organizing this work is starting to emerge, and I've been thinking about in terms of four key areas:

Visualisation

  • Priority here is to release a stable dataviz API using the tools and wrappers we currently have so that we can start releasing guides and tutorials that follow a consistent style.
  • The long-term goal is to develop a robust, flexible, and stable data visualization library in Clojure itself based on the grammar of graphics.

Machine learning

  • Priority is to decide which APIs we will commit to supporting in the long term and stabilize the "glue" libraries that provide the high-level APIs for data-first users.
  • Long term goal is to support the full spectrum of libraries and models that are in everyday use by data science professionals.

Statistics

  • Priority is to document the current options for accomplishing basic statistical modelling tasks, including Clojure libraries we do have, Java libs, and Python interop.
  • Long term goal is to have tablecloth-compatible stats libraries implemented in pure Clojure.

Foundations

  • Priority is to build a tidyverse for Clojure. This includes battle-testing tablecloth, fully documenting its capabilities, and fixing remaining, small, sharp edges.

Going forward

My overarching goal (personally) is still to write a canonical resource for working with Clojure's data science stack (the Clojure Data Cookbook), and I'm still working on finding the right balance of documenting "work-in-progress" tools and libraries vs. delaying progress until I feel they are more "ready". Until now I've let the absence of stable or ideal APIs in certain areas hinder development of this book, but I'm starting to feel very confident in my understanding of the current direction of the ecosystem, enough so that I would feel good about releasing something a little bit more formal than a tutorial or guide and recommending usages with the caveat that development is ongoing in some areas. And while it will take a while to get where we want to go, I feel like I can finally see the path to getting there. It just takes a lot of work and lot of collaboration, but with your support we'll make it happen! Thanks for reading.

Permalink

OSS Updates March and April 2024

This is a summary of the open source work I've spent my time on throughout March and April, 2024. Overall it was a really insightful couple of months for me, with lots of productive discussions and meetings happening among key contributors to Clojure's data science ecosystem and great progress toward some of our most ambitious goals.

Sponsors

This work is made possible by the generous ongoing support of my sponsors. I appreciate all of the support the community has given to my work and would like to give a special thanks to Clojurists Together and Nubank for providing me with lucrative enough grants that I can reduce my client work significantly and afford to spend more time on these projects.

If you find my work valuable, please share it with others and consider supporting it financially. There are details about how to do that on my GitHub sponsors page. On to the updates!

Grammar of graphics in Clojure

With help from Daniel Slutsky and others in the community, I started some concrete work on implementing a grammar of graphics in Clojure. I'm convinced this is the correct long-term solution for dataviz in Clojure, but it is a big project that will take time, including a lot of hammock time. It's still useful to play around with proofs of concept whilst thinking through problems, though, and in the interest of transparency I'm making all of those experiments public.

The discussions around this development are all also happening in public. There were two visual tools meetups focused on this over the last two months (link 1, link 2). And at the London Clojurians talk I just gave today I demonstrated an example of one proposed implementation of a grammar-of-graphics-like API on top of hanami implemented by Daniel.

There are more meetups planned for the coming months and work in this area for the foreseeable future will look like researching and understanding the fundamentals of the grammar of graphics in order to design a simple implementation in Clojure.

Clojure's ML and statistics tools

I spent a lot of time these last couple of months documenting and testing out Clojure's current ML tools, leading to many great conversations and one blog post that generated many more interesting discussions. The takeaway is that the tools themselves in this area are all quite mature and stable, but there are still ongoing discussions around how to best accommodate the different ways that people want to work with them. The overall goal in this area of my work is to stabilize the solutions so we can start advocating for specific ways of using them.

Below are some key takeaways from my research into all this stuff. Note none of these are my decisions to make alone, but represent my current opinions and what I will be advocating for within the community:

  • Smile will be slowly sunsetted from the ecosystem. The switch to GPL licensing was made in bad faith and many of the common models don't work on Apple chips. Given the abundance of suitable alternatives, the easiest option is to move away from depending on it.
  • A greater distinction between statistical modelling and machine learning workflows will be helpful. Right now there are many uses of the various models that are available in Clojure, and the wrappers and tools surrounding them are usually designed with a specific type of user in mind. For example machine learning people almost always have separate training and testing datasets, whereas statisticians "train" their models on an entire dataset. The highest-level APIs for these different usages (among others) look quite different, and we would benefit from having APIs that are ergonomic and familiar to our target users of various backgrounds.
  • We should agree on standards for accomplishing certain very common and basic tasks and propose a recommended usage for users. For example, there are almost a dozen ways to do linear regression in Clojure and it's not obvious which is "the best" way to someone not deeply familiar with the ecosystem.
  • Everything should work with tablecloth datasets and expect them as inputs. This is mostly the case already, but there is still some progress to be made.

Foundations of Clojure's data science stack

I continue to work on guides and tutorials for the parts of Clojure's data science stack that I feel are ready for prime time, mainly tablecloth and all of the amazing underlying libraries it leverages. Every once in a while this turns up surprises, for example this month I was surprised at how column header processing is handled for nippy files specifically. I also fixed one bug in tablecloth itself, which I discovered in the process of writing a tutorial earlier in March. I have a pile of in-progress guides focusing on some more in-depth topics from developing the London Clojurians talk that I'm going to tidy up and publish in the coming months.

The overarching goal in this area is to create a unified data science stack with libraries for processing, modelling, and visualization that all interoperate seamlessly and work with tablecloth datasets, like the tidyverse in R. Part of achieving that is making sure that tablecloth is rock solid, which just takes a lot of poking and prodding.

London Clojurians talk

This talk was a big inspiration for diving deep into Clojure's data science ecosystem. I experimented with a ton of different datasets for the workshop and discovered tons of potential areas for future development. Trying to put together a polished data workflow really exposed many of the key areas I think we should be focusing on and gave me a lot of inspiration for future work. I spent a ton of time exploring all of the possible ways to demonstrate a broad sample of data science tools and learned a lot along the way.

The resources from the talk are all available in this repo and the video will be posted soon.

Summary of future work

I mentioned a few areas of focus above, below is a summary of the ongoing work as I see it. A framework for organizing this work is starting to emerge, and I've been thinking about in terms of four key areas:

Visualisation

  • Priority here is to release a stable dataviz API using the tools and wrappers we currently have so that we can start releasing guides and tutorials that follow a consistent style.
  • The long-term goal is to develop a robust, flexible, and stable data visualization library in Clojure itself based on the grammar of graphics.

Machine learning

  • Priority is to decide which APIs we will commit to supporting in the long term and stabilize the "glue" libraries that provide the high-level APIs for data-first users.
  • Long term goal is to support the full spectrum of libraries and models that are in everyday use by data science professionals.

Statistics

  • Priority is to document the current options for accomplishing basic statistical modelling tasks, including Clojure libraries we do have, Java libs, and Python interop.
  • Long term goal is to have tablecloth-compatible stats libraries implemented in pure Clojure.

Foundations

  • Priority is to build a tidyverse for Clojure. This includes battle-testing tablecloth, fully documenting its capabilities, and fixing remaining, small, sharp edges.

Going forward

My overarching goal (personally) is still to write a canonical resource for working with Clojure's data science stack (the Clojure Data Cookbook), and I'm still working on finding the right balance of documenting "work-in-progress" tools and libraries vs. delaying progress until I feel they are more "ready". Until now I've let the absence of stable or ideal APIs in certain areas hinder development of this book, but I'm starting to feel very confident in my understanding of the current direction of the ecosystem, enough so that I would feel good about releasing something a little bit more formal than a tutorial or guide and recommending usages with the caveat that development is ongoing in some areas. And while it will take a while to get where we want to go, I feel like I can finally see the path to getting there. It just takes a lot of work and lot of collaboration, but with your support we'll make it happen! Thanks for reading.

Permalink

OSS updates March and April 2024

In this post I'll give updates about open source I worked on during March and April 2024.

To see previous OSS updates, go here.

Sponsors

I'd like to thank all the sponsors and contributors that make this work possible. Without you, the below projects would not be as mature or wouldn't exist or be maintained at all.

Current top tier sponsors:

Open the details section for more info about sponsoring.

Sponsor info

If you want to ensure that the projects I work on are sustainably maintained, you can sponsor this work in the following ways. Thank you!

If you're used to sponsoring through some other means which isn't listed above, please get in touch.

On to the projects that I've been working on!

Updates

Here are updates about the projects/libraries I've worked on last month.

  • squint: CLJS syntax to JS compiler
    • #509: Optimization: use arrow fn for implicit IIFE when possible
    • Optimization: emit const in let expressions, which esbuild optimizes better
    • Don't wrap arrow function in parens, see this issue
    • Fix #499: add support for emitting arrow functions with ^:=> metadata
    • Fix #505: Support :rename in :require
    • Fix #490: render css maps in html mode
    • Fix #502: allow method names in defclass to override squint built-ins
    • Fix #496: don't wrap strings in another set of quotes
    • Fix rendering of attribute expressions in HTML (should be wrapped in quotes)
    • Compile destructured function args to JS destructuring when annotated with ^:js. This benefits working with vitest and playwright.
    • #481: BREAKING, squint no longer automatically copies all non-compiled files to the :output-dir. This behavior is now explicit with :copy-resources, see docs.
    • Add new #html reader for producing HTML literals using hiccup. See docs and playground example.
    • #483: Fix operator precedence problem
  • neil: A CLI to add common aliases and features to deps.edn-based projects.
    Released version 0.3.65 with the following changes:
    • #209: add newlines between dependencies
    • #185: throw on non-existing library
    • Bump babashka.cli
    • Fetch latest stable slipset/deps-deploy, instead of hard-coding (@vedang)
    • Several emacs package improvements (@agzam)
  • clj-kondo: static analyzer and linter for Clojure code that sparks joy.
    Released 2024.03.13
    • Fix memory usage regression introduced in 2024.03.05
    • #2299: Add documentation for :java-static-field-call.
    • #1732: new linter: :shadowed-fn-param which warns on using the same parameter name twice, as in (fn [x x])
    • #2276: New Clojure 1.12 array notation (String*) may occur outside of metadata
    • #2278: bigint in CLJS is a known symbol in extend-type
    • #2288: fix static method analysis and suppressing :java-static-field-call locally
    • #2293: fix false positive static field call for (Thread/interrupted)
    • #2296: publish multi-arch Docker images (including linux aarch64)
    • #2295: lint case test symbols in list
      Unreleased changed:
    • #1035: Support SARIF output with --config {:output {:format :sarif}}
    • #2309: report unused for expression
    • #2135: fix regression with unused JavaScript namespace
    • #2302: New linter: :equals-expected-position to enforce expected value to be in first (or last) position. See docs
    • #2304: Report unused value in defn body
  • CLI: Turn Clojure functions into CLIs!
    Released version 0.8.58-59
    • Fix #96: prevent false defaults from being removed/ignored
    • Fix #91: keyword options and hyphen options should not mix
    • Fix #89: long option never represents alias
  • rewrite-edn: Utility lib on top of rewrite-clj with common operations to update EDN while preserving whitespace and comments
    Released 0.4.8 with the following update:
    • Add newline after adding new element to top level map with assoc-in
  • nbb: Scripting in Clojure on Node.js using SCI
    • nbb bundle JS output will ignore nbb.edn
    • #351: Update bun docs/example.
    • Add cljs.core/exists?
  • clojure-mode: Clojure/Script mode for CodeMirror 6.
  • instaparse-bb: Use instaparse from babashka
    • Serialize regexes in parse results
  • scittle: Execute Clojure(Script) directly from browser script tags via SCI
    Released v0.6.17
    • #77: make dependency on browser (js/document) optional so scittle can run in webworkers, Node.js, etc.
    • #69: executing script tag with src + whitespace doesn't work
    • #72: add clojure 1.11 functions like update-vals
    • #75: Support reader conditionals in source code
  • cherry: Experimental ClojureScript to ES6 module compiler
    • #127: fix duplicate cherry-cljs property in package.json which caused issues with some bundlers
    • Bump squint common compiler code
  • clerk
    • #646 Fix parsing + location issue which fixes compatibility with honey.sql
  • http-client: babashka's http-client
    Released 0.4.17-19
    • #55: allow :body be java.net.http.HttpRequest$BodyPublisher
    • Support a Clojure function as :client option, mostly useful for testing
    • #49: add ::oauth-token interceptor
    • #52: document :throw option
  • bbin: Install any Babashka script or project with one command
    These fixes have been made by @rads:
  • SCI: Configurable Clojure/Script interpreter suitable for scripting and Clojure DSLs
    • Fix #626: add cljs.core/exists?
    • Fix #919: :js-libs + refer + rename clashes with core var
    • Fix #906: merge-opts loses :features or previous context
  • deps.clj: A faithful port of the clojure CLI bash script to Clojure
    • Fix Windows issue related to relative paths (which took me all day, argh!)
  • fs - File system utility library for Clojure
    • #122: fs/copy-tree: fix copying read-only directories with children (@sohalt)
    • #127: Inconsistent documentation for the :posix-file-permissions options (@teodorlu)
  • babashka: native, fast starting Clojure interpreter for scripting.
    • Fix #1679: bump timbre and fix wrapping timbre/log!
    • Add java.util.concurrent.CountDownLatch
    • Add java.lang.ThreadLocal
    • Bump versions of included libraries

Other projects

These are (some of the) other projects I'm involved with but little to no activity happened in the past month.

Click for more details

Permalink

Rama is a testament to the power of Clojure

It took more than ten years of full-time work for Rama to go from an idea to a production system. I shudder to think of how long it would have taken without Clojure.

Rama is a programming platform that integrates and generalizes backend development. Whereas previously backends were built with a hodgepodge of databases, application servers, queues, processing systems, deployment tools, monitoring systems, and more, Rama can build end-to-end backends at any scale on its own in a tiny fraction of the code. At its core is a new programming language implementing a new programming paradigm, at the same level as the “object-oriented”, “imperative”, “logic”, and “functional” paradigms. Rama’s Clojure API gives access to this new language directly, and Rama’s Java API is a thin wrapper around a subset of this language.

There’s a lot in Clojure’s design that’s been instrumental to developing Rama. Three things stand out in particular: its flexibility for defining abstractions, its emphasis on immutability, and its orientation around programming with plain data structures. Besides these being essential to maintaining simplicity in Rama’s implementation, Rama also embraces these principles in its approach to distributed programming and indexing.

Ability to do in libraries what requires language support in other languages

Rama’s language is Turing-complete and defined largely via Clojure macros. So it’s still Clojure, but its semantics are different in many fundamental ways. At its core, Rama generalizes the concept of a function into something called a “fragment”. Whereas a function works by taking in any number of input parameters and then returning a single value as the last thing it does, a fragment can output many times (called “emitting”), can output to multiple “output streams”, and can do more work between or after emitting. A function is just a special case of a fragment. Rama fragments compile to efficient bytecode, and fragments that happen to be functions execute just as efficiently as functions in Java or Clojure.

Even though Rama contains this new programming language implementing this new programming paradigm, it’s still Clojure. So it interoperates perfectly. Rama code can invoke Clojure code directly, and Clojure code can invoke Rama directly as well. There’s no friction between them. Rama itself is implemented in a mixture of regular Clojure code and Rama code.

Neither Rich Hickey nor John McCarthy ever envisioned this completely different programming paradigm being built within their abstractions, much less one that reformulates the basis of nearly every programming language (the function). They didn’t need to. Clojure, along with its Lisp predecessors, are languages that put almost no limitations on your ability to form abstractions. With every other language you at least have to conform to their syntax and basic semantics, and you have limited ability to control what happens at compile-time versus runtime. Lisps have great control over what happens at compile-time, which lets you do incredible things.

Lisp programmers have struggled ever since it was invented to explain why this is so powerful and why this has a major impact on simplifying software development. So I won’t try to explain how powerful this is in general and will focus on how instrumental it was for Rama. I’ll instead point you to Paul Graham’s essay “Beating the Averages”, which was the essay that first inspired me to learn Lisp back when I was in college. When I first read that essay I didn’t understand it completely, but I was particularly compelled by the lines “A big chunk of our code was doing things that are very hard to do in other languages. The resulting software did things our competitors’ software couldn’t do. Maybe there was some kind of connection.”

The new language at Rama’s core is an example of this. Other languages can only have multiple fundamentally different paradigms smoothly interoperating if designed and implemented at the language level. Otherwise, you have to resort to string manipulation (as is done with SQL), which is not smooth and creates a mess of complexity. No amount of abstraction can hide this complexity completely, and attempting to often creates new complexities (like ORMs).

With Clojure, you can do this at the library level. We required no special language support and built Rama on top of Clojure’s basic primitives.

Rama’s language is not our only example of mixing paradigms like this. Another example is Specter. Specter is a generically useful library for querying and manipulating data structures. It’s also a critical part of Rama’s API (since views in Rama, called PStates, are durable data structures of any composition), and it’s a critical part of Rama’s implementation. About 1% of the lines of code in Rama’s source and tests are Specter callsites.

You can define Specter’s abstractions in any language. What makes it special in Clojure is how performant it is. Queries and manipulations with Specter are faster than even hand-rolled Clojure code. The key to Specter’s performance is its inline caching and compilation system. Inline caching is a technique I’ve only seen used before at the language or VM level. It’s a critical part of how the JVM implements polymorphism, for example. Because of the flexibility of Clojure, and the ability to program what happens at compile-time for a Specter callsite, we’re able to utilize the technique at the library level. It’s all done completely behind the scenes, and users of Specter get an expressive and concise API that’s extremely fast.

Power of immutability and data structure orientation

Clojure is unique among Lisps in the degree that it emphasizes immutability. It’s core API is oriented to working with immutable data structures. Additionally, Clojure encourages representing program state with plain data structures and having an expressive API for working with those data structures. The quote “It’s better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures.” is part of Clojure’s rationale.

These philosophies have had a major impact on Rama’s development, helping a tremendous amount in managing complexity within Rama’s implementation. The less state you have to think about in a section of code, the easier it is to reason about. When a project gets as big as Rama (190k lines of source, 220k lines of tests), with many layers of abstractions and innumerable subsystems, it’s impossible to keep even a fraction of the whole system “in your head” for reasoning. I frequently have to re-read sections of code to remind myself on the details of that particular subsystem. The dividends you get from lowering complexity of the system, with immutability being a huge part of that, compounds more and more the bigger the codebase gets.

Clojure doesn’t force immutability for every situation, which is also important. Rama tracks a lot of different kinds of state, and we find it much simpler in some cases to use mutability rather than work with state indirectly as you would through something like the State Monad in Haskell. There are also some algorithms that are much simpler to write when they use a volatile internal in the implementation. That said, the vast majority of code in Rama is written in an immutable style. When we use mutability it’s almost always isolated within a single thread. Rather than have concurrent mutability using something like an atom, we use a volatile and send events to its owning thread to interact with it.

Rama embraces and expands upon Clojure’s principles of immutability and orienting code around data structures. These principles are fundamental to Rama’s approach for expressing end-to-end backends. A lot of Rama programming revolves around materializing views (PStates), which are literally just data structures interacted with using the exact same Specter API as used to interact with in-memory data structures. This stands in stark contrast with databases, which have fixed data models and special APIs for interacting with them. Any database can be replicated in a PState in both expressivity and performance, since a data model is just a specific combination of data structures (e.g. key/value is a map, column-oriented is a map of sorted maps, document is a map of maps, etc.).

Rama’s language extends Clojure’s immutable principles into writing distributed, fault-tolerant, and async code. There’s a lot of similarities with Clojure like anonymous operations with lexical closures, immutable local variables, and identical semantics when it comes to shadowing. Rama takes things a step further for distributed computation, doing things like scope analysis to determine what vars needs to be transferred across network boundaries. Rama’s loops have similar syntax to Clojure and have the additional capability of being able to be a distributed computation that hops around the cluster during loop iterations. With Rama this is all written linearly through the power of dataflow, with switching threads/nodes being an operation like anything else (called a “partitioner”).

Clojure’s principles are just sound ideas that really do make a huge impact on simplifying software development. These principles are even more relevant in distributed systems / databases which historically have been overrun with complexity. That’s why these principles are so core to Rama and its implementation.

Conclusion

There’s a seeming contradiction here – if Clojure enables such productivity gains, then why is it still a niche language in the industry? Why aren’t those using Clojure crushing their competition so thoroughly that every programmer is now rushing to adopt Clojure to even the playing field?

I believe this is simply because Clojure does not address all aspects of software development. This is not a criticism of Clojure but a recognition of its scope. Things like durable data storage, deployment, monitoring, evolving an application’s state and logic over time, fault-tolerance, and scaling are huge costs of building end-to-end software. Oftentimes the principles of Clojure are corrupted when using a database, as the database forces you to orient your code around its data model and capabilities.

This is why we’re so excited about Rama and have worked so long on it, because Rama does address everything involved in building end-to-end backends, no matter the scale. Rama provides flexible data storage expressed in terms of data structures, has deployment and monitoring built-in, has first-class features for evolving an application and updating it, is completely fault-tolerant, and is inherently scalable. It does all this while maintaining Clojure’s great principles and functional programming roots.

If you’d like to discuss on the Clojure Slack, we’re active in the #rama channel.

Permalink

Clojurists Together project - Scicloj community building - April 2024 update

The Clojurists Together organisation has decided to sponsor Scicloj community building for Q1 2024, as a project by Daniel Slutsky. The project is taking place in February, March, and April 2024. Here is Daniel’s update for April. Here are the previous ones: Feb 2024, Mar 2024 Comments and ideas would help. 🙏 Clojurists Together update - April 2024 - Daniel Slutsky # April 2024 was the last of three months on the Clojurists Together project titled “Scicloj Community Building and Infrastructure”.

Permalink

Clojure 1.12.0-alpha11

Clojure 1.12.0-alpha11 is now available! Find download and usage information on the Downloads page.

  • CLJ-2848 - Qualified instance methods without param-tags should use the qualified method class, not the target object type

  • CLJ-2847 - Improve error message when a qualified method in value position matches no methods

Permalink

Copyright © 2009, Planet Clojure. No rights reserved.
Planet Clojure is maintained by Baishamapayan Ghose.
Clojure and the Clojure logo are Copyright © 2008-2009, Rich Hickey.
Theme by Brajeshwar.