How G+D Netcetera used Rama to 100x the performance of a product used by millions of people
“Rama enabled us to improve the performance of two critical metrics for Forward Publishing by over 100x, and it reduced our AWS hosting costs by 55%. It sounds like a cliché, but learning Rama mostly involved us unlearning a lot of the conventional wisdom accepted by the software industry. It was a strange experience realising how much simpler software can be.”
Ognen Ivanovski, Principal Architect at G+D Netcetera
Forward Publishing from G+D Netcetera powers a large percentage of digital newspapers in Switzerland and Germany. Forward Publishing was launched in 2019 and handles millions of pageviews per day. The backend of Forward Publishing was rewritten on top of Rama in the past year to greatly improve its performance and expand its capabilities.

Forward Publishing determines what content to include on each page of a newspaper website with a complex set of rules considering recency, categories, and relations between content. Home pages link to dozens of stories, while story pages contain the content and links to other stories. Some portions, especially for showing lists of related content, are specified with dynamic queries. Rendering any of these pages requires determining what content to display at each position and then fetching the content for each particular item (e.g. summary, article text, images, videos). Determining what content to render on a page is a compute-intensive process. The original implementation of Forward Publishing had some serious performance issues:
- Since computing a page from scratch on each pageview is too expensive, the architecture was reliant on a Varnish cache and only recomputing pages on an interval. This caused the system to take multiple minutes before new content would be displayed on a page. This was especially bad for breaking news.
- Computing what to render on a page put a large amount of load on the underlying content-management system (CMS), such as Livingdocs. This load frequently negatively affected the experience of using the CMS by worsening performance.
- The large amount of load sometimes caused outages due to the system becoming overwhelmed, especially if the cache had any sort of issue.
With Rama, they were able to improve the latency for new content becoming available on pages from a few minutes to less than a second, and they reduced the load on the CMS from Forward Publishing to almost nothing. Both of these are over 100x improvements compared to their previous implementation.
As a bonus, their Rama-based implementation requires much less infrastructure. They went from running 18 nodes per customer for Forward Publishing for various pieces of infrastructure to just 9 nodes per customer for their Rama implementation. In total their Rama-based implementation reduced their AWS hosting costs by 55%.
The new implementation reducing load on the CMS so much also has enabled G+D Netcetera to improve how the business is structured. Before, the CMS and Forward Publishing for each customer had to be operated as one integrated unit and required a lot of DevOps work to be able to handle the load. Now, the CMS can be operated independently of Forward Publishing and customers can customize it how they wish. It’s a better experience for G+D Netcetera’s customers, and it’s much less DevOps work for G+D Netcetera.
Before, Forward Publishing only worked with the Livingdocs CMS because supporting a new CMS would require a large engineering effort to get it to handle the necessary load. Now, Forward Publishing is not tied to a single CMS anymore. To support a new CMS, they just need to write a little bit of code to ingest new data from the CMS into Rama. This greatly expands the market for Forward Publishing as they can interoperate with any CMS.
Shortly before Rama was announced, the engineers at G+D Netcetera realized the backend architecture of Forward Publishing was wasteful. The number of updates to the CMS is only hundreds per day, so the vast majority of the work to compute pages was repeated each time the TTL of the cache expired. Pages usually only have a few changes when they’re recomputed. Flipping the architecture by maintaining a denormalized view of each page that’s incrementally updated as new content arrives seemed like it would be a big improvement.
They initially built a few prototypes with core.async and Missionary to explore this idea. However, these did not address how to store and query the denormalized views in a data-parallel and fault-tolerant manner. As soon as Rama was announced, they realized it provided everything needed to make their idea a reality.
Traditional databases, especially RDBMS’s, serve as both a source of truth and an indexed store to serve queries. There’s a major tension from serving both these purposes, as you want data in a source of truth to be fully normalized to ensure data integrity and consistency. What G+D Netcetera found out the hard way is that only storing data fully normalized caused significant performance issues for their application.
Rama explicitly separates the source of truth from the indexed datastores that serve queries. It provides a coherent and general model for incrementally materializing indexed datastores from the source of truth in a scalable, high-performance, and fault-tolerant way. You get the data integrity benefits of full normalization and the freedom to fully optimize indexed datastores for queries in the same system. That tension between data integrity and performance that traditionally exists just does not exist in Rama.
Backend dataflow
At the core of G+D Netecetera’s Rama-based implementation is a microbatch topology that maintains a denormalized view of each page as new content comes in. The denormalized views are PStates (“partitioned state”) that reduce the work to render a page to just a single lookup by key.
New content could be a new article, edits to an existing article, or edits to the layout of a page. The microbatch topology determines all denormalized entities affected by new content, which involves updating and traversing the graph of entities. This whole process takes less than a second and results in content always being fresh for all pages.
G+D Netcetera built a small internal library similar to Pregel on top of Rama’s dataflow abstractions. This allows them to easily express the code performing graph operations like the aforementioned traversals.
The core microbatch topology relies heavily on Rama’s batch blocks, a computation abstraction that has the same capabilities as relational languages (inner joins, outer joins, aggregation, subqueries). Batch blocks are the core abstraction that enables G+D Netcetera’s graph computations.
PState examples
Unlike databases, which have fixed data models (e.g. relational, key/value, graph, column-oriented), PStates can have any data model conceivable. PStates are based on the simpler primitive of data structures, and each data model is just a particular combination of data structures. This flexibility allows each PState to be tuned to exactly match the use cases they support. G+D Netcetera materializes many PStates of many different shapes in their Forward Publishing implementation.
Let’s take a look at some of their PState schemas to see how they use the flexibility of PStates. These examples are in Clojure because G+D Netcetera uses Rama’s Clojure API, but Rama also has a Java API.
All the PState definitions below make use of this data type to identify entities:
1 | (defrecord EntityId [tag id]) |
An entity ID consists of a “tag” (e.g. page, article, image) and an ID in the scope of that tag. You can store any types inside PStates, so this
defrecord
definition is used directly.
Two of their PStates are similar to how Datomic indexes data, called
$$eavt
and
$$avet
(PState names always begin with
$$
). The
$$eavt
definition is this:
1 2 3 4 5 6 7 | (declare-pstate topology $$eavt {EntityId (map-schema Keyword ; attribute #{Object})}) ; values |
This PState is a map from entity ID to a map from attribute to a set of values. The operations efficient on a PState are the same kinds of operations that are efficient on corresponding in-memory data structures. This PState efficiently supports queries like: “What are all the attributes for a particular entity?”, “What are all the values for an entity/attribute pair?”, “how many attributes does an entity have?”, and so on.
The
$$avet
PState is defined like this:
1 2 3 4 5 6 7 8 | (declare-pstate topology $$avet {Keyword ; attribute (map-schema Object ; value (set-schema EntityId {:subindex? true}) {:subindex? true})}) |
This PState is a map from attribute to a map from value to a set of entities. This allows different kinds of questions to be answered efficiently, like: “What are all the values associated with an attribute?”, “What are all the entities associated with a particular attribute/value pair?”, “How many entities have a particular attribute value?”, and many others.
This PState uses subindexing, which causes those nested data structures to index their elements individually on disk rather than serialize/deserialize the entire data structure as one value on every read and write. Subindexing enables reads and writes to nested data structures to be extremely efficient even if they’re huge, like containing billions of elements. As a rule of thumb, a nested data structure should be subindexed if it will ever have more than a few hundred elements. Since the number of values for an attribute and the number of entities for a particular attribute/value is unbounded, these nested data structures are subindexed.
While these PStates are similar to the data model of an existing database, G+D Netcetera has many PStates with data models that no database has. Here’s one example:
1 2 3 4 5 6 7 8 9 10 | (declare-pstate topology $$dynamic-lists {EntityId (fixed-keys-schema {:spec [[Object]] :max Long :items (set-schema Object {:subindex? true}) ; set of [timestamp, EntityId] :count Long })}) |
This PState supports an incremental Datalog engine they wrote for finding the top matching entities for a given criteria. Dynamic lists are specified by users of Forward Publishing and can be part of the layout of pages. The idea behind “lists” on a page is to show related content a reader is likely to click on. Manually curating lists is time-intensive, so dynamic lists allow users to instead specify a list as a query on the attributes of all stories in the system, such as story category, location, and author.
A dynamic list has its own ID represented as an
EntityId
, and the fields of each dynamic list are:
-
:spec
: the Datalog query which specifies which content to find for the list. A query could say something like “Find all stories that match either: a) from the region ‘Switzerland’ and tagged with “technology”, or b) tagged with ‘database'”. -
:max
: the maximum number of items to keep from entities that match -
:items
: the results of the Datalog query, tuples of timestamp andEntityId
-
:count
: the total number of entities found of which at most:max
were selected for:items
Dynamic lists can be dynamically added and removed, and they’re incrementally updated as new content arrives. The code implementing this incremental Datalog engine is only 700 lines of code in the Forward Publishing codebase. By storing the results in a PState, the dynamic lists are durable and replicated, making them highly available against faults like process death or node death.
Module overview
A Rama module contains all the storage and computation for a backend. Forward Publishing is implemented with two modules. The first module manages all content and contains:
The second module is smaller and manages user preferences. It consists of:
Topologies
Stream topologies are used for interactive features that require single-digit millisecond update latency. Microbatch topologies have update latency on the order of a few hundred milliseconds, but they have expanded computation capabilities (the “batch blocks” mentioned before) and higher throughput. So microbatch topologies are used for all features that don’t need single-digit millisecond update latency.
Two of the microbatch topologies implement the core of Forward Publishing, while the other ones support various miscellaneous needs of the product. The two core microbatch topologies are:
- “ingress”: Ingests new content from the CMS and appends it to a Rama depot called
*incoming-entities
- “main”: Consumes
*incoming-entities
to integrate new content into a bidirectional graph. This topology then implements the denormalization algorithm that traverses the graph.
The miscellaneous topologies include:
- “aliases”: A stream topology that maps entity IDs to other entity IDs. Some entities can be referenced by an “alias” entity ID, and this tracks the mapping from an alias to the primary entity ID.
- “bookmarks”: A stream topology that tracks pages bookmarked by users.
- “recently-read”: A stream topology that tracks articles recently read by users.
- “redirects”: A stream topology that maps URLs to alternate URLs to support browser redirects.
- “selected-geo”: A stream topology that maps users to geographical regions of interest.
- “indexing”: A microbatch topology tracking metadata about regions and lists.
- “routing”: A microbatch topology tracking metadata for each page of the website.
- “sitemap”: A microbatch topology that incrementally updates a sitemap for the website. Sitemaps allow search engines to index the full website. Before Rama, Forward Publishing could only update the sitemap once per month in an expensive batch process. Now, the sitemap is always up to date.
- “dynamic-lists”: A microbatch topology that implements the incremental Datalog engine described above.
Summary
Rama has been a game changer for G+D Netcetera, improving the performance of multiple key metrics by over 100x while simultaneously reducing the cost of operation by 55%. Additionally, their system is much more stable and fault tolerant than it was before.
G+D Netcetera found it takes about two weeks for a new engineer to learn Rama and become productive, with “batch blocks” being the biggest learning hurdle. The benefits they’ve achieved with Rama have made that modest investment in learning well worth it.
You can get in touch with us at consult@redplanetlabs.com to schedule a free consultation to talk about your application and/or pair program on it. Rama is free for production clusters for up to two nodes and can be downloaded at this page.