As always, let’s get straight to the point.

what is a time series?

If you’ve spent any time as a software engineer you have likely looked at a graph that plots important information about whats going on over time on computers that run your software.

Something that is so intuitive to analyze as a graph isn’t quite as intuitive to define formally. As a database developer I think of a timeseries graph in three components:

1. Samples, which are the raw measurements. These are typically polled from a machine that reports them constantly by reading some metric value (stored in something like an atomic f64). Since they are taken at a specific time, they’ll be associated with the timestamp that they were measured at.

2. Series are vectors of samples that are associated with a particular context, which most systems refer to as labels (e.g. {instance='a2', region='us-west-2'}).

3. Queries distill series into useful information. Typically there is too much information to directly examine series. If you have thousands of nodes, you might want to look at the CPU utilization on average to know if you have well utilized cores.

This post covers how we store series and samples in order to efficiently serve query.

samples and indexes

To make sense of how the data is laid out on disk, it pays to understand the anatomy the queries that use that data:

The selector determines which series should be fetched and included in the query while the aggregation defines the logic for combining together samples across different series.

Think about it like a 2D matrix where each series is a row and each timestamp is a column. The cells hold the measurements at each timestamp.

Efficiently computing the query requires figuring out how to prune series to only the series matched by the selector, extracting the samples for the time range needed, grouping them by the aggregation and finally computing the results.

This is done in three steps:

1. An inverted index is used to narrow down which series need to be fetched

2. A forward index determines which series need to be aggregated together

3. The raw samples are retrieved and aggregated

the inverted index

The query starts here. The inverted index narrows down potentially hundreds of thousands of series into just the ones required to serve the query.

For example the query http_requests_count{instance='a2'} needs to fetch all the series for http_requests_count that also have the label instance=’a2’ and can ignore the rest.

A naive approach is to store series sorted by name. That would allow us to binary search the entire database to isolate only series with the name http_request_count and then scan those to find which belong to instance='a2'.

The problem is that there may be thousands of instances, each collecting hundreds of metrics, so narrowing down the set to just series that match instance=’a2’ isn’t trivial.

The solution to this problem is borrowed from search systems: the inverted index.

For every label that exists, the database stores a list of the series that have that label. This list is called a “posting list”.

Finding the requires series is reduced to an intersection of posting lists.

To make this intersection efficient, each series is assigned a unique ID and the posting lists are sorted by this ID. That makes the algorithm simple: you load the posting lists for the labels you care about and advance pointers into the lists until they all align on a series ID, that’s one you need to fetch (in the diagram above, series 1 and 5). You continue until you’ve exhausted one of the posting lists and you can ignore posting lists that aren’t part of your query.

roaring bitmaps

There are some encoding tricks you can use to store and leverage the inverted index efficiently. The state of the art mechanism for this particular use case is called a roaring bitmap.

The insight with this data structure is that posting lists tend to be either sparse or dense, and the sparse ones tend to be clustered. For timeseries, you can imagine that the posting list for container_id=123 (a sparse list) holds significantly fewer series than region:us-west-2 (a dense list). If you assign series ids monotonically, though, it’s likely that all the series generated by container_id=123 have ids within a small range of the total series id domain (they all get registered on startup).

To leverage this observation, roaring bitmaps split the representation of posting lists into “dense blocks” and “sparse blocks”. Since they only encode 32-bit integer values, they index these blocks by the “high 16” bits and then the blocks encode which exact values exist. Dense blocks encode bitsets to represent which values are present while sparse blocks just encode the array of values naively.

In the example below, we have representations for both sparse and dense blocks. The sparse block encodes the values [5, 1000] while the dense block encodes every number between 196608 and 250000.

If we were to encode the sparse block with a full bitset it would take 8KB since we need one bit for every value that starts with 0x0000. But since there are only two numbers, it’s more efficient to just encode 4 bytes (the lower 16 bits for each 5 and 1000).

On the other hand if we were to encode the dense block as a simple array, it would require 212KB (2 bytes per entry). It’s more efficient to store a single 8KB bitset.

Roaring bitmaps are not only efficient in storage space. Because the lists are sorted, and because bitmap intersections/unions can be done with SIMD instructions, set operations on roaring bitmaps are incredibly efficient.

the forward index

If you followed the inverted index section closely, you’ll notice that we made a small decision with big implications: the inverted index only stores series ids not the series definitions themselves.

This means that while the inverted index helped us find which series IDs we need to serve a query, it doesn’t help us at all for determining which series should be grouped together for an aggregation.This means we need some mapping of series IDs to their definitions (the label set).

This is called the forward index:

Since queries need to load the forward index for every series it matches, the retrieval of this data can get expensive.

Most data that is stored on disk, including indexes like the forward index, is stored in something similar to an SST (see this previous post on that for more detail), which means you choose a key that you sort by. This is called the “clustering” key, and how you choose it makes a big difference to your storage performance.

To make this concrete, here are two alternative layout options for storing series and forward indexes:

In the first example, we just choose to sort by series_id. This makes it easy to go from the inverted index to then retrieve the samples and the forward index because each posting in the inverted index is just the series ID.

The problem is that series IDs have no encoding or relationship to the metric name. If a query targets only a specific metric (which is the case for almost every query), it may have to fetch series that are scattered on disk, which can dramatically increase the read amplification of a query.

The layout on the right sorts the forward index and samples based on a composite key of (metric_name, series_id) instead. This means that consecutive series IDs are not necessarily clustered, but blocks on disk are dedicated to a single metric. If you need to fetch many series for the same metric, this can be significantly more efficient.

storing samples

Now that we have a way to know which series we need and which ones group together in an aggregation, we just need a mechanism for storing and fetching the raw data.

An interesting observation is that timeseries data is both abundant and repetitive, a combination that results in some interesting characteristics that make it particularly amenable to compression (something we discussed in depth in a previous post).

A data point in a series, called a sample, has two parts: the timestamp the measurement was taken and the value of the measurement.

timestamps

If you’ve ever configured prometheus, you know that you configure your scraper to scrape at some regular interval (perhaps every 15s). That means it shouldn’t surprise you when the timestamps in any particular series look a lot like this:

┌series A timestamps────────────────────────────────────┐
│┌────────┐┌────────┐┌────────┐┌────────┐     ┌────────┐│
││00:00:00││00:00:15││00:00:31││00:00:45│ ... │01:00:00││
│└────────┘└────────┘└────────┘└────────┘     └────────┘│
└───────────────────────────────────────────────────────┘

You may occasionally see gaps that are wider (e.g. if a pod is overloaded), but in stead state you expect the timestamps to always be ~15s apart.

The typical encoding of a timestamp is an i64 long, which takes up 8 bytes. If you were to encode 100k series every 15s that would take you 192 MB/hour just to store timestamps. In production, however, most systems are able to store this with just ~3 MB/hour (a 60x reduction). How?

Some smart people figured out that if you store only the first timestamp, and then encode just the difference between them, you can store these points using much less data:

┌─series A timestamps (deltas)────────────────────────────┐
│ ┌────────┐┌────────┐┌────────┐┌────────┐     ┌────────┐ │
│ │00:00:00││00:00:15││00:00:31││00:00:45│ ... │01:00:00│ │
│ └────────┘└────────┘└────────┘└────────┘     └────────┘ │
│     ┌────────┐ ┌────────┐ ┌────────┐                    │
│     │  15s   │ │  16s   │ │  14s   │ ...                │
│     └────────┘ └────────┘ └────────┘                    │
└─────────────────────────────────────────────────────────┘

Using 2 bytes per delta, you can encode up to gaps of 18 hours (65k seconds) and reduce your overall data by 4x.

Because the timestamps are reported at 15s intervals, It turns out we can do even better. The difference of the difference between timestamps is even smaller, in fact it’s almost always pretty close to 0.

┌series A timestamps (delta of deltas)───────────────────┐
│┌────────┐┌────────┐┌────────┐┌────────┐     ┌────────┐ │
││00:00:00││00:00:15││00:00:31││00:00:45│ ... │01:00:00│ │
│└────────┘└────────┘└────────┘└────────┘     └────────┘ │
│    ┌────────┐ ┌────────┐ ┌────────┐                    │
│    │  15s   │ │  16s   │ │  14s   │ ...                │
│    └────────┘ └────────┘ └────────┘                    │
│          ┌────────┐ ┌────────┐                         │
│          │   1s   │ │  -2s   │                         │
│          └────────┘ └────────┘                         │
└────────────────────────────────────────────────────────┘

There are some fancy ways that you can encode deltas of deltas so that you can store timestamps in approximately 1 bit per sample, a 64x reduction from the original 8 byte format.

measurements

If you got the gist of the last section, you may have already deduced that measurements might follow a similar pattern. Metrics typically change slowly over time (unless something special is happening, which by definition doesn’t happen often).

This sound pretty similar to what we saw when we were discussing timestamps, but the problem is that a measurement is a floating point value. A small difference between two floating point numbers can still require many bits.

Take for example the IEEE 754 (the standard) representation for 0.0001, such a simple decimal surprisingly looks like this:

0011111100011010001101101110001011101011000111000100001100101101

  ┌────────────────────┬────────────────┬───────────────────┐
  │     Component      │      Bits      │       Value       │
  ├────────────────────┼────────────────┼───────────────────┤
  │ Sign (1 bit)       │ 0              │ +                 │
  ├────────────────────┼────────────────┼───────────────────┤
  │ Exponent (11 bits) │ 01111110001    │ 1009 − 1023 = −14 │
  ├────────────────────┼────────────────┼───────────────────┤
  │ Mantissa (52 bits) │ 1010...0101101 │ fractional part   │
  └────────────────────┴────────────────┴───────────────────┘

The sign bit is simple (positive or negative). The mantissa is multiplied by 2 ^ exponent to recreate the fractional value, in this case we get:

(−1)^0 × 1.1010...0101101₂ × 2^−14 ≈ 0.00010000000000000000479

That’s not exactly 0.0001 but it’s as close as we can represent with floating point numbers. The interesting thing is that you might notice that 0.0001 and 0.0002 look… very similar (in fact, the mantissa is the same):

  ┌────────┬──────┬─────────────┬────────────────────────────┐
  │ Value  │ Sign │  Exponent   │  Mantissa                  │
  ├────────┼──────┼─────────────┼────────────────────────────┤
  │ 0.0001 │ 0    │ 01111110001 │ 101000110110111...00101101 │
  ├────────┼──────┼─────────────┼────────────────────────────┤
  │ 0.0002 │ 0    │ 01111110010 │ 101000110110111...00101101 │
  └────────┴──────┴─────────────┴────────────────────────────┘

If you XOR these two values you get a value with a lot of 0s:

0000000000110000000000000000000000000000000000000000000000000000

This isn’t coincidence, many similar floating point numbers have lots of 0s when you XOR them. This means you can store a three part number: first store the number of leading 0s (10) then store the meaningful length (2) then store the meaningful bits 0b11 which is only 2 bits. This takes you down from the original 64 bits to just 15 bits (in this case).

Together these two concepts (delta of delta encoding for timestamps and XOR encoding for measurement values) allow us to store many measurements extremely efficiently, storing on average only ~11 bits per measurement, down from 16 bytes of the naive format.

Going back to our original 100k series reported at 15s interval, you go from 384 MB/h to 33 MB/h, a nearly 12x improvement.

series churn

Let’s revisit the 2D matrix model of timeseries where the horizontal dimension is time and the vertical dimension are the series. If you look at a few hours of production data you would probably see something like this:

The ██ represents series with data during that time and a . represents series with no data at that time.

Series S06-S12 are likely tagged by something like container_id that is not static, so whenever you deploy a new series is generated with nearly identical labels except for a change in container_id.

This is called “series churn”, and it’s the biggest problem in timeseries databases. If we were to ignore this pattern in observability systems, we’d end up with some pretty nasty side effects.

For illustration, imagine there were only 2 labels on each of these metrics: name (the metric name) and container_id. If we indexed this in a single inverted index you’d see something like this:

name=metric_1: [1, 3, 5, 7, 9, 11]
name=metric_2: [2, 4, 6, 8, 10, 12]
container=a:   [1, 2, 3]
container=b:   [4, 5]
container=c:   [6, 7]
container=d:   [8, 9, 10]
container=e:   [11, 12]

If my query targeted the time range 16:00-24:00, and I’m querying for metric_2 then I’d need to examine all of [1, 3, 5, 7, 9, 11] because that’s the only filtering condition I have. I would then discover that series [7, 9] don’t have any data associated with the window I’m querying, but only after I’ve fetched the samples for those series.

The solution to this is bucketing: storing a full trio of samples, inverted and forward indexes for each “bucket” of time. The series IDs are also scoped to a bucket, which allows you to quickly find the series ID for incoming data without looking through the entire history of series.

Notice that the downside of this is that this generates many more distinct series IDs, even though some of them refer to the same “logical” series. In the example above, A1, B1, and C1 all have the same labels.

This is a fundamental tradeoff between read, write and space amplification in timeseries storage. You can manage this by merging series across time blocks together for older buckets, and this is in fact exactly how many timeseries databases handle the problem.

fsync()

That’s it for today! If you want to read up more details on how we encode and store these data types on disk, we have a detailed RFC on GitHub.

We’ve covered the three main types of data that a timeseries system stores: samples, inverted indices and forward indices as well as how they are represented on disk. Coming soon, I’ll write a full post on the query execution pipelines for timeseries engines so make sure to subscribe to get it delivered straight to you.

The short version is that database vendors have incentives that push them away from simplicity and toward defensibility, resulting in databases that seemingly get worse over time.

Disclaimer: I’m working on an OSS database project called OpenData that was partly motivated by the dynamics in this post, but this is the only time I’ll mention it.

making sense of earnings reports

There is some puzzling data available in historical earnings reports of public database companies that were acquired or shutdown. Cloudera, MariaDB, and Confluent (where I was a relatively early employee) all reported gross margins between 78% and 91%. In the traditional economic sense, gross margins this high are pretty rare and you’d expect these companies to be wildly profitable. It is clear that none of the referenced companies were.

For a database company, gross margins are the ratio between the amount of revenue the company brings in and the cost to “run the database”. The accounting here is pretty loose, but there are in essence two factors that play into the cost: the hardware cost and the operations cost.

The hardware cost is more obviously measurable, this is what the company pays the cloud providers for the hardware. The operations cost is estimated based on the number of engineering hours spent on-call and on “other operational tasks”.

When I was at Confluent, this was the result of a survey that accounting sent out to the engineering teams, asking us how much time we spent on operations. As you can imagine, that’s an imprecise estimate, but investors care a lot about it.

What explains this half of the story is operating margin, which indicates how much of the revenue went toward R&D, Sales & Marketing and Administration. After accounting for that, these companies generated very modest margin (from 2-8%, and deep in the red if you include all GAAP expenses).

These gross and operating margins are interesting on their own, but together they paint a concerning economic picture: database companies depend on a delicate balance, and even a small sustained dip in gross margins can be fatal.

monopoly and competition

If you aren’t too interested in economic theory or already have a solid foundation, you can skip this section.

In economic theory there’s a spectrum between monopoly markets and perfect competition. At one end, a monopolist has meaningful control over price, and therefore demand. At the other, firms sell effectively identical products into a market with many competitors, so any one firm has almost no pricing power.

Economists represent this by relating marginal revenues MR (revenue from selling one more unit) and marginal costs MC (cost to serve one more unit) with respect to demand. Firms, whether monopolies or not, want to increase output so long as selling one more unit brings in more revenue than cost. This equilibrium is denoted with MR = MC.

We can explore the intuition using a database example. I love to pick on MongoDB because they’re generous enough to have detailed public pricing. An M50 instance is priced at $1,460/mo. They chose this price intentionally.

Imagine the alternatives to help understand the intuition behind the MR=MC condition:

1. At $2,000/month, they would make more on each sale but likely lose enough customers that total profit falls.

2. At $1,000/month, they would win more customers but probably not enough to make up for the lower revenue per cluster.

3. Somewhere in between is the “best” point that preserves high margins but remains low enough to avoid losing too much demand.

That math looks very different depending on the shape of the demand curve.

The graphs above show the extreme ends of the spectrum.

In a perfect monopoly, the firm’s own pricing decision meaningfully affects the quantity it can sell. If MongoDB was a true monopoly, customers may dislike the M50 price tag, but they would not have a close substitute to jump to immediately (their only alternative is not having MongoDB).

In perfect competition, the opposite is true. If there were a thousand vendors selling truly identical managed M50 clusters, MongoDB could not raise prices much without losing customers to cheaper alternatives, and it could not sustainably cut prices below cost forever either.

The reality is that MongoDB lives somewhere in the middle. The question for us is what characteristics does the database market as a whole have in the long term?

The answer depends on how “different” databases really are. Postgres and Kafka are relatively different systems and you’re usually ill advised to use them interchangeably. Confluent’s Kafka and Amazon’s MSK, on the other hand, are close substitutes. In economic models, even a single close substitute can quickly erode margins and force vendors to compete on price (the Bertrand Paradox).

That puts significant pressure on vendors to avoid becoming interchangeable, else risk losing their high gross margins.

why vendors need high margins

There’s an interesting parallel to draw between two seemingly unrelated markets: databases and pharmaceuticals. For both, the initial R&D investment to bring a product to market is substantial. Developing a serious database or a successful drug takes years of specialized work, but selling one more unit of either isn’t particularly expensive.

Economists describe this as high fixed costs (the cost to develop the initial product) and low variable costs (the cost to manufacture a single instance of the product). This creates a basic requirement where the initial investment has to be earned back somehow. Whether that happens through high margins, high volume, or both depends on the demand curve we discussed earlier.

There are a few ways that this can play out, and where the comparison with pharma becomes useful. In both industries, the company whose R&D investment results in something genuinely innovative can charge what’s known as Schumpeterian Rent: temporary profits that exist because competitors have not yet caught up. During this period the original company is effectively a monopoly. You can charge as much as you want for Insulin or Kafka if you’re the only one selling it (for now).

The core tension here is that innovation is not the same thing as sustained value capture. Innovation creates temporary pricing power, which unlocks the temporary high gross margin we see in earning reports, but maintaining that requires constant innovation and sustained R&D costs.

In situations like this, who captures value from an innovation is shaped by two factors: appropriability and complementary assets. Appropriability defines how easily an invention can be imitated and the Complementary Assets define what is required to monetize the innovation itself.

The reason the pharmaceutical industry is a useful foil is that it tends to have a stronger answer to both questions. Government-issued patents create a much more formal appropriability regime, and the path from inventing a drug to monetizing it is usually more directly controlled by the innovator (they directly own or have strong relationships with manufacturing).

Databases do not get the luxury of government protection, so vendors have to rely on weaker defenses like closed source code, restrictive licenses, and operational know-how. None are reliable barriers in the long term, and the half-life of these protections is quickly degrading with advancements in AI. That means the Schumpeterian rents from database innovation usually decay more gradually and less predictably than they do in pharma.

The bigger problem is in the complementary assets. In pharma, the innovator usually has a much more direct claim on the assets required to commercialize the product. In databases, the crucial complement is the infrastructure and hardware that runs the system, which is typically owned by cloud providers like AWS rather than by the inventor of the database.

This asymmetry is the core of the industry’s strange economics.

Bringing it back to the demand curves and what options are available to database vendors, it starts to make sense why they must sell databases at high margins to survive. They need to recover high fixed costs in a market where the innovation diffuses rapidly, the protection is weak, and the owners of the key complements (the hyperscalers) are structurally better positioned to win a price war.

This means their only strategy is to continuously generate innovations that renew the Schumpeterian advantage, differentiate in some other manner, or fundamentally change their business models. But continuously generating innovation requires additional investment, which has a significant fixed cost and starts the cycle all over again.

why databases become worse over time

This conclusion, on its own, is not especially troubling. It is good that vendors have to keep innovating if they want to earn outsized profits. The problem is that if these innovations run dry, the company ends up resorting to tactics that hurt the product to avoid a price war with AWS.

The unpalatable truth is that the useful feature set of a database stabilizes relatively quickly. Database developers tend to release a wealth of new features to stay ahead of competition, but on average these features don’t get adopted (and at worst cause issues).

When feature development is no longer enough, the focus shifts from pure product improvement to avoiding commoditization.

There are three characteristics of a commodity: fungibility, price transparency, and low switching costs. With a mature database, fungibility tends to rise first. If most buyers view multiple managed offerings of the same database as “close enough,” then the remaining levers of differentiation shift elsewhere.

Some of those levers can be healthy, and even benefit developers. Sometimes a vendor creates high switching costs by providing such tight, high-quality integration across a bundle of services that detangling it is simply not feasible without significant investment. All database companies want this kind of defensibility, but few achieve it. Databricks is one of the clearer success cases.

Some levers are toxic. Vendors can, and do, manufacture price opacity by hiding behind complicated pricing schemes. Snowflake pricing, for example, is so complex that Gartner released a report (available for purchase) that helps you understand and negotiate your bill.

But there is another, more subtle lever: operational complexity.

To see why, we can revisit the MongoDB Atlas example. If you recall, an M50 instance is priced at $1,460/mo but the equivalent cluster of three 1-year reserved r5.xlarge instances in AWS us-east-1 cost only $360/mo (including 160GB EBS volumes). This means that MongoDB thinks it’s reasonable for you to value the operational work they do for you at $1,100/mo, or 3x the hardware cost.

This also means a 50% reduction in hardware cost will reduce your price by ~12% while a 50% reduction in operational costs reduces your price by ~38%, assuming savings are passed on directly to you.

That creates an uncomfortable incentive structure. MongoDB has strong reasons to make operations easier inside Atlas, but weaker reasons to make the database easy for everyone else to operate. Operational expertise is, after all, a large part of what the managed service is selling. If the system becomes dramatically easier to run everywhere, you would be less willing to pay such high margins for their management layer.

In other words, they are incentivized to do the opposite. The tougher it is to operate a database and the more specialized the expertise required, the harder it is for competitors (including the self-hosting alternative) to offer an equivalent experience.

This usually does not happen overtly. I’ve been a database engineer my whole career, and I can guarantee you that we are not sitting around trying to make systems harder to run for no reason. The dynamic is subtler. We are rewarded for adding functionality that helps the product stand out, and each additional layer of functionality tends to make the system more operationally complex. Over time, the database becomes more capable, but also more “interesting” to debug at 3AM when the pager goes off.

To summarize, the market incentivizes defensibility over simplification. This, in turn, results in databases getting “worse” over time.

the end

You may not have sympathy for multi-billion dollar database companies, and that’s fine, but the inconvenient truth is that most database development (and OSS software at large) is funded by enterprise, which means that your user experience is directly dictated by corporate profits. We’ve explored why this may not be in your favor.

You likely also need no proof that the hyperscalers are not a solution to this. Their incentives are not particularly aligned with users either, and they are often happy to capture the value created by upstream database companies and open source communities without taking on the same level of product risk. They are excellent at scaling proven systems, but much less naturally motivated to make the kind of focused, opinionated bets that produce new categories in the first place.

Since I’ve already inundated you with economic theory and this post is quite long, I’ll end it with a glimmer of optimism: I think object storage is already making a different economic model possible. It is generic enough to absorb a large share of the heavy infrastructure burden, while still being simple enough to serve as a foundation for building narrower, more focused systems on top. That changes the fixed-cost equation in a way that could matter a lot, opening up the door to lower-margin business models. You can already see systems like Warpstream, Turbopuffer, and Quickwit taking advantage of it.

But that’s a subject for another post!

Next up, I’ll get back to some deep engineering content and let the economics settle for a bit. As always, thanks for your support and reading bits & pages.

I started thinking about compression when implementing prefix compression for SlateDB. When I ran benchmarks, I noticed that performance seemed "worse" despite improved compression ratios.

This got me thinking deeper about why databases use compression, and what the right framework is for reasoning about whether that tradeoff is worth it. This post is the result of that exploration, and I hope it helps you understand why, when and how data systems use techniques to reduce data size.

understanding “compression math”

The first rule to understand about why compression is important is that there are effectively three resources that databases can be bottlenecked on1:

1. I/O Bandwidth: bytes/sec from disk, network, or even DRAM into your CPU caches

2. CPU: cycles available on your machine for processing

3. Memory: transient storage that is orders of magnitude faster to access than disk/network

Compression gives you a knob to tune between these resources. More specifically, compression directly trades I/O for CPU because it takes CPU cycles to compress and decompress data but moving compressed data requires less I/O bandwidth.

Mathematically, we can reason about this tradeoff by computing the cost of compression and applying it to different resource utilization. Let’s assume for a given workload and compression we have the following properties2:

is compression about latency?

Naively, we can compute a hypothetical breakeven I/O bandwidth β that determines whether compression will speed up a point-to-point transfer.

An uncompressed data transfer takes S/B_x seconds. Compressing then transferring the same dataset takes T_c + S_c/B_x + T_d seconds3, since you first need to compress the data before sending it and decompress it after sending it. In other words, compression improves the overall transfer latency when:

Using this we can compute our breakeven bandwidth:

To make this concrete I timed the (de)compression of a sample data taken from a compression challenge with a baseline of 191MiB using both zstd level 4 (fast, lighter compression) and level 10 (slower, more aggressive compression):

Using the breakeven formula from above, the breakeven points for compressing the data are 328 MiB/s (2.76 Gbps) and 76 MiB/s (0.64 Gbps) for levels 4 and 10 respectively.

An r5d.xlarge instance has baseline B_network of 1.25 Gbps, meaning if latency is all you cared about zstd with level 4 compression is worthwhile but level 10 is not: the improved compression is overshadowed by the time it takes to achieve it.

The very same instance, however, has a speedy NVMe disk with B_NVMe of 10s of Gbps. This would make it seem like if you aren’t transferring data over the network, you shouldn’t compress with either of these options! And if your only bottleneck were the latency of a one-shot transfer from disk into the CPU, that conclusion would be right… but the breakeven analysis assumes the I/O pipe is otherwise idle, which is rarely true in a database.

saturation and the CPU ↔ I/O exchange rate

In practice, databases are not doing one-shot transfers of a blob from disk to RAM and calling it a day. Instead, they need to constantly read significant amounts of data to handle incoming requests, compress old files, fill caches, etc. Put another way, databases will saturate their I/O bandwidth much before they saturate CPU. Sustained throughput matters far more than latency.

Instead of thinking about a breakeven latency point, we can think about compression as a way to exchange bandwidth for CPU. I’ll refer to the effective throughput after accounting for compression as logical bandwidth:

Getting back to our example from before, zstd level 4 gives us ~9.4x compression ratio. This means that our logical bandwidth on a 1.25 Gbps network line is 11.75 Gbps. Put another way, we can sustain a throughput of 11.75 Gbps if we compress/decompress as compared to 1.25 Gbps if we send raw data over.

Of course, this logical bandwidth only materializes if you can afford the CPU cost. A useful way to think about codecs is in terms of (de)compression throughput θ:

To fully saturate a pipe of bandwidth B_x you need:

In other words you need to be able to compress data faster than you can send it over the network.

This math starts to influence decisions when you look at the increasing amount of time it takes for rather trivial improvements at higher levels of compression (for the same 191MiB dataset from before):

There’s an obvious diminishing return of the higher compression levels, and this is why databases don’t simply choose to use the most aggressive compaction. But we can go one step further and put a dollar value to the exchange rate between CPU and logical bandwidth.

putting a dollar value on compression

The final principle to consider when evaluating whether (and how aggressively) to compress is in the context of managed services that markup their cost per byte, meaning they put an artificial premium on bytes that make compression even more valuable. The typical example here are egress and ingress fees with cloud providers.

This ends up looking quite similar to the exchange rate between CPU and I/O, except instead you are trading CPU for dollars. Let’s consider the r5d class of instances on AWS; you’ll have the following rates:

• $0.072 / vCPU hour (in us-west-2)

• $0.02 / GB transferred across zones (ingress + egress)

Using these numbers, sustaining 1 Gbps of physical bandwidth (450 GB/h) costs $9/hour in transfer fees alone. To sustain 1 Gbps of logical (uncompressed) throughput, you only need to transfer 1/R Gbps on the wire, so your bandwidth cost drops to $9/R per hour. But to get this you need enough CPU to compress at that rate. Since θ measures how many Gbps a single vCPU can compress, you need 1/θ vCPUs to keep up. At $0.072 per vCPU-hour, your total cost to sustain 1 logical Gbps is:

In other words, you pay less GB transfer costs the higher your compression ratio is but to achieve a higher compression ratio and sustain that throughput you need to pay more CPU. Plugging in the table of observed values from the previous section:

For these particular compression ratios, choosing zstd level 8 is your best bet (if what you care about is reducing cost).

sometimes it is about latency

Bringing it back to where we started, it’s often worth being conservative about compression levels. While the cost equation above suggests that something like zstd level 8 minimizes dollars per unit throughput, databases don’t live entirely in the world of steady-state bandwidth.

In many systems, decompression happens directly on the query critical path. In those cases, every additional millisecond of extra CPU time can show up in tail latencies, and the most cost-efficient compression level may not be the latency-optimal one.

One useful property (see the data in the latency section) is that decompression time is nearly constant across levels. The CPU cost asymmetry is heavily skewed toward the write path. This means you can invest in aggressive compression at write time where latency is often more tolerable without penalizing read-side tail latencies. The practical implication is that for write-once, read-many workloads, the compression level decision is primarily a write-path throughput question.

As with all things databases, these are tradeoffs you need to make with your workload and preferences in mind.

consider getting hands on

Before I get into the techniques used, I often feel the best way to learn about performance optimizations (including data size) is to play around with them yourself. To that end, I created a challenge to compress a sample dataset of one million rows as much as possible. The naive dataset serialized as JSON is 210MB, zstd at level 22 gets down to 11MB, and after a few hours of bit twiddling I got it down to 9.4MB.

I’m maintaining a leaderboard, so you can try your hand at the challenge yourself: https://github.com/agavra/compression-golf

In addition, there are some interesting submissions that detail their approaches to compressing the dataset (typically as comments at the top of the file or in the PR they submit). You may want to start by reading this blog, and then applying that understanding to understand the leading submissions.

techniques for data reduction

There are two main techniques for reducing data size:

1. Semantic Encoding: these techniques understand the data patterns and change the binary representation to be more compact. Examples include prefix, varint and run-lenght encoding (discussed more later).

2. Entropy Compressors: these have no knowledge of the data semantics, but they can analyze the bytes to find redundancy and then use various techniques to reduce the entropy. Examples include zstd, snappy and gzip.

These two techniques are often applied together. First you will use semantic encoding to reduce your logical data size, and then apply entropy compression to further compress the data.

Entropy compression alone gets you pretty far. On the compression golf dataset, zstd -9 took 210MB down to 18MB (level 22 reached 11MB). Adding standard semantic encodings got us to 7MB with the most aggressive techniques getting down to 5MB.

The more important property is the CPU cost. Since semantic encodings are mostly O(n) passes with simple ops they are cheap to run. If you apply them first your entropy compressor has less (and lower entropy) data to chew on, requiring fewer CPU cycles.

Connecting this back to the cost model shows how this two-layer compression pays dividends. On the compression-golf dataset, my semantic codec reduced the data from 191 MiB to ~11 MiB. Applying zstd on top brought it down to 7.2 MiB in a total of 1.4s, which is comparable CPU time to zstd level 8 alone (1.3s) but with 2.5x better compression.

Plugging this into our cost formula computes θ ≈ 1.14 Gbps (similar to level 8) and R ≈ 26.5 (vs 10.67 for level 8). Since the bandwidth term $9/R dominates the cost equation, that ratio improvement is dominating bringing the total cost to $0.40/hr per logical Gbps, less than half of the previous best (see cost table in the section on dollar value of compression).

Entropy compression techniques are well explored4 and can be applied to your data with little to no modification from you. There are many libraries (zstd, gzip, snappy) that implement entropy-based compression you can pick up off the shelf and apply to your workload. Since it is unlikely you will need to implement any of these techniques yourself, the rest of this post will focus on how you can apply semantic encoding to your data.

small encodings of common datatypes

There are standard, in-memory byte representations for many data types that I suspect you are familiar with.

An example of this is u32 (an unsigned5, big-endian, 32-bit integer). For example the number 2,031,620 is represented like this:

Each of the four bytes (in this case represented as hexadecimal codes) represent orders of magnitude more significant values than the next (by a factor of 256, the number of values that each hex code can represent).

If we were to store these as bytes (on disk, in memory, etc…) each one takes up 32 bits. If I store 1024 of these u32 integers I’m storing 32KB of data. Can we do better?

Turns out when you have a u32 field, most values are typically pretty small and the first and second byte would both be 0x00. To avoid storing unnecessary 0x00 bytes on disk, there are strategies such as variable encoding that will use the most significant bit to indicate whether or not more bytes are needed past the current byte (and then the remaining 7 bits to store data).

Here is an example of encoding the same integer as above, but using only 3 bytes. Note that we need to encode the least-significant bytes first (little endian encoding) so that we can stop as soon as we no longer need more bytes.

This gets more efficient the smaller the numbers are. Numbers less than 128 only take 1 byte!

My favorite characteristic of varint encoding is that there is no size limit, you can encode a u16, u32 and u64 all with the same encoding. This makes it backwards compatible to change the in-memory representation (upcast from a four byte to eight byte integer) without changing the underlying storage and is a technique used by many formats such as Parquet and Protobuf.

There are various other strategies that help achieve similar results for other data types, but varint is the most common of them.

delta, run-length, prefix and XOR encoding

The next class of techniques exploit the fact that in-memory types are more general than necessary. This makes sense for mutable data since you need headroom for values you haven’t seen yet. Once the data set is fixed (on disk, for example) you can examine it and choose tighter encodings.

Each of these four techniques help take advantage of the observation that data is typically sorted or clustered in a way where similar values are next to one another.

I’ll summarize these techniques below and help you with understanding when each should be chosen but for the full algorithms there are various more detailed explanations online:

• Delta: use when integers are semantically close to one another. Instead of encoding the raw values, encoding only the first raw value and then the differences (deltas) between them. Often the deltas will be much smaller than the raw data, allowing varint encoding to work better than it would on the raw numbers.

• Run-Length Encoding (RLE): use when there are many repeated values or sparse data. When many values repeat, encode the value once and then the number of times it repeats. This is commonly used in columnar data formats.

• Prefix Encoding: use with sorted strings. Since sorted strings often share prefixes you only need to encode the length of the shared prefix and the suffix. Combined with Delta encoding or RLE you can reduce the size of the encoded prefix lengths.

• XOR Encoding: use mostly for similar floating point values. This is similar to Delta encoding, but sometimes a small difference between floating points requires a ‘large’ representation. It works by XOR’ing the bits, so values that are similar to one another will typically end up with many 0s. There are then techniques you can use (see Gorilla for example) for compressing values that are sparse.

dictionary encoding

Dictionary encoding helps when you have repeated data (or data patterns) that aren’t necessarily close in proximity, meaning you can’t use one of the techniques previously discussed. In other words, the word “engineer” might show up frequently in a database of users as the job title but not next to users that are near one another.

To solve this, dictionary encoding is a technique that maps commonly used tokens to numeric identifiers, and then replaces the tokens with the identifier themselves. There are many ways to represent the dictionary concisely (FST, Front Coded Lists, Radix/MARISA Tries, etc…) depending on the data distribution and requirements, but the main concept remains the same.

The interesting thing about dictionaries is that they’re often used by entropy-based compressors (such as zstd) because they can see repeated sequences of byte strings. This means that using a semantic dictionary alone is not very useful if all you care about is compressed data size, you’re just repeating work that your compressor is going to do anyway.

The benefits of the dictionary are second order.

First, semantic dictionary encoding is faster because it can add a string to a dictionary as soon as it encounters it. In the meantime an algorithm like zstd needs to consider all possible sequences of bits and doesn’t know which ones repeat without a complicated lookback mechanism.

Second, it unlocks other encodings (like delta encoding and bitpacking), which compounds to improve the overall compression ratio.

bit-packing

Bitpacking is a technique used to ignore byte-alignment and pack multiple values into a single byte. A classic example of bit-packing is to pack eight boolean values into a single byte, using just one bit to represent each boolean.

The technique extends to more complicated data types as well. For example, if you know that deltas between all of your values are between 0 and 32 you only need 5 bits to represent each delta. The smallest varint is still 1 byte (8 bits), so you can do even better if you pack eight 5-bit values into 5 bytes:

You should use bitpacking with extreme caution.

Not only is it easy to get your encodings wrong, but it can cause your final compressed dataset to be larger than if you hadn’t bitpacked. This is because most entropy-based compressors (zstd included) are byte-aligned, meaning they look for repetition along byte boundaries. If you have many repeated 7-bit strings the data looks “scrambled” to the eyes of a byte-aligned algorithm, so it won’t be able to do its job properly.

To show this I ran a quick experiment (see code and full results) that generated 10k values between 0 and 127 (values that fit in 7 bits), sorted them and then compared the final compressed size if I were to just use full 8-bit values or if I bitpacked them into 7-bit values.

Despite the bitpacked representation being smaller before compression (8,750 bytes vs 10,000 bytes), zstd compresses the raw bytes down to just 246 bytes, which is nearly 3x better than the 662 bytes it achieves on the bitpacked version.

lossy compression

The techniques described earlier in this blogpost are all “lossless”. In other words, you can reconstruct the exact dataset you compressed initially. Sometimes, though, you can accept a lossy compression. Most of these techniques are domain-specific, and you are probably familiar with this concept in image and video processing.

While these techniques are interesting in their own right, I’ll save covering them for a future post (stay tuned for a post on vector databases, which leverage lossy compression quite elegantly).

fsync()

And we’re done for today! If you take away just one thing from this post it should be that compression is (almost) always worth doing, but how much compression depends on certain parameters of your workload.

reasoning about structure

As with all indexing structures, what you need to keep in mind is the read, write and space amplification characteristics of a particular structure. If you haven’t read our database foundations post, I’ll give a quick summary of what that means here:

1. Read amplification1 describes the overhead for reading an entry in a database. If I need to fetch 32KB from disk to read a single 1KB row, my read amplification is 32x.

2. Write amplification quantifies the overhead when a database writes more bytes than strictly necessary to store a piece of data. If writing a single 1KB row to disk required rewriting a 32KB block, my write amplification is 32x.

3. Space amplification captures the ratio between actual storage consumed and the logical data size, accounting for fragmentation, tombstones, and redundant copies. If I need 32GB disk space to comfortably run a workload with 16GB data, my space amplification is 2x.

Since there’s no perfect data structure on all three (the RUM conjecture), the rest of the post will go over different ways to structure SSTs that have different tradeoffs on each of these dimensions.

naive strategies

No matter what indexing structure you choose, the initial write path for all databases that use SSTs is the same: you buffer data in memory until you have enough to flush out an SST and you write that SST to disk. As we discussed in the last post, there’s numerous reasons for why that write path is an advantageous design.

What happens after that is called compaction, and that defines the process of giving these immutable files some structure and periodically combining the immutable SSTs together.

The wonderful thing about separating the write path and compaction is that you have an extraordinarily flexible mechanism for trading off between read, write and space amplification. You can design a system that accepts writes as fast as your SSD can handle, or you can have a system that attempts to always find the data you need with a single index lookup and pay for it with write amplification. In practice, you typically want something that lands somewhere in the middle.

This section outlines the extremes and use those as anchor points when we look into the typical strategies used in production systems.

Examples throughout this post assume that SSTs have the following properties (for more details on what these mean, see the previous post on SSTs):

a simple log

The easiest way to work around the immutability limitation of SSTs is to just write new ones.

A log index structure continuously prepends new SSTs to the head of the log. When finding a key within the log, a scan is performed sequentially on each SST until the key is found.

write amplification

This structure is surprisingly versatile. The main benefit is that there is no write amplification: rows that are written to a log of SSTs are written exactly once and never rewritten. You pay for this structure with both read and space amplification.

read amplification

The read amplification for a log is the number of bytes you need to read before you find your key. Since the log is structured by time, you know that the first SST in the log that contains your key is the most recent version of it. Assuming you have an empty cache and the SST containing key is the n-th SST, you can estimate the number of bytes you will need to read in the worst case:

  N * sizeof(bloom_filter)
+ (N - 1) * bloom_filter_fp_rate * sizeof(index + block_size)
+ sizeof(index + block_size)

This formula explained in words: you need to read bloom filters for every SST, you will need to read the index + block_size to verify that the key is not in an SST for every false positive, and then read the index + block_size for the SST that contains the final key.

This means that the cost of reading data grows linearly with the number of SSTs.

If you get unlucky and the key you have is old, you can see how this quickly adds up (assuming the “real” data you want to read is a single 1KB row):

Read Amplification with 0.1% False Positive Rate
(Bloom: 16KB, Index+Block: 36KB)

Read Amplification by SST Count (1KB rows, 0.1% FP bloom filters)

  SSTs |   Total Read |     Amp
-------|--------------|--------
    10 |       196 KB |    196x
   100 |      1.64 MB |  1,679x
  1000 |      16.1 MB | 16,486x

The dominating portion of that table are reading the bloom filters. This is why production databases aim to maintain the bloom filters and indexes of all frequently accessed SSTs in memory. It would drop the disk-read amplification by orders of magnitude (though as you have more SSTs in a log-like system the memory pressure for this would be infeasible).

space amplification

This gets us to the space amplification of the log structure. Since we never delete or merge SSTs in a log (more on that later), any SST that contains a duplicate key will have garbage data. How much you pay in space amplification with your log depends on your use case: some (like an audit log use case) would pay nothing since you never overwrite entries, others can pay a significant price.

This is a good time to introduce the concept of a tombstones. How do you delete data if SSTs are immutable? The answer is to, ironically, write more data. If I have an SST 0 on disk that contains keyA -> valueA, I could write a new value in SST 1 that contains keyA -> [deleted]. When I follow my algorithm for “find first instance of keyA in the database”, I’ll find [deleted] in SST1 and know that I should not continue looking in SST 0.

As you can tell, deletions and updates cause more disk space amplification in a naive log system.

eager merge

I have yet to see a production system that exhibits this behavior, but it’s interesting to illustrate the polar opposite of a simple log. I’m calling this an “eager merge tree”, a data structure that will always merge any new SST with the old SST as soon as its written into a new merged SST. This works around the immutability by constantly rewriting the old data and discarding it:

read amplification

This system is ideal for reads. You only ever load up the single SST that is on disk, check the Bloom Filter, verify in the index and read the appropriate block. Assuming your single bloom filter and index are in memory (which they would be, since every query reads the same ones) you read a total of 1 block for your key. Using the same parameters as before, that’s a 4KB read for a 1KB key or a 4x read amplification.

space amplification

This system is excellent for steady-state space amplification. You only ever have a single SST on disk, so there’s no garbage data maintained. Compaction is also memory-efficient since you hold one block from each input SST at a time, interleaving them and writing out new blocks as you go.

The catch is temporary disk space during compaction. You can’t delete the old SST until the new merged SST is fully written and synced to disk or a crash mid-compaction would lose data. This means you briefly need 2x your data size in disk space every time you compact. For a 1TB dataset, you need 2TB of disk available even though you’ll only use 1TB in steady state2.

In other words, you need to over-provision your disks to keep steady state utilization below 50%.

write amplification

I’m certain you saw this coming. The eager merge tree is so atrociously bad from a write amplification perspective that it becomes infeasible for all but effectively immutable data sets. Every time you write a new SST and merge with the old one, all data is read and rewritten.

Eager Merge Tree: Write Amplification
(4GB buffer, 1 MB/s writes, 1KB keys)

Data Size | Logical Writes | Physical Writes |   Amp (x)
----------|----------------|-----------------|----------
     4 GB |           4 GB |            4 GB |        1x
    40 GB |           4 GB |           40 GB |       10x
   100 GB |           4 GB |          100 GB |       25x
   400 GB |           4 GB |          400 GB |      100x
     1 TB |           4 GB |            1 TB |      250x
     4 TB |           4 GB |            4 TB |    1,000x
    10 TB |           4 GB |           10 TB |    2,500x

There is almost a brutal simplicity to this: even at 400GB, every 4GB of new data costs you 100x write amplification. Your SSDs won’t last long with this strategy.

log structured merge trees

Let’s build intuition for LSM trees by exploring the middle ground between the two extremes of logs and eager merging. The log based approach shows us that the read amplification is proportional to the number of SSTs. The eager merge approach fixes this, but introduces significant write amplification caused by repeatedly merging small new SSTs with a single large “main” SST.

We could start searching for a middle ground by increasing the size of the “small new SST” before merging it with the main one in an eager-tree. If my dataset is 1GB, I can improve my write amplification by by buffering larger “new” SSTs before merging them:

Assuming the new SST only updates existing rows so the new compacted “main” SST is the same size as the old one, you can see that the write amplification is improved the more data I buffer up before compacting. This improvement is helpful but still limiting. Buffering more data before flushing it out causes increased memory pressure and doesn’t help you if the size of your main SST continues to grow.

The main insight of an LSM tree is that it provides a continuum between the log and the eager merge structures by maintaining a log for the most recent data, and then “levels” of merged data that increase in size to keep amplification manageable.

SSTs from new writes to the database go into the log. Then you maintain a series of groups of SSTs where each group is larger than the previous group by some multiplier. You can then trade off read and write amplification by tuning the multiplier or the number of SSTs you allow to accumulate in a group.

Before we dive into the specifics of how these trees are structured, there are some common concepts that are shared between all of them:

1. A level of a tree is typically referred to as LN (e.g. L1), with the number representing the depth of the tree. In other words L0 contains the most recent data, and data sinks downward to L1, L2, ... over time through compaction. Note that if you think of trees in terms of “height” instead of “depth” you’re going to trip over the number scheme. I recommend you disabuse yourself of that notion whenever dealing with LSM trees.

2. A sorted run is a set of SSTs that do not overlap in keys, but together cover the entire keyspace. In other words, if Sorted Run 100 contains SSTs 101, 102 and 103 then those SSTs might cover keys [a-g], [h-m], [n-z] respectively (you would not find a key that starts with k in SST 101 or 103, only in SST 102). These are useful because they allow you to maintain smaller filters and indexes for each SST while using the min/max to find the right SST within a sorted run.

leveled trees

A discussion of LSM trees usually starts with Leveled Trees, since this was their first widely used implementation with LevelDB from Google3. The structure of a Leveled Tree looks like this:

The first level (L0) is special4: each SST in L0 covers the entire key range. In the diagram above it’s possible that key foo exists both in SST100 and SST101. Every deeper level in the tree (L1+) is made up of sorted runs, each of which are split into SSTs with non-overlapping keys. In other words, if the Sorted Run for L1 is split into three SSTs (with ids 88, 89 & 90) then a key that shows up in SST88 is guaranteed not to show up in SST 89 or 90:

The algorithm for making this happen is called Leveled Compaction, and it looks a lot like a combination of the two “naive” strategies covered previously. New writes go into a simple log structure by just appending new SSTs into L0. This allows a Leveled tree to accept writes quickly without paying any cost for compaction.

Then in order to improve read performance, L0 is periodically compacted when it accumulates too many SSTs. Since L0 files can overlap with each other, compaction typically selects multiple L0 files and merges them with any overlapping SSTs in L1 (if any exist). The output is a set of new, non-overlapping SSTs that are placed into L1.

When a deeper (L1+) level exceeds a configured size (configured as a multiple of the previous size to keep the sizes growing exponentially, 10x is common), one or more SSTs are selected and merged with the overlapping SSTs from the next level down. This cascades as needed: if the new level exceeds its size limit, some of its SSTs will be merged into L2 (10x larger than L1), and so on until the tree stabilizes.

read amplification

Leveled compaction’s main point contribution over a simple log is to improve read amplification by bounding the number of SSTs you need to check. For L0, you still need to check every SST since they can overlap, but deeper levels only need to check at most one SST. This means the worst case lookup is:

L0_count * sizeof(bloom_filter)
+ num_levels * sizeof(bloom_filter)
+ bf_fp_rate * (L0_count + num_levels) * sizeof(index + block_size)
+ sizeof(index + block_size)

Assuming nothing is kept in memory (which is unlikely, but helps illustrate the point) we can compare a leveled tree to a simple log when it comes to read amplification:

Read Amplification: Leveled vs Simple Log (10x ratio, 4 L0 files)

Data Size | Levels | Leveled |  Simple Log
----------|--------|---------|-------------
     1 GB |      4 |  160 KB |      196 KB
    10 GB |      5 |  176 KB |    1,640 KB
   100 GB |      6 |  192 KB |   16,036 KB
     1 TB |      7 |  208 KB |  160,036 KB
    10 TB |      8 |  224 KB |    1,600 MB

The leveled tree’s read amplification grows logarithmically (one bloom filter per level) while the simple log grows linearly (one bloom filter per SST).

space amplification

Alongside with read amplification, Leveled compaction improves on both naive designs on space amplification. There’s very little garbage data because (a) SSTs in each level has no key overlap and (b) compaction aggressively merges data downward.

Roughly 90% of your data lives in the last level (assume 10x ratio). This means that in the worst case when the entire remaining tree are updates to the last level, the garbage in the tree is still only 10% of the data size (space amplification is ~1.1x).

Temporary disk space during compaction is also well-bounded. Unlike eager merge, which rewrites the entire dataset and needs 2x disk space, leveled compaction only rewrites the SSTs involved in a single compaction. You need enough headroom to hold the old and new SSTs simultaneously, but that’s a small fraction of your total data rather than all of it. To summarize the comparison:

                   |  Leveled Tree  |  Simple Log  |  Eager Merge
-------------------|----------------|--------------|-------------
Disk overhead      |         ~1.11x |       1-10x+ |       ~1.0x
Memory overhead*   |   logarithmic  |       linear |    constant
Temp disk space**  |     ~11 SSTs** |          0%  |       ~100%

*  Bloom filters + indexes scale with SST count  
** Space for old + new SSTs during a single compaction; multiple 
    concurrent compactions would increase this proportionally

write amplification

Hopefully you’ve been paying attention, so before you read this section take a step back and guess what you think the write amplification behavior of a Leveled Tree is (generally, compared to a simple log).

If you applied the RUM conjecture to guess that this is where the cost comes in, you are right! Every time data compacts from one level to the next, it gets rewritten. Because each level is 10x larger than the previous, data from the smaller level must merge into the larger one about 10 times before the larger level crosses its size threshold and compacts down.

To determine the write amplification, we want to know how many times a key get rewritten at each level. On average, a key will arrive halfway through the level being filled out, so it gets rewritten about 5 times before moving to the next level. This leads us to estimate the write amplification:

write_amp ≈ num_levels × (size_ratio / 2)

Leveled Tree: Write Amplification
(10x level ratio, 1KB rows)

Data Size | Levels | Expected Write Amp*  
----------|--------|---------------------
     1 GB |      3 |                  15x 
    10 GB |      4 |                  20x 
   100 GB |      5 |                  25x 
     1 TB |      6 |                  30x 
    10 TB |      7 |                  35x 

* Actual may be lower because not all data makes it to the deepest level, and compaction selection tries to minimize overlap.

While this is way worse than the simple log, it is many times better than the strict eager merge strategy. For read-heavy workloads leveled compaction is often worth it.

size tiered trees

The second main type of LSM compaction strategy attempts to improve with respect to write amplification of a Leveled Tree at the cost of read amplification. The main innovation with size tiered trees is that each tier (the analog to a Level in Leveled Compaction) can have multiple sorted runs which are never partially modified. This is more similar to having each tier be its own “simple log”, periodically compacting an entire section of that log down into the next tier.

In the diagram below, you can imagine that Sorted Run 11 was created when 4 L0 SSTs each of size 64MB were ready to compact. The next four created Sorted Run 12 without touching 11 at all. When there were 4 sorted runs in Tier 1 they are merged together into a new sorted run in Tier 2 with size 1GB, so on and so forth until they’ve merged their way down to Tier N.

write amplification

A good description of the comparison between the two comes from the RocksDB documentation, so I’ll paraphrase that here:

In size tiered storage, new writes move entries from a smaller sorted run to a much larger one. Every compaction is likely to make the update exponentially closer to the final sorted run, which is the largest.

In leveled compaction new writes are compacted more as a part of the larger sorted run where a smaller sorted run is merged into, than as a part of the smaller sorted run. As a result, in most of the times an update is compacted, it is not moved to a larger sorted run, so it doesn’t make much progress towards the final largest run.

If each tier merges R sorted runs into one larger run, then a key is rewritten approximately once per tier. This yields:

write_amp ≈ number_of_tiers

Or, using the same assumptions as before, here’s a write amplification comparison between a Size Tiered and a Leveled Tree, which you can see is about an order of magnitude of an improvement:

Write Amplification: Size-Tiered vs Leveled

Data Size | Tiers | Tiered WA (≈ T) | Levels | Leveled WA (≈ L×5)
----------|-------|-----------------|--------|---------------------
     1 GB |     3 |             ~3x |      3 |                ~15x
    10 GB |     4 |             ~4x |      4 |                ~20x
   100 GB |     5 |             ~5x |      5 |                ~25x
     1 TB |     6 |             ~6x |      6 |                ~30x
    10 TB |     7 |             ~7x |      7 |                ~35x

read amplification

And as expected, the order-of-magnitude improvement in write amplification is paid for in read amplification. Unlike leveled compaction, size-tiered compaction does not enforce a single sorted run per tier. Instead, each tier may contain up to R overlapping sorted runs (where R is a configurable merge threshold). In the worst case, a point lookup must check all runs in every tier before determining whether a key exists.

The worst-case read cost becomes:

(N × R) * sizeof(bloom_filter)
+ bloom_filter_fp_rate × (N × R) * sizeof(index + block_size)
+ sizeof(index + block_size)

This grows logarithmically with data size (like leveled), but multiplied by R. Compare:

Strategy    | Read Amplification  
------------|-----------------------
Naive log   | O(data_size)
Size-tiered | O(R × log(data_size))
Leveled     | O(log(data_size))

Read Amplification: Size-Tiered vs Leveled
(nothing in memory) - 4 L0 SSTs

          |       |          SSTs |            Reads
Data Size | Depth | Tiered / Level| Tiered   / Leveled
----------|-------|---------------|----------------------
     1 GB |     3 |        12 / 7 |  ~240 KB /  ~148 KB
    10 GB |     4 |        16 / 8 |  ~320 KB /  ~164 KB
   100 GB |     5 |        20 / 9 |  ~400 KB /  ~180 KB
     1 TB |     6 |       24 / 10 |  ~480 KB /  ~196 KB
    10 TB |     7 |       28 / 11 |  ~560 KB /  ~212 KB

space amplification

Size Tiered Trees sit somewhere in the middle between a simple log and leveled compaction for their space amplification requirements. Because sorted runs overlap in key space, obsolete versions and tombstones can accumulate within a tier until a full merge happens. In the worst case, each tier can contain up to R copies of the same key (one per run), but in practice real workloads tend to do better than this.

time windowed structures

The last main type of tree structure commonly seen in LSM trees are time-windowed ones. They are worth an honorable mention because they serve as an “asterisk” on the RUM conjecture: if you know that a workload follows certain patterns, you can improve all three of read, write, and space amplification characteristics. If the assumptions are violated, however, you pay a steep price in either correctness or by rewriting the entire dataset in a major compaction.

Time-Windowed Compaction Strategy (TWCS) works by assuming that data arrives roughly in time order and that there is a practical bound on how late a write can arrive. With this simplification, the database is split into two parts: a mutable region for the current time window and a set of immutable regions for historical windows. New SSTs are written into the current window, and once a window closes, it becomes immutable except for compactions within that window.

This improves read amplification (assuming reads are also time-scoped) because queries can skip entire historical windows instead of scanning the full tree. It improves write amplification because data does not “bubble down” through progressively larger structures; once a window is sealed, its data is never merged with newer data again. Finally, because compaction is restricted to occur within a single window, both steady-state and temporary space amplification remain bounded by the size of a window rather than the total size of the dataset.

I’ll keep the discussion on this short for now, since this is already a lengthy post, but if you’re curious about more details there’s an interesting discussion on the original Cassandra JIRA ticket that introduced it.

compact()

That’s it for today! Join us next time for a deep dive on B-Trees and why (I believe) LSM trees are having their moment in the sun for object-storage native systems.

SSDs and memory

First, it’s important to understand how data gets from disk in to usable memory. Most cloud instances use SSDs, so we’ll focus on that instead of spinning disks.

If you haven’t read the initial frameworks blog post of this series I recommend starting with that. In this context, the main point to takeaway is that not all read amplification is created equal. All databases need to abide by the laws of physics: fetching data that’s already in memory is hundreds of times faster than fetching data from over the network. Data structure design is all about minimizing the amount of data you need to fetch from expensive storage tiers while reducing the memory overhead (this an application of the RUM conjecture).

pages on storage devices

A high performance data system needs to reduce the amount of unnecessary bytes read to serve a query, reading only the necessary (”hot”) data.

An ideal system would read exactly the bytes it needs for the data it wants, but the fundamental unit of I/O isn’t a byte. On SSDs, it’s a page (typically 4KB). This means that whether you request a single byte, a hundreds bytes, or four thousand bytes from your disk you’ll still get the 4KB.

You can verify this on your own machine:

# on MacOS, check the size of the page on your disk
> stat -f %k /
4096

To see this in action, I ran an experiment measuring read latency for 1KB vs 4KB reads using Direct I/O (bypassing the OS page cache). Despite requesting 4x less data, the 1KB reads took about the same time as the 4KB reads (about 9.2µs) on my SSD1:

Here’s where it gets interesting for database design. Imagine you’re serving a query that needs a single 256B row that lives somewhere in a 4KB page alongside other rows:

To read row3, you have to read the entire page. Rows 1 through 10 come along for the ride whether you want them or not. The ratio of “data you read” to “data you needed” is an instance of read amplification. In our example, if you needed 256 bytes but read 4KB, your read amplification is 16x. That sounds sub-optimal, but it’s unavoidable given the hardware constraints.

spatial & temporal locality

The read amplification from page size and sizes isn’t all bad news.

There’s a fixed cost overhead to reading a single page from disk. On an SSD the actual data transfer part is less than 1-2% of the total latency (the rest comes from processing the command, translating the logical address to the physical location on the disk, etc…). This means that it takes approximately the same amount of time to read 1 page, independent of whether that block is sized 512B on a 512B system or 4KB on a 4KB system.

To take advantage of the page size, databases attempt to place data that is commonly read together physically close together on the storage device, ideally in the same page. Typically this means one of two things: either the data keys are the similar (spatial locality) or the keys were written around the same time (temporal locality).

This technique helps reduce the overhead of read amplification that comes from reading data you won’t access. As an example, if I’m executing a query that sums the total cost of orders from a particular customer my query execution path is likely to perform a loop that looks something like:

total = 0
for order_id in [cust2_order1, cust2_order2, cust2_order3]:
	let order := db.get(order_id)
	total += order.cost

If my first call to db.get(cust2_order1) fetches a 4KB page from disk that contains the orders cust2_order2 and cust2_order3 then when I next call db.get(cust2_order2) that data will already be in the OS page cache, which will make it much faster to load.

This is why designing the key for your database is so important. It is the first attempt to make sure that you can take advantage of the spatial locality of the data on disk.

mutability on SSDs & blocks

Unlike hard drives and spinning disks, data that’s written onto an SSD cannot be directly replaced without first erasing that data. The electromagnetic physics behind it is beyond my expertise, but the way I think about it is that while you can write a small (4K) page, you cannot target erasure at that level. Instead, erasure works at a much higher level called a block (typically 128-256KB).

Because of this effect, when you rewrite data on an SSD you are actually writing a new data block and telling the SSD controller (a piece of firmware that comes loaded on your SSD) the new location of the data. Eventually, the controller will erase large chunks of garbage data in a mechanism that looks very similar to handling garbage collection in a memory allocator.

SSDs, therefore, much prefer that pages are not modified over and over again. Instead they become invalid in large ranges at a time. This means that immutable on-disk data structures work well with SSDs and any mutable structure will cause significant write amplification at the hardware level. We’ll get to the implications of this a little more in a future blog post about B-Trees and LSM trees.

For now, we’ll draw the conclusion that immutable storage formats have an edge.

storing data durably on SSDs

To recap the above sections, SSDs push you to use a data structure that:

1. Is written and deleted in large batches aligned to the internal block size (e.g. 256KB)

2. Is immutable to avoid the overhead associated with rewriting data

3. Organizes data in a way that clusters related data together to take advantage of the page size

There are several ways to organize immutable data durably that meet these requirements, the simplest of which is an append-only log. In a log, the system writes records sequentially (aligned to block size) and scans from the beginning for reads. You can see why this works well with SSDs:

1. Logs let you batch data together and write them when you have enough data.

2. Logs are immutable, typically with some retention at which point you drop entire sections of it.

3. Logs organize data in a way that is optimized for the particular read pattern of reading data in the order it was written.

The downside of a log is the performance of random reads: you might need to scan the entire file to find a specific key. Some systems (like BitCask) address this by storing the entire key-set in memory with pointers to the location on disk stored in memory.

To make reads directly from disk efficient without exploding memory usage, you need some other kind of structure. There are a number of different strategies, but for this post I’ll focus on a typical approach used by row-based systems (SSTs).

Sorted String Tables (SSTs) build off the ideas from the log to play well with the limits of SSDs (you’ll see the similarity even further when we talk about LSM trees). If you think of a log as an array sorted on an implicit timestamp key, SSTs are essentially the same structure but sorted on a user-defined key instead. To make this possible, databases that use SSTs first buffer data sorted by that key in memory until they’ve collected enough data to write a large, immutable batch.

sorted string tables (SSTs)

This section will go over the design and implementation of a basic SST: an immutable storage data structure for key-value data. Here’s a reference to look back on that covers the big-picture of an SST layout (this is inspired from the SlateDB SST layout, though other implementations such as RocksDB are quite similar):

data blocks

The core concept of a sorted string table is to store data on disk sorted by the lexicographical ordering of the byte representation of their keys. In other words, a single data block of an SST can be thought of as a byte array constructed like this:

struct Record {
	key: Vec<u8>,
	val: Vec<u8>,
}

struct SstDataBlock {
	data: Vec<Record>,
}

impl SstDataBlock {
    fn from_unsorted(data: Vec<Record>) -> Self {
        let mut data = data;
        // sorts the records by the lexicographic order of the keys
        data.sort_by(|a, b| a.key.cmp(&b.key));
        Self { data }
    }
}

This strategy is nice for two reasons: first, it takes advantage of the spatial locality principle we discussed above. Similar keys are stored next to one another. Second, and perhaps more fundamental, it allows us to binary search the data within an SstDataBlock:

fn find_record(&self, key: &[u8]) -> Option<&Record> {
    self.data.binary_search_by(|record| record.key.as_slice().cmp(key))
        .ok()
        .map(|index| &self.data[index])
}

But this relies on the in-memory representation of the SstDataBlock. We still need some way to get this struct onto disk which only understands simple byte arrays [u8]. There are many strategies for serializing the data block of an SST; production data systems will often use techniques such as prefix encoding for keys and varint encoding for lengths to reduce the amount of space a block takes up but for this exercise we’ll do something simpler: we’ll just encode the block as:

And a code mockup for reading and writing these SstDataBlock instances:

fn encode(&self) -> Vec<u8> {
    let mut encoded = Vec::new();
    encoded.extend_from_slice(&(self.data.len() as u32).to_le_bytes());

    for record in &self.data {
        encoded.extend_from_slice(&(record.key.len() as u32).to_le_bytes());
        encoded.extend_from_slice(&record.key);
        encoded.extend_from_slice(&(record.val.len() as u32).to_le_bytes());
        encoded.extend_from_slice(&record.val);
    }

    encoded
}

fn decode(data: &[u8]) -> Self {
  // decode is more verbose so it's emitted here but it just reads
  // the number of records, then deserializes them one by one reading
  // the length of the key, then the key, then length of the value,
  // then the value itself
  ...
}

indexes

The data block covers how to get a record from within a block, and technically this is all you need for a functioning SST.

Simply storing data as a series of sorted blocks, however, isn’t ideal. If that’s all you did, your query algorithm would end up reading blocks in a binary-search pattern fetching entire blocks to decompressing them just to see whether or not the key you’re looking for is even within the range of the block.

To solve this problem, SSTs introduce multiple layers of indexes.

the main index

The first index (often just called “the index”) is a smaller data structure that just contains the first key of every data block and the location of the block (an offset within a larger file). This allows you to load the index into memory and binary search on that, which is an order of magnitude smaller than trying to load the actual blocks, to find the required data block.

To put the index size for corresponding data blocks into perspective, let’s imagine you have 4KB blocks with keys that average 8 bytes (a u64) and values that average 256 bytes. This means a single data block can contain approximately 15 records. The index in the most naive format can contain 512 keys (with delta compression you can squeeze significantly more). If each key is the first key of a block, a 4KB index block can index 2MB of data blocks, or 7680 records.

The code for the index block would look something like this (without the encoding/decoding section, which looks very similar to the data block encoding and decoding):

pub(crate) struct IndexEntry {
    key: Vec<u8>,
    offset: u64,
}

pub(crate) struct SstIndexBlock {
    entries: Vec<IndexEntry>,
}

/// used to find which block in the SST contains the key
/// that you want using the index. once the data block is
/// identified, it should be deserialized into a SstDataBlock
/// and then searched using SstDataBlock::find_record for the
/// exact record
impl SstIndexBlock {
		fn find_entry(&self, key: &[u8]) -> Option<&IndexEntry> {
        match self.entries
            .binary_search_by(|entry| entry.key.as_slice().cmp(key))
        {
            Ok(i) => Some(&self.entries[i]),
            Err(insertion_point) => {
                if insertion_point > 0 {
                    Some(&self.entries[insertion_point - 1])
                } else {
                    None 
                }
            }
        }
    }
}

filter indexes

Up until this point, we made the assumption that all the data you’d need exists in a single SST. We’ll discuss why you might not want that in a future blog post about LSM trees, but for now I’ll ask you to accept that it’s often better to store data in multiple SSTs instead of one big one.

If you setup your storage like that, it is helpful to know whether or not a given SST has the key you’re looking for before you even try reading the index and data blocks. This information is stored in a filter block, and there’s typically two forms of filters that are used: min/max filters and Bloom filters.

Min/Max filters are extremely simple filters that just encode the minimum and maximum keys that exist in the SST. This very simple data structure takes up very little space and can save a lot of computational work in certain storage systems that lay out their SSTs across very wide key ranges (think using a number such as timestamp as the key).

Bloom filters are more complicated and come with some interesting tradeoffs. There have been many sources that explain how these filters work, if you’re curious the Wikipedia page has an excellent overview. If you’re not, the important thing to understand is that they’re a probabilistic data structure that can tell you with 100% certainty that a key does not exist in SST but it cannot guarantee that a key does exist. Their accuracy is measured by their false positive chance: each false positive means you need to dig into the index and data blocks unnecessarily, finding out that the key in fact does not exist.

index space amplification

Indexes and filters aren’t free and, typically, you want to keep the entire index along with any filters in memory to avoid deserializing indexes over and over again.

The implication is that indexes are a fundamental way to tradeoff between read and space amplification.

To tune between the two, you can change the size of the data blocks (smaller blocks means larger indexes, but less read amplification as less data needs to be paged in). I ran an experiment on RocksDB in practice to show this in action2. The experiment set up a data set of 4GB size and varied the block size, checking the total size of the SST (on disk), the size of the index (in memory) and the read throughout.

$ space_amp --block_sizes=4096,8192,32768 --read_ops=50000
Raw payload bytes: ~4GB (33554432 entries)

Block Size    Total SST   Table Mem   Reads/s
4.00KB           3.51GB      38.9MB     13144
8.00KB           3.48GB      19.3MB     11310
32.0KB           3.45GB      4.88MB      9311

This shows what we expect to see: smaller block sizes mean larger indexes, higher memory utilization but faster reads. It also shows the diminishing returns of large indexes. When we decrease the block size 8x, memory grows by approximately the same factor but the read throughput only improved by 1.4x.

Bloom filters are not exempt from this tradeoff either: to reduce the false positive rate of a bloom filter, you need a large one (more bits). The good thing is that the math to compute the false positive rate is straightforward. Assuming your SST stores 1 billion keys, the diagram below shows the false positive rate and the corresponding Bloom Filter size.

metadata block

The final block in an SST is the metadata block. Despite its humble size, typically just 48-64 bytes, it plays a critical role by storing the offset and length of the index and filter blocks, plus a magic number and version that allows the SST format to evolve over time.

But here’s an interesting question: Why are the index and bloom filter blocks at the end of the file?

The answer reveals a constraint of SST construction. When building a large SSTable you face a choice. Your first option is to buffer the entire SST in memory, build the complete index and bloom filter, then write everything in the “correct” order: index, bloom filter, data blocks. Alternatively, your second option is to stream data blocks to disk one at a time as they’re built, constructing the index and bloom filter incrementally as you go. If you take the second path, then you can’t know the locations of the blocks until the end, so you can’t write out the index block until then.

Note that this optimization only really applies when your data is already in sorted order (such as when merging two SSTs), otherwise you need to buffer the data in memory to sort it in first place.

SSTs are general purpose building blocks

While SSTs are an obvious fit for key-value storage, sorted rows turn out to be a remarkably general abstraction. Almost any access pattern you care about can be encoded into SSTs if you’re clever about how you structure your keys.

The insight is that lexicographic sorting on byte strings gives you hierarchy for free. When your keys share a prefix, they’re stored physically adjacent. This means a “range scan over all keys starting with X” is just a sequential read which is the best possible access pattern for storage devices (remember that the minimum fetch of data from a disk is 4KB, even for an SSD, so if all of that data is relevant then you aren’t paying excess read amplification).

encoding access patterns into keys

There’s a spectrum of cleverness in how you shove data models into key-value form. This section goes through strategies you can use, from obvious to devious:

level 0: plain key-value lookups

The trivial case models the keys exactly the way you are looking them up. There’s nothing clever about this and often this is all you need:

"user:alice" → {name: "Alice", email: "alice@bitsxpages.com"}
"user:bob"   → {name: "Bob", email: "bob@example.com"}

level 1: key namespacing

The next requirement might be to store different types of data in the same SST rather than split each data type into its own SST. To do this, you can prefix your keys with the type of data they represent and then construct your lookup key based on that.

The following example has three key types: metadata keys, users and sessions:

"meta:schema_version" → "3"
"meta:created_at"     → "2024-01-15T00:00:00Z"
"user:alice"          → {name: "Alice", ...}
"user:bob"            → {name: "Bob", ...}  
"session:abc123"      → {user: "alice", expires: ...}

Because keys are sorted, all the meta: keys cluster together, all the user: keys cluster together, and so on. A range scan with prefix user: gives you all users. This is effectively multiple logical tables in one physical SST.

level 2: composite keys for hierarchy

This is when things start getting interesting. You can effectively “decompose” a single complicated key into multiple rows for efficient lookups of both the composite key as well as the single key. Imagine our users had a set of orders, so the logical data would look like:

"user:alice" → {
	name: "Alice", 
	orders: [
		{"id": "7", "item": "catnip", "count": 2},
		{"id": "112", "item": "dog_treats", "count": 11},
	]
}

Using level-0, and level-1 techniques outlined before we could get decently far:

1. Level 0: Store "user:alice" as the key, and include the full order details in the document value (match the logical model)

2. Level 1: Store "user:alice" as the key, include the order ids in the value, and then store "order:7" as a separately namespaced key.

Each have downsides. Level 0 has significant read amplification if you don’t need all the order details (e.g. you just want to retrieve Alice’s email address). Level 1 has significant read amplification if you do want all the order details because you need to fetch all the order ids, which may not be in the same data blocks (they’re ordered by ids, so order:7 and order:112 may not be next to one another).

The third alternative is to use a composite key:

"user:alice" -> {name: "Alice", ...}
"user:alice,order:7" -> {"id": "7", "item": "catnip", "count": 2}
"user:alice,order:112" -> {"id": "112", "item": "dog_treats", "count": 11}

Now the query for “get order 7 for Alice” is a point lookup for user:alice,order:7 and the query “get all orders for Alice” is a prefix scan on user:alice,order:*. Both are efficient since the sort order gives you the hierarchy for free3.

level 2.5: wide columns

The same technique can be used for handling documents with many sparse columns. For example, you could chose to store a phone_number column separately if most users don’t submit a value for that when creating a profile:

"user:alice" -> {name: "Alice", ...}
"user:alice,col:phone" -> "+1-650-123-4567",

This has a nice property: sparse columns are free. If Alice has a “phone” column and Bob doesn’t, you simply don’t store a key for Bob’s phone. The downside is that reading the full row now requires a prefix scan, which involves more read amplification.

level 3: embedding secondary indexes

The hierarchy trick only works if you know every element in the hierarchy you want to query for. This doesn’t help me if I want to find “what user has the email alice@bitsxpages.com?” The solution to that is leveraging namespacing to store special keys that represent the secondary index:

# primary: lookup by user ID
"user:alice" → {name: "Alice", email: "alice@example.com"}

# secondary index: lookup by email  
"<idx:email:alice@example.com>" → "user:alice"

Now looking up a user by email is two steps: scan the index to get the ID, then fetch the primary record. This is one of the most powerful tricks you can use, and allows you to implement many different types of systems on top of SSTs.

The previous example I gave is a simple secondary index, but you can implement more complicated ones:

• A covering index can store more than just document ids in line with the value. For example if I always want to get the user’s full name when I lookup by email the index value for idx:email:alice@example.com could contain {"id": "user:alice", "full name": "Alice Ecila"}.

• An inverted index can be stored in SSTs where the keys are term:value (e.g. department:engineering) and the value is a sorted list of document ids, compressed as a roaring bitmap. If I want to find “all documents that are in engineering and have the name alice” I lookup department:engineering and name:alice, then retrieve the intersection of the two lists.

• A queue can be stored in SSTs where the key is the offset in the key and the value is the full row.

why this matters

The deeper point is that SSTs derive their power from a simple focus on playing to the advantages of SSDs and Object Storage: immutable data and block-aligned reads (index and filter structures that minimize I/O).

This alignment with hardware means you can map a surprising variety of use cases onto SSTs without fighting the storage layer. Cassandra and ScyllaDB use SSTs for scalable KV storage. Yugabyte and MyRocks use SSTs to implement full SQL engines. DataDog’s metrics backend stores data in SSTs. Kafka’s log segments are conceptually SSTs optimized for append-only access. If you generalize SSTs a little further and consider systems that have converged on “sorted immutable files with indexes and filters” as their storage primitive is long and growing (Clickhouse, etc…).

Once you internalize that “sorted bytes on disk + binary search + immutability” is the primitive so much else builds on, the design space for databases becomes clearer.

But the reality is that most databases aren’t immutable… stay tuned for the next entry in this series to learn how multiple SSTs are composed together to form LSM trees.

This introductory post first presents a mental model for understanding data system tradeoffs and then outlines the common building blocks. We’ll follow this up with deep dive series on each of the nine data systems types1.

the mental model

All databases have one purpose: to store data so you can retrieve it. This core similarity means that, if you squint, all databases look similar. Evaluating them becomes a question of understanding tradeoffs that can be made.

I evaluate data systems on three dimensions, and evaluate each of those dimensions with a set of frameworks (we’ll dive deeper into these throughout the post):

1. For reasoning about performance the Read, Write and Space Amplification of a system determine how much work is done on query, ingest and compaction respectively. Then, the indexes and data orientation are evaluated with the RUM conjecture.

2. For reasoning about availability in a distributed system, the PACELC framework is an evolution of the widely referenced CAP theorem. In steady state, you trade off between latency and consistency — only during a network partition do you choose between availability and consistency.

3. For reasoning about durability (especially in cloud native systems), the LCD framework illustrates that you may pick two of Latency, Cost and Durability.

Usage of these frameworks answers how to choose a data system that serves your use case at the acceptable cost (both on hardware requirements as well as operational overhead).

performance

framework

There’s the famous Latency Numbers Every Programmer Should Know chart2 from Jeff Dean that helps reason about why I/O efficiency is so important.

The goal of a data systems’ storage layout is to (a) ingest data into a durable medium as fast as possible and (b) get required data to serve a query into L1 cache as fast as possible. Sometimes that data is already there, other times is on the disk of a machine in another AZ. The latency cost of fetching the latter dominates system performance.

Different systems make tradeoffs on read, write and space amplification:

1. Read amplification measures how many bytes are required to serve a single logical query. More read amplification happens in systems optimized for write throughput.

2. Write amplification quantifies the overhead when a database writes more bytes than strictly necessary to store a piece of data, often due the index structure or compaction.

3. Space amplification captures the ratio between actual storage consumed and the logical data size, accounting for fragmentation, tombstones, and redundant copies. A good example here is a size of a bloom filter (a large bloom filter can help reduce false positives, but takes up more memory).

indexes

A naive database with no index must scan the entire raw storage, potentially across many machines, filtering out points that don’t match the query. Since this is more data than what can fit in L1, much of that data will be elsewhere.

Indexing is the strategy used by data systems to reduce the amount of data scanned. I think about indexes in three categories: primary, secondary and filters.

1. Clustering indexes are how the raw data is organized. For example an LSM Tree (used by SlateDB, Cassandra, etc…) stores the key-values pairs in SSTs, which sort the keys in lexicographical order so a query can binary search to find the required key.

2. Secondary indexes are auxiliary structures that return primary keys3, which can then be used to lookup the raw data. For example, an inverted index (used by Lucene) will return a list of primary keys that contain a particular value (this is particularly useful in search use cases).

3. Filter indexes are often embedded within the primary index to help filter out blocks of data. You’ve likely heard of bloom filters (used by SlateDB, Clickhouse, etc…), which can guarantee that a key you are attempting to lookup does not exist in a particular block of data.

I reference the RUM conjecture when evaluating indexes. This grounds the discussion with the understanding that a single index structure cannot be optimized for read, update AND memory overhead simultaneously. In other words, indexes tradeoff across read, write and space amplification to improve performance of reads, updates or memory utilization.

data orientation

The previous section discussed how the index layout affects how fast you can get data into L1 cache, this section discusses the raw data format and its impact on the types of computations a system can execute.

The primary difference between the way databases store raw data is the way that data is oriented: row or columnar. If your workload reconstructs complete rows choose a row-based data system. If your workload aggregates values across rows choose a columnar system.

Why? In a row orientation, the entire value for a single row is stored in one place, but aggregating across rows for a few columns means reading unnecessary data. In a column orientation, values across all rows for the same columns are stored in one place, but reconstructing an original row means reading unnecessary data.

Data orientation directly impacts read amplification. To illustrate, consider rows of 16KB with 32 columns of 500B each stored in blocks of 64KB. In a row-based system, fetching one row reads a 64KB block but discards 48KB of unnecessary data (read amplification of 4x). In a column-based system, fetching that same row requires reading 32 separate 64KB columnar blocks and discarding ~2MB (read amplification of over 100x). The opposite is true when aggregating columns across multiple rows.

Beyond minimizing read amplification, aligning data layout with workload patterns enables optimizations like vectorized operations (SIMD) in columnar systems, which are only possible because each loaded memory block is a single-typed array.

compaction & garbage collection

Compaction and garbage collection (GC) are common strategies for reducing write amplification in favor of read/space amplification. A data system can only write a single block at a time, which means that if the contents of that block are modified it is no longer entirely valid. Since rewriting a block in-place may be impossible due to fragmentation, the strategy most systems use are to append new blocks that model updates to the old blocks.

Compaction takes files that have overlapping data and compact them back into files with non-overlapping data. These compactions jobs don’t immediately need to clean up the old files that are still sitting around. Garbage collection can run separately, cleaning up orphaned files. This separation allows keeping “backups” to old data.

availability

framework

Few applications need more than 3 nines of availability (~9h of downtime yearly) because downtime violations in excess of that frequently occur as the result of human, not machine, failure.

Despite this, more availability is always better so unless your system falls into the small select group of requiring high availability (HA) it becomes a cost tradeoff: how much does a minute of downtime cost compared to the overhead of operating with HA.

To make sense of tradeoffs in HA systems, I recommend you read up on PACELC, but the summary is that during a network partition the CAP theorem applies, but otherwise you tradeoff between consistency and latency.

system architecture

Availability in systems is tied to the system’s deployment architecture. Consider the two-dimensional matrix below. The first axis is the leader/leaderless dimension which models whether or not all writes must be funneled through a single node. The second axis is the disaggregated/distributed dimension which models whether replication is done within the system or delegated to a separate (typically object storage) system:

1. Single Node Deployment: omitted from the diagram above is a non-HA deployment strategy that makes sense for some data systems like caches and edge database deployments. Example: sqlite

2. Single Writer, Multi-Reader: a single-node system with the additional feature of writing to shared storage (e.g. S3) so other readers can read it. Example: slatedb

3. Multi-Writer, Multi-Reader: a deployment mode that allows for writes and reads across many nodes, typically resulting in a async coordination layer. Examples: warpstream, quickwit

4. Leader / Follower: a distributed system that requires writes to go to a single leader, which will replicate to followers before acknowledging the write. Example: kafka

5. Leaderless: this is a distributed system that can accept writes to any machine, but will still replicate to other nodes in the cluster. Example: scylla

The left half of the quadrant prioritize consistency. Since valid writes will only be handled by a single machine, it is easier to provide ACID semantics. Single-writer systems are always consistent, but allow trading off latency for durability (see next section). Leader-follower systems allow configurable consistency by reducing the number of acquired acks from followers. Both systems experience downtime in the event of a writer or leader failure.

Leaderless systems prioritize availability, since a single node failure does not require a leader election process or a new writer to restart. Consistency, on the other hand, requires increased latency because all writes must meet quorum.

durability

framework

The primary consideration for durability is a tradeoff between latency, durability and cost. You can see the durability guarantees of each of the following durability strategies in the chart below:

If you keep data only in-memory on a single node, you have no durability. When you fsync that data to a local disk, you gain a bit more. When you replicate that to a network disk (EBS) or across multiple zones, you get more with each step.

In coupled storage and compute systems, durability was more closely tied to availability. If you were running with 3 replicas across zones and prioritized consistency then you had three nodes that potentially had the data in memory, making durability of the storage a slightly less important concern. In a disaggregated system you no longer replicate within the compute subsystem and delegate that to object storage, the tradeoff becomes more acute.

storage options

Once you’ve made your decision on how durable your data needs to be, the next choice is where to store durable data. The chart below shows the latency and cost deltas between different solutions:

It is possible to have a tiered write system: recent writes can go to a faster storage solution and eventually tier to the slower solutions. This is an example of the LCD tradeoff: if recent writes go to local disk, they can be fast (and cheaper than in memory) but are less durable to an outage.

WAL (write ahead log)

Write ahead logs (WALs) are a strategy that illustrate the tradeoffs of both the LCD and RUM conjectures. Data systems will write changes to the database in a write-optimized format (effectively logging the request with no transformations) to a durable storage. Using the LCD framework, WALs increase cost (by increasing write amplification) so that writes can be durable with lower latency. WALs are also an extreme example of UM data structures (update & memory optimized), but reading from them is inefficient and reserved for failure situations.

fsync()

That’s it for today folk! I hope this has helped shaped the way you understand and evaluate different data systems. Next time, we’ll pick up key value stores as the first data system we’ll examine in more depth to see how we can apply this framework to a specific type of database.