For years, Apache Kafka has been the undisputed backbone of event streaming. From a technical standpoint, it works seamlessly: high throughput, robust fault tolerance, and guaranteed ordering per partition. The issue isn’t that Kafka is slow; it’s that in hyperscaler cloud environments, Kafka is expensive. These costs don’t stem from code inefficiencies, but from structural architectural choices that are difficult to compress within the current model.
In this article by Cristian Bianco, Software Engineer at Bitrock, we will explore one of the most radical evolutions of Apache Kafka: the shift to the Diskless model, a true game-changer for infrastructure cost optimization.
With KIP-1150, accompanied by the related proposals KIP-1163 and KIP-1164, the Apache Kafka community is tackling this problem head-on. This isn’t a feature designed to reduce latency or increase throughput; it is a proposal intended to drastically reduce the operational cost of a Kafka cluster in the cloud by permanently decoupling the compute layer from the storage layer.
Why Kafka is Expensive in the Cloud
Kafka’s classic design generates high infrastructure costs in cloud environments. This isn’t due to inefficiency, but rather how it is built: every architectural choice that guarantees reliability and ordering has an economic counterpart that was negligible in on-premise environments but becomes highly visible in the cloud.
- Cross-AZ Traffic. Replica synchronization across different Availability Zones is continuous and inevitable. On platforms like AWS or GCP, data transfer between AZs comes at a cost. With large clusters and three-AZ replication, this item quickly becomes the most significant line on the infrastructure bill. It is not a cost that can be optimized via code; it is structural to the replication model.
- Disk Over-provisioning. Kafka brokers are stateful nodes: the local disk is an integral part of the process. This means disks must be sized for peaks, not averages. Those with variable workloads inevitably end up paying for storage they only use part of the time. When space runs out, the solution is rebalancing: physically moving data between brokers—a slow, expensive operation that locks up resources during execution.
- The Limits of Tiered Storage (KIP-405). Tiered storage, introduced to move “dormant” segments to object storage, was a first step in the right direction. However, it only solves the problem for historical data. Active data—produced and consumed in real-time—still resides on local disks, incurring block storage costs and Cross-AZ replication fees. The structural problem remains.
Architectural Innovation: Diskless Topics (KIP-1150)
KIP-1150’s answer to these problems isn’t to optimize the existing model; it’s to change it at its root. The core idea is to move the source of truth from the brokers’ local disks to object storage: Amazon S3, Google Cloud Storage, or equivalents. Brokers stop being stateful nodes and become pure compute nodes. The costs derived from block storage and Cross-AZ replication traffic largely disappear in this model.
- Object Storage as the Source of Truth. Produced messages are buffered in memory and uploaded directly to object storage in immutable files called WAL Segments (Write-Ahead Log). There is no longer a primary write to a local disk. Object storage is the log, not a secondary archival layer. The cost of block storage, which is significantly higher than object storage per GB, is replaced by a much more economical model.
- The Role of the Diskless Coordinator. Object storage lacks native logical ordering: two concurrent writes do not automatically produce a deterministic order. The Diskless Coordinator is the component introduced to solve this, assigning definitive offsets and establishing the global order of messages for each partition. It performs the function previously held by the leader broker but is decoupled from physical storage.
- Disks as Cache, Not Storage. Follower brokers no longer replicate directly from the leader. They download the necessary segments directly from object storage, using local disks only as a cache to accelerate reads. Cross-AZ replication traffic—the hardest cost to compress in the classic model—disappears: all brokers read from the same globally accessible source.
- Elasticity Without Over-Provisioning. Adding new brokers to handle traffic peaks no longer requires massive data copying. The new node simply begins downloading segments from object storage. It’s not just about speed; it means there’s no longer a need to keep oversized brokers to absorb peaks. You pay for what you use, not for what you might need.
The Cost of Diskless Topics: Latency
Reducing costs with diskless topics isn’t free. There is a specific and important trade-off to understand before adopting them: latency increases.
In the classic model, a message written to a local disk is available to consumers within milliseconds. In the diskless model, messages are first buffered in memory, then uploaded to object storage, and only then made available. Object storage APIs introduce additional latency, pushing the P99 into the 1-2 second range. This isn’t a solvable technical bug; it is the direct cost of the savings.
This makes the choice much easier to frame: if sub-second latency is a requirement, diskless topics are not the right choice. If, however, you work with workloads where a few seconds of delay is tolerable—log aggregation, telemetry, archiving, analytics—the cost savings are real and immediate.
When to Adopt Diskless Topics
The selection criterion is simple: diskless topics are the right choice when storage and network costs are the primary concern, and a few seconds of latency is acceptable for the workload in question.
- High-Volume Log Aggregation and Telemetry. Thousands of microservices producing events, metrics, and traces generate massive volumes at high costs with the classic model. A few seconds of latency does not impact functionality, and the savings are directly proportional to the volume produced.
- Long-Term Archiving. Keeping months or years of events on block disks is expensive. Object storage is designed exactly for this: extremely high durability at a much lower cost per GB. With diskless topics, long-term retention stops being a hard-to-justify cost.
- Workloads with Unpredictable Traffic. Those who currently size their clusters for peaks, paying for unused capacity most of the time, benefit directly from decoupled elasticity. You scale when needed without the storage “baggage.”
When not to adopt them: If sub-100ms latency is a requirement—trading, real-time payments, control pipelines—the diskless model is not the correct choice, at least in its current specifications. Similarly, in on-premise environments without managed object storage, the economic advantages vanish, leaving only additional complexity.
Current Status and Roadmap
At the time of writing, KIP-1150, KIP-1163, and KIP-1164 are still under discussion and voting within the Apache Kafka community and are not part of any stable release. Those wishing to follow the evolution can consult the official pages on Confluence, where the design continues to be refined through community feedback.
The adoption of this architecture will not be automatic: diskless topics will coexist with classic topics. The choice will be made at the individual topic level based on the workload profile, just as retention parameters or replication factors are chosen today.
Conclusion
Diskless topics weren’t created to make Kafka faster. They were created to make Kafka cheaper in the cloud. This is an important distinction because it completely changes how adoption is evaluated: it is not a technical upgrade, but an economic decision with a clear and well-defined trade-off.
Those managing large-volume Kafka clusters in multi-AZ cloud environments, whose workloads tolerate latency in the one-second range, have a concrete proposal for reducing operational costs. Those working with stringent latency requirements can continue to use the classic model; the two approaches will coexist, selectable at the single-topic level.
At Bitrock, we turn infrastructure challenges into competitive advantages. Discover our Engineering services and how we can support you in the next phase of your digital evolution.
References
KIP-1164: Diskless Coordinator
___________________________________________________________________
Main Author: Cristian Bianco, Software Engineering @ Bitrock