TL;DR
Aiven benchmarks Diskless Kafka (KIP-1150) achieving 1.6s P99 latency while reducing annual infrastructure costs from $3.32M to $288k through object-storage-first architecture.
Key Points
- 94% cost reduction: $3.32M annual infrastructure spend cut to $288k on 6 m8g.4xlarge instances
- Performance: 1 GiB/s ingress, 3 GiB/s egress across 3 AZs with <30% CPU utilization and ~1.6s P99 end-to-end latency
- Open benchmarks published with reproducible OMB configs and raw results; Diskless runs alongside classic Kafka on same cluster
- Trade-off: Eliminates local disk/cross-zone replication costs but adds S3 latency; metadata operations generate 1.4 MiB/s cross-AZ traffic
Why It Matters
For operators running high-throughput, fan-out-heavy Kafka workloads (99% of deployments), Diskless topics offer dramatic cost savings without forking Kafka or sacrificing protocol compatibility. The open-source benchmarks provide production operators concrete data to evaluate object-storage-first streaming architecture rather than relying on marketing claims.
Source: aiven.io