Thoughts on benchmarking streaming systems

Last updated 2021-09-06

See this post for an overview of the systems in question.

What is the point of benchmarking?

Output of a good benchmark:

The above output is per hardware configuration, per query set, per data set. Far too many variables to explore exhaustively. We need a model of performance.

Existing public benchmarks do none of this. They usually contain one (simple) query, one data set, one hardware config. They're only useful for marketing.

Benchmarking practices:

Focus on benchmarking vertical scaling first:

Micro benchmarks:

Macro benchmarks: