Benchmark Overview

This document describes the benchmark methodology, test principles, and release-specific benchmark history for rekuiper.

---

1. Test Principles

All benchmarks adhere to three strict principles:

1. Equal Resource Limits: Each engine runs inside a Docker container on Linux. The host limits each container to one physical CPU core and 1 GiB of RAM. Swap memory is disabled.
2. Exact Output Proof: Sinks write all processed output to disk. Verification scripts parse the output to prove zero message loss, zero duplicate messages, and correct calculations.
3. Reproducible Workloads: Dedicated generators send synthetic events on strict schedules outside the engine container.

---

2. Test Environment

Both rekuiper and competitor engines run with identical hardware constraints:

Parameter	Constraint
Host System	x86-64 multi-core processor, Linux kernel 6.x
CPU Assignment	Pinned dedicated physical core (--cpuset-cpus=2 --cpus=1)
Memory Limit	1 GiB maximum (--memory=1g --memory-swap=1g)
Network	Dedicated Docker bridge network (rk-bench-net)
Metrics Collection	Linux cgroup v2 interfaces (cpu.stat, memory.stat, memory.current) sampled every 1.0 second

---

3. Sustainability Criteria

A test run is sustainable when the engine processes events at arrival rate without internal backlog:

Stable Memory Plateau

Under steady load, engine memory must form a flat level line.

• The sampler records anonymous heap memory (anon from memory.stat) each second.
• If memory variation in the second half of the test window is small ($\Delta M \le 2.0\text{ MiB}$), the queue is empty.
• If memory increases continuously, internal queues store undelivered messages. This condition is not sustainable.

Zero Drain Latency

When the generator stops, the engine must finish processing immediately.

• If the output sink closes within 1.0 second of the send window, drain latency is zero.
• If the sink continues to write records after the generator stops, data accumulated in internal buffers during the test.

---

4. Benchmark Catalog by Release Version

Different benchmark suites evaluate specific capabilities across versions:

Suite Name	Evaluated Version	Workload Type	Key Result	Link
Concurrent Parallel Rules	rekuiper 0.505-beta	Stream fan-out to $N$ parallel SQL rules	Sustains 1,000 parallel rules (500,000 evals/s) on 1 core; eKuiper ceiling is 100 rules.	View Benchmark
Single-Rule MQTT Throughput	rekuiper 0.500-beta	Ingestion of 5 industrial MQTT shapes	Sustains 150,000 to 200,000 msg/s loss-free on 1 core; 6.3x to 10x higher than competitors.	View Benchmark
120-Second Bounded Sustained	rekuiper 0.426-beta	Long-duration MQTT ingest (12M events)	Sustains 100,000 msg/s for 120s across all five workloads with 0.00% loss.	View Benchmark
Multi-Engine Ingestion Ladder	rekuiper 0.425-beta	30s rate ladder (5k to 100k msg/s)	First published 4-engine comparison against eKuiper 2.4.1, Telegraf, and Redpanda Connect.	View Benchmark

---

5. Summary of Release Benchmarks

rekuiper 0.505-beta: Parallel Rule Concurrency

• Evaluated maximum rule fan-out from a single stream.
• Reached 1,000 concurrent rules (500,000 evaluations/s) on 1 core with flat 251.4 MiB heap memory and 0.0s drain lag.
• Demonstrated 10.0x higher concurrency than eKuiper 2.4.1 (ceiling: 100 rules).

rekuiper 0.500-beta: Peak Ingestion Ceilings

• Introduced zero-disk in-memory catalog and hot-path connection pooling.
• Established exact 1k-resolution peak capacity ceilings: 150,000 msg/s on W1 and W3; 200,000 msg/s on W2 and W4; 126,000 msg/s on W5.
• Reduced single-core CPU utilization at 100k msg/s by over 9 percentage points compared to 0.426.

rekuiper 0.426-beta: Sustained Duration Qualification

• Verified long-duration stability under 120 seconds of continuous traffic at 100,000 msg/s.
• Processed 12 million events per trial without memory leaks or queue backlog.

rekuiper 0.425-beta: Baseline Comparison Ladder

• Initial differential testing across four streaming engines on identical hardware.
• Proved memory-bounded incremental window aggregation for IoT streaming data.