Memory Allocator
This page describes the memory allocator that the rekuiper daemon uses and why.
What changed
On Unix systems, the daemon uses the jemalloc allocator in place of the system allocator. This applies to the kuiperd, rekuiperd, rule-engine and ruleengine binaries. On Windows, the daemon continues to use the system allocator.
Why
The official Docker image is built for the musl C library (Alpine Linux). The musl allocator uses one coarse lock. When many threads allocate memory at the same time, the threads wait for each other. Adding CPU cores then gives little or no extra throughput.
jemalloc gives each thread its own allocation arenas. Threads do not wait for each other during normal operation.
Single-record requests
A request to POST /streams/{name}/data can contain one record or many records.
- A request with one record uses a lock-free publish path.
- A request with two or more records keeps the all-or-nothing batch admission gate.
Both paths deliver each record to each subscriber of the stream exactly once. The lock-free path matters for workloads that send one event per request. The batch admission gate lets only one batch request at a time enter a stream.
Limits
- You cannot select another allocator at run time. To use a different allocator, build the daemon from source and change the
#[global_allocator]incrates/kuiperd/src/lib.rs. - Memory-profiling tools that assume the system allocator can report different numbers.