Skip to content

Memory Allocator ​

This page describes the memory allocator that the rekuiper daemon uses and why.

What changed ​

On Unix systems, the daemon uses the jemalloc allocator in place of the system allocator. This applies to the kuiperd, rekuiperd, rule-engine and ruleengine binaries. On Windows, the daemon continues to use the system allocator.

Why ​

The official Docker image is built for the musl C library (Alpine Linux). The musl allocator uses one coarse lock. When many threads allocate memory at the same time, the threads wait for each other. Adding CPU cores then gives little or no extra throughput.

jemalloc gives each thread its own allocation arenas. Threads do not wait for each other during normal operation.

Single-record requests ​

A request to POST /streams/{name}/data can contain one record or many records.

  • A request with one record uses a lock-free publish path.
  • A request with two or more records keeps the all-or-nothing batch admission gate.

Both paths deliver each record to each subscriber of the stream exactly once. The lock-free path matters for workloads that send one event per request. The batch admission gate lets only one batch request at a time enter a stream.

HTTP ingestion: one record or many recordsPOST /streams/{name}/datacount the recordsOne recordlock-free publish to each subscriberTwo or more recordsall-or-nothing batch admission gateRule queueseach record delivered once

Limits ​

  • You cannot select another allocator at run time. To use a different allocator, build the daemon from source and change the #[global_allocator] in crates/kuiperd/src/lib.rs.
  • Memory-profiling tools that assume the system allocator can report different numbers.

rekuiper is open source (Apache-2.0 / MIT) · For an all-in-one MQTT broker with embedded rekuiper rules, see IndraMQTT.