Durable Execution and Recovery Contracts
This document describes the core reliable execution contracts, atomic checkpoint storage, and ordered ingress controls in rekuiper.
Runtime Reachability Status (Gates 1–2)
| Scope | Components | Current Status |
|---|---|---|
Live in kuiperd runtime | rate_limit DAG operator (with bounded FIFO and default backpressure), graph node validation (POST /rules, PUT /rules/{id}, POST /rules/validate), BUFFER_FULL_POLICY (block / dropOldest) + kuiper_stream_buffer_* metrics, schema persistence error propagation (500 on failure), typed-plan REST/CLI reports with secret redaction | Wired into server handlers and execution paths today |
component_done library & qualification modules | ReliableSqliteStore, SqliteOutbox, OutboxDispatcher + DispatchCoordinator, GenerationFence, SavepointBundle + MigrationRegistry, BarrierAligner, window/join/CEP state participants, ReliableSingleSourceRunner, QualifiedPlanRunner, ActionBatchAssembler, execute_with_retry, ReliableInputSender, KafkaPositionCodec, KafkaReliablePartitionReader | Tested in unit, contract, and failpoint integration suites; not yet wired into default kuiperd rule execution (scheduled for Gate 3+ cutover) |
Core Contracts
rekuiper-core defines versioned contracts for durable rules:
- Identities (
contracts/identity.rs): Logical IDs (RuleId,OperatorId,ActionId,ConnectionId,SourceId,PartitionId,PositionCodecId) use 1 to 128 UTF-8 bytes. Display names stay separate from logical ownership.GenerationandCheckpointIduse unsigned 64-bit counters. - Source Positions (
contracts/position.rs):SourcePositionstores opaque cursor bytes with source, partition, generation, and codec metadata. Comparisons fail if partitions, generations, or codecs do not match. - Structured Errors (
contracts/error.rs):StructuredErrormaps each error code to a fixed public message. Deserialization discards incoming wire messages to prevent secret leaks. - Capabilities (
contracts/capability.rs): Components declare supported input modes, result modes, ordering scope, and limits explicitly. Missing declarations mean unsupported. - Clocks and Ordered Controls (
contracts/clock.rs,contracts/control.rs):OrderedEnvelopekeeps data, checkpoint barriers, watermarks, and end-of-input markers in one FIFO queue. - State, Checkpoints, and Effects (
contracts/state.rs,contracts/checkpoint.rs,contracts/effect.rs):CheckpointEnvelopebinds plan, schema, and extension fingerprints to participant snapshots and outbox boundaries. Effect completion is idempotent: one terminal outcome (DeliveredorPermanentFailure) locks the record.
Atomic SQLite Checkpoint and Outbox Store
ReliableSqliteStore (crates/rekuiper-core/src/reliability/sqlite.rs) stores checkpoints, source positions, participant state, and outbox effects in one SQLite transaction:
- The store enables
PRAGMA journal_mode = WAL,PRAGMA synchronous = FULL(orEXTRA), andPRAGMA foreign_keys = ON. - The store reads back each PRAGMA setting and fails closed if SQLite does not confirm it.
- Commits check
reliable_generationsand reject stale generations before writing. SqliteOutbox(crates/rekuiper-core/src/reliability/outbox.rs) enqueues outbox entries in the same SQLite transaction, enforcesmax_itemsandmax_byteslimits, deduplicates repeated effect IDs, and scopesclaim_pendingby(rule_id, action_id).- Legacy
SqliteKvStorebehavior remains unchanged for lightweight rules.
Operator State Snapshots
crates/rekuiper-server/src/engine/state/analytic_count_tumble.rsprovidesAnalyticStateParticipant,CountWindowStateParticipant, andTumbleWindowStateParticipant.- Each participant captures an owned, SHA-256 checksummed snapshot between records without touching the lightweight execution hotpath.
Ordered Reliable Ingress and Delivery Accounting
ReliableInputSender(crates/rekuiper-server/src/engine/reliable_input.rs) enforces ownership generation and cancellable bounded admission without changing the defaultStreamReceiverfast path.PartitionDeliveryTracker(crates/rekuiper-core/src/reliability/delivery.rs) tracks multi-row fanout per source position.- Zero-branch filtered messages complete immediately when at the head of the partition queue.
- Out-of-order completions never skip an incomplete offset, and malformed inputs block progress until a terminal disposition is recorded.
Schema Persistence Error Propagation
Schema endpoints (POST /schemas/:kind, PUT /schemas/:kind/:name, PUT /schemas/:kind/:name/upload, and DELETE /schemas/:kind/:name) now propagate storage errors and return HTTP 500 instead of returning success when KV persistence fails.
Generation Fencing and Outbox Dispatch
- Generation Fencing (
core/reliability/generation.rs):GenerationFenceactivates new rule generations and revokes older generations. Revoked writers cannot commit checkpoints or claim pending outbox entries, while durable unsettled effects remain intact. - Outbox Dispatch (
server/engine/outbox_dispatch.rs):OutboxDispatcherandDispatchCoordinatorenforce exclusive(rule_id, action_id)leases, preserve per-order_keyordering, pass deterministic idempotency keys, and route permanent failures to an explicit DLQ policy. UnsettledRetryableandAmbiguouseffects stay in SQLite so restarts resend them.
Resource Budgets, Batching, and Bounded Retries
- Resource Budgets (
core/resources/budget.rs):ResourceBudgettracks item, logical-byte, observed physical-byte, and task charges with cancellable reservations and idempotent commit release. - Action Batching (
server/engine/batching.rs):ActionBatchAssemblerflushes independently on count, encoded-byte, deadline, barrier, and shutdown boundaries. Partial failures retain only unsettled retryable items, and unset byte limits preserve legacybatchSizeandlingerbehavior. - Bounded Retries (
server/engine/retry.rs):execute_with_retryclassifies adapter results intoDelivered,Retryable,PermanentFailure, andAmbiguous, keeps HTTP2xxbody-drain errors asDelivered, and applies deterministic seeded jitter with boundedRetry-Afterhints.
Direct Kafka Replay and Single-Source SQL Recovery
- Kafka Position Codec (
connectors/src/kafka.rs):KafkaPositionCodecencodesi64next-offset cursors, detects retention gaps before intake, emits metadata and position before payload decoding, and advances only on full multirecord commit boundaries. - Single-Source Runner (
server/engine/runtime.rs):ReliableSingleSourceRunnervalidates supported single-source plans (analytic, count, and tumbling windows), restores state and Kafka offset before intake, flushes sink output before checkpoint commit, and exposesAtLeastOnceguarantee and checkpoint age telemetry.
Typed SQL Planning Descriptors
- Typed Planning (
sql/planning/types.rs,sql/functions.rs):type_check_exprandFunctionDescriptorRegistryvalidate known schema fields, arity, nullability, determinism, side effects, and statefulness. Unknown functions receive conservative descriptors that block unsafe rewrites and durable replay, while legacy schemaless runtime evaluation remains unchanged. - Capability-Aware Plan Validation (
sql/planning/capabilities.rs):validate_plan_capabilitieschecksPlanExecutionProfilerequirements against source replay/offset, sink idempotency, state-snapshot, and ordered-barrier capabilities. Strict reliable or durable profiles fail before execution with structuredPlanDiagnosticKinddiagnostics, while default best-effort plans continue unchanged.
Opt-In Lossy Subscriber Queues and Window Snapshots
- Opt-In Lossy Subscriber FIFO (
core/resources/lossy_queue.rs,core/runtime.rs):lossy_channelprovides a boundeddropOldestring queue for non-reliable subscribers viasubscribe_lossyandsubscribe_lossy_filtered. The default blockingmpscfast path has zero hotpath overhead (has_lossy: AtomicBoolguard), andpush_reliable_controlrejects control barriers and watermarks on lossy queues. - Stream Buffer Policy Wiring (
core/runtime.rs,server/engine/mod.rs,server/handlers/{streams,rules,metrics}.rs):resolve_buffer_full_policynormalizesBUFFER_FULL_POLICY(blockdefault vsdropOldest) across stream creation, rule validation, and subscription; reliable plans rejectdropOldeststreams, andmetricsexposes low-cardinalitykuiper_stream_buffer_depthandkuiper_stream_buffer_dropped_totalcounters. - Hopping and Sliding Window Snapshots (
server/engine/state/hop_slide.rs):HoppingWindowStateParticipantandSlidingWindowStateParticipantcapture overlapping buckets, pending triggers, watermarks, and logical deadlines with SHA-256 checksums and plan-fingerprint validation, applyingMissedDeadlinePolicy(FireImmediatelyInOrderorDropExpired) on recovery. - Session and State Window Snapshots (
server/engine/state/session_state.rs):SessionWindowStateParticipantandStateWindowStateParticipantpersist per-key open sessions, gap and max-duration boundaries,on_beginstate flags, and logical deadlines without serializingInstantor re-evaluating historical nondeterministic predicates. - Join and CEP Snapshots (
server/engine/state/joins_cep.rs):JoinBufferStateParticipantandCepSequenceStateParticipantpersist retained rows, match progress, expiry timestamps, and input identities, deterministically rebuild derived key indexes on restore, and reject plan or key-schema mismatches without migration. - Multi-Input Barrier Alignment (
server/engine/barrier_alignment.rs):BarrierAligneraligns checkpoint barriers across active and idle input edges, bounds paused-edge buffers withResourceBudget, drains or snapshots outstanding async lookups before crossing, cancels stalled lookups on timeout/abort, andvalidate_acyclic_reliable_graphrejects reliable feedback cycles in v1. - Immutable Savepoints (
core/reliability/savepoint.rs):SavepointBundleexports immutable manifests pinning catalog, state, schema, extension, and effect versions plus source-position lineage.prepare_restore_to_new_ruleverifies SHA-256 checksums, extensions, and rule-identity non-collision and runsMigrationRegistryupgrades before creating a restored rule plan. - Watermark State Machine (
server/engine/time/watermark.rs):WatermarkStateMachinetracks per-input maximum event timestamps, unseen/closed input status, and monotonic minimum watermark progress with saturatingi64bounds. - Storage and Recovery Qualification (
server/tests/ticket_p03_t20.rs,server/tests/qualification_harness.rs): In-process failpoint tests (ticket_p03_t20.rs) verify commit I/O failures, corrupt snapshot rejection, generation fencing, Kafka retention loss, outboxAmbiguousrecovery, andcertify_durability_modeseparatingProcessCrashOnlyfrom host power-loss claims, alongside the process-levelkuiperdharness (qualification_harness.rs). - Bounded Rate-Limit Buffers (
server/engine/operators.rs,server/engine/dag.rs):RateLimitBufferenforcesmax_bufferFIFO capacity andResourceBudgetbyte/item limits using a single cancellableClock-driven token wait. Invalid rate, burst, or capacity values fail validation at rule creation, update, validation, and start, and buffer overflow defaults to losslessBackpressure(DropNewest,DropOldest, orFailwhen configured). - Unified Typed Plan and Explain Adapters (
sql/planning/adapter.rs,server/handlers/rules.rs,cli/lib.rs):build_typed_plan_reportpowers REST/rules/validate,/rules/:id/explain, rule creation preflight checks, and CLI manifest validation. Existing response envelopes stay intact while adding stable diagnostic codes, locations, effective guarantees, limiting components, and credential redaction. - Qualified State Participant Contracts (
server/engine/{runtime,windows,dag}.rs): Library-levelQualifiedPlanRunnerandvalidate_participant_coveragerequire registered participants for analytic, tumbling, count, hopping, sliding, session, state, join, CEP, and multi-input barrier alignment plans prior to Gate 3+ runtime cutover. Unqualified DAG shapes andLEAD/LAGremain rejected until their adapters land, and default lightweight execution remains unchanged.