Guided Rule Designer and Visual Schema Builder

This document explains the Guided 3-Step Rule Designer and Visual Schema Builder embedded in the rekuiper web console.

These tools give operators a simple, visual workflow to build SQL streaming pipelines, configure multi-sink delivery targets, test engine validation, and declare strongly-typed stream schemas without writing raw JSON by hand.

Overview

 +-------------------------------------------------------------------------------+
 |                       GUIDED RULE PIPELINE DESIGNER                           |
 +-----------------------------------------------+-------------------------------+
 |               STEP-BY-STEP FORM               |      LIVE PREVIEW & ENGINE    |
 |                                               |                               |
 | [1. Source & SQL Query]                       | [Pipeline JSON Preview]       |
 |   - Rule ID: rule_motor_overheat_alert        |  {                            |
 |   - Visual Builder vs Raw SQL                 |    "id": "...",               |
 |   - Stream/Table Picker                       |    "sql": "SELECT ...",       |
 |   - Projections, WHERE, Windowing             |    "actions": [...],          |
 |   - GROUP BY, HAVING, ORDER BY, LIMIT         |    "options": {...}           |
 |                                               |  }                            |
 | [2. Multi-Sink Outputs]                       |                               |
 |   - Multiple Targets (MQTT, REST, Kafka, ...) | [Mask Secrets Toggle]         |
 |   - DataTemplate & SendSingle                 | [Copy JSON]                   |
 |                                               |                               |
 | [3. Runtime & Resilience]                     | [✔ Validate with Engine]     |
 |   - QoS 0 / 1 / 2, Checkpoints, Concurrency   |  Status: Engine Validated     |
 |   - Event-Time Semantics & Late Tolerance     |                               |
 +-----------------------------------------------+-------------------------------+

Visual Workflow Diagram

Guided 3-Step Rule Designer

The designer modal opens when you click + New Rule Pipeline on the Pipelines and Rules screen or Designer 🪄 on any existing rule row.

Step 1: Source & SQL Query

1. Pipeline ID: Unique identifier for the pipeline.
2. Construction Mode:
   • Visual Builder: Generates standard ANSI SQL from field controls.
   • Raw SQL: Allows direct editing of complex queries with snippet helpers.
3. Source Selection: Dropdown populated from all registered streams and lookup tables.
4. Projections: Comma-separated list of columns or * for all attributes.
5. WHERE Filter: Boolean predicate to filter unneeded records early.
6. Streaming Windowing:
   • None: Record-by-record continuous stream evaluation.
   • Tumbling Window (Time): Non-overlapping fixed time slice (e.g. 10 seconds).
   • Tumbling Window (Count): Fixed batch of records (e.g. 100 rows).
   • Hopping Window: Overlapping sliding window with periodic step.
   • Session Window: Dynamic temporal aggregation bounded by inactivity gaps.
7. GROUP BY, HAVING, ORDER BY, LIMIT: Standard streaming SQL aggregations.

Step 2: Multi-Sink Outputs

Pipelines can forward filtered or transformed data to one or more egress targets:

• MQTT: Broker URL, publish topic, QoS level, client ID.
• REST / HTTP: Target URL, HTTP method (POST/PUT), headers.
• Kafka: Broker seed list, topic.
• File / Parquet: Storage path, file rotation policy.
• Memory: In-memory bus for chaining into other local rules.
• Log / Nop: Debugging and high-speed benchmarking sinks.

Each sink supports formatting controls:

• sendSingle: Emits individual JSON objects instead of array envelopes.
• dataTemplate: Golang text/template format for customized payloads.
• omitIfEmpty: Drops empty transformation payloads.
• runAsync: Uses non-blocking background workers.

Step 3: Runtime & Resilience Options

• QoS Level:
  • 0: At Most Once (lowest latency).
  • 1: At Least Once (guaranteed delivery).
  • 2: Exactly Once (state snapshot recovery).
• Checkpoint Interval: Frequency of distributed state snapshots in milliseconds.
• Operator Concurrency: Worker thread allocation per streaming operator.
• Buffer Length: Ring buffer capacity for inter-operator queues.
• Event-Time Processing: Extracts timestamps from payloads (ts field) and applies lateTolerance windows.
• Resilience Toggles: Attach __meta__ tags or route error envelopes directly down the pipeline.

Live Engine Preflight Validation

Before saving, operators can click Validate with Engine Preflight. This calls POST /rules/validate against the live rekuiper engine to check:

1. SQL syntax and projection semantics.
2. Source stream availability in catalog.
3. Sink configuration validity.
4. Runtime options schema compliance.

---

Visual Stream and Table Schema Builder

The schema builder opens when you click + New Stream or + New Table in the catalog view.

Key Capabilities

1. Topology Kind: Switch between append-only event streams (STREAM) and static or dimension scan tables (TABLE).
2. Schemaless Ingestion: Toggle schemaless mode for dynamic, open JSON payloads without pre-declaring columns.
3. Nested Fields Table:
   • Field name.
   • Strong data types: bigint, float, string, boolean, datetime, bytea, array, struct.
   • Nullable flag.
4. Configuration Keys (CONF_KEY): Select predefined connector credentials and endpoints or click + New ConfKey to create them inline.
5. Live DDL Preview: Generates the exact SQL statement in real-time:

CREATE STREAM plant_telemetry (
    device_id string,
    temperature float,
    ts bigint
) WITH (
    DATASOURCE = "factory/+/sensors",
    FORMAT = "json",
    TYPE = "mqtt",
    CONF_KEY = "plant_cluster"
);

---

API Reference

Endpoint	Method	Description
/rules	POST	Create a new rule pipeline
/rules/:id	PUT	Update an existing rule pipeline
/rules/validate	POST	Preflight validation of SQL and sinks
/streams	POST	Create a stream using DDL SQL
/streams/:name	DELETE	Delete a declared stream
/tables	POST	Create a lookup table using DDL SQL
/tables/:name	DELETE	Delete a declared table
/metadata/sources/:type/confKeys/:name	PUT	Save connector configuration key