File Sink

The file sink writes analysis results to a specified file. It overwrites the file if it already exists. The file source can read output files generated by the file sink.

Properties

Property name	Optional	Description
path	false	The target file path, such as /tmp/result.txt. Supports template syntax for dynamic file names. Refer to dynamic properties.
fileType	true	The file type: lines, json, csv, or parquet. Default: lines. Refer to File Types.
hasHeader	true	Controls whether to generate a header line. Applies only to csv files. Deduces the header from the first record and sorts keys alphabetically.
rollingInterval	true	Minimum time interval in milliseconds before rolling to a new file. rekuiper checks this interval based on checkInterval.
checkInterval	true	Interval in milliseconds for checking time-based rolling policies.
rollingCount	true	Maximum number of messages in a file before rollover.
rollingNamePattern	true	Controls where timestamps are placed in rolled file names: prefix, suffix, or none.
compression	true	Compresses the file payload with the specified method: gzip or zstd.
rollingHook	true	Defines an action executed after a file rolls over.
rollingHookProps	true	Configuration properties required by the rollingHook action.
allowExternalFileAccess	true	Permits writing to paths outside the local workspace root. Default: false. Refer to File Path Security.

Other common sink properties are supported. Refer to sink common properties for more information.

The format property defines the encoding of data inside the file. Specific file types require specific formats. Refer to File Types.

File Path Security

By default, the file sink blocks relative path traversal. The engine rejects paths containing parent directory segments (..):

{"error": "path traversal detected in file sink path: '../logs/out.json'"}

To permit writes to relative external paths that contain .., set allowExternalFileAccess to true:

{
  "file": {
    "path": "../external_logs/out.json",
    "allowExternalFileAccess": true
  }
}

Do not enable allowExternalFileAccess unless your deployment explicitly requires external directory access.

File Types

The file sink supports the following file formats:

• lines: Default type. Writes line-separated records. For example, to write newline-delimited JSON strings, set fileType to lines and format to json.
• json: Writes records as a standard JSON array. To use this format, set format to json.
• csv: Writes comma-delimited CSV records. Custom delimiters are supported. To use this format, set format to delimited.
• parquet: Writes columnar Apache Parquet files using Arrow schema inference and Snappy compression. To use this format, set format to parquet or fileType to parquet.

Rolling Strategy

The file sink provides rolling strategies to control file size and file count:

1. Time-Based Rolling: Controlled by rollingInterval and checkInterval. Set rollingInterval to a positive value and rollingCount to 0. For example, with rollingInterval=86400000 (1 day) and checkInterval=3600000 (1 hour), rekuiper checks open files hourly and rolls files open longer than 24 hours.
2. Message-Count-Based Rolling: Controlled by rollingCount. Set rollingCount to a positive value and rollingInterval to 0. rekuiper rolls the file when its message count exceeds rollingCount.
3. Combined Rolling: Set both rollingInterval and rollingCount to positive values. rekuiper rolls the file when either condition is satisfied.

Sample Usage

The following sample selects temperature values greater than 50 and saves results to /tmp/result.txt:

{
  "sql": "SELECT * from demo where temperature>50",
  "actions": [
    {
      "file": {
        "path": "/tmp/result.txt",
        "checkInterval": 5000,
        "fileType": "lines",
        "format": "json"
      }
    }
  ]
}

The following example writes results to separate CSV files based on the device field. Each file rolls over after 1 hour or when it reaches 10,000 messages. Rolled files use a timestamp prefix:

{
  "sql": "SELECT * from demo where temperature>50",
  "actions": [
    {
      "file": {
        "path": "{{.device}}.csv",
        "fileType": "csv",
        "format": "delimited",
        "hasHeader": true,
        "delimiter": ",",
        "rollingInterval": 3600000,
        "checkInterval": 600000,
        "rollingCount": 10000,
        "rollingNamePattern": "prefix"
      }
    }
  ]
}