Sources and Pipelines

Sources are project-wide identities under sources/*.yml. Each direct child directory under pipelines/ defines one pipeline.

streambuild_project.toml
sources/
  order_events.yml
pipelines/
  pl__orders/
    orders.sql
    order_items.sql

Only direct sources/*.yml files are discovered. Pipeline, source, and model names share one project-wide namespace and must be unique.

Managed Kafka source

Projects with multiple Kafka sources can set their shared broker once:

[defaults]
kafka_broker_list = "redpanda:9092"

sources:
  - name: order_events
    kind: kafka
    topic: source.order_events.live
    consumer_group: optional_group
    format: JSONAsString
    ttl: _replay_landed_at + INTERVAL 30 DAY
    settings:
      kafka_num_consumers: "1"
    replay_boundary:
      mode: offsets

StreamBuild creates:

kafka__order_events   Kafka engine table
raw__order_events     MergeTree landing table
mv__order_events      landing materialized view

Managed sources support offsets, timestamp, and landed_at. JSONAsString is currently the only supported Kafka format. StreamBuild owns these source objects and blocks incompatible source drift rather than silently recreating them. broker_list may be declared on the source or inherited from [defaults].kafka_broker_list in streambuild_project.toml. A source-level value takes precedence. One of the two is required for every managed Kafka source. ttl is an optional ClickHouse expression applied to the managed raw__order_events landing table. It overrides [defaults].managed_source_ttl from streambuild_project.toml. Use _replay_landed_at for ingestion-age retention, for example _replay_landed_at + INTERVAL 30 DAY. If the source and project default both omit TTL, StreamBuild creates the landing table without a TTL clause. format defaults to JSONAsString. When consumer_group is omitted, the effective ClickHouse group is streambuild_<source>_<source>_<database>. Hyphens in the database suffix are normalized to underscores so each target database receives an isolated consumer group.

Adopted stream table

Adopt an existing table without transferring ownership:

sources:
  - name: orders
    kind: stream_table
    table_name: orders_existing
    replay_boundary:
      mode: offsets
      columns:
        _replay_partition: kafka_partition
        _replay_offset: kafka_offset
        _replay_timestamp: kafka_timestamp

The table must already exist as a bare relation name in the selected target database. StreamBuild validates its replay columns but never creates, drops, or replaces the adopted table. Because StreamBuild does not own adopted tables, ttl is rejected on stream_table sources.

Mode	Required mappings
`offsets`	`_replay_partition`, `_replay_offset`, `_replay_timestamp`
`timestamp`	`_replay_timestamp`
`cursor`	`_replay_cursor`, `_replay_timestamp`

Integer roles must use ClickHouse integer types. Timestamp roles must use DateTime or DateTime64. Adopted sources do not support landed_at.

Pipeline directories

The direct child directory name is the pipeline name. It must start with pl__ by default. Nested directories organize models but do not change pipeline identity. Each model name is its SQL filename stem. StreamBuild follows table-model __source(...) and untyped __ref(...) dependencies to infer the pipeline source. A pipeline with table models must resolve to one registered source. View-only pipelines may be source-less. Empty pipelines, multiple source roots, and duplicate logical names are rejected. The rule is one source per table pipeline, not one pipeline per source. Independent pipelines may consume the same source and share its managed landing tables. Both direct pipelines infer the same source independently and may have a same-mode relationship between their models. Their directory names remain the pipeline identities even if the SQL files are nested more deeply. An edge from either direct pipeline into the virtual reporting pipeline is rejected because relationships cannot cross the mode boundary. Configure the prefix or enforce a project-specific naming rule in streambuild_project.toml:

[naming]
pipeline_prefix = "custom__" # Use "" to allow unprefixed names.
pipeline_naming_macro = "pipeline_name"

from streambuild.compiler.macros.models import PipelineNamingContext

def pipeline_name(ctx: PipelineNamingContext) -> str:
    return ctx.name if ctx.source_name is None else f"pl__{ctx.source_name}"

The optional macro receives an immutable context with the pipeline name, source name, sorted model names, relative directory, and mode. It returns the required directory name. A mismatch stops discovery; StreamBuild does not rename files during compile, plan, or build. An optional pipeline.toml may set virtual-environment replay policy for every model in that pipeline:

mode = "virtual"
bounded_replay_fallback = "full"

[replay_on_change]
breaking = "full"
non_breaking = "bounded-7d"

[naming]
view_prefix = "reporting__"

These policy fields are invalid in direct mode because direct builds always rebuild their selected closure. Naming overrides work in both modes and may partially override project [naming] values. Pipelines that need no pipeline-wide override omit pipeline.toml entirely. mode overrides [defaults].pipeline_mode for that pipeline. Relationships between direct and virtual pipelines are rejected symmetrically during graph validation. Same-mode cross-pipeline relationships remain valid, as do shared sources.

Protected pipelines

Add a protection gate when replacing a pipeline has operational impact:

mode = "direct"

[protection]
warning = "Interrupts the protected order feed while objects are replaced."
confirmation = "DEPLOY_ORDERS"

Both fields are optional. The warning defaults to a generic protected-pipeline message. Confirmation defaults to the pipeline name when it is already shell-safe, or a CONFIRM_-prefixed sanitized name otherwise, so an empty [protection] block is valid. Confirmation values are shell-safe tokens: they must start with a letter or number and may contain only letters, numbers, ., _, :, and -. Protection is available only for direct pipelines. When a build closure contains protected pipelines, StreamBuild requires every exact confirmation value. This is independent of ordinary plan approval: --auto-approve, --json, and --events never bypass protection. Repeat --confirm for multiple protected pipelines. The Plan view shows one confirmation input for each pipeline.

Getting Started

Core Concepts

Development UI

CLI Reference

Examples

Sources and Pipelines

Managed Kafka source

Adopted stream table

Pipeline directories

Protected pipelines

​Managed Kafka source

​Adopted stream table

​Pipeline directories

​Protected pipelines

Managed Kafka source

Adopted stream table

Pipeline directories

Protected pipelines