Project Configuration

Every project has a committed streambuild_project.toml. Legacy YAML project files are rejected.

name = "analytics"
default_target = "dev"
adapter = "clickhouse"

[connection]
host = "localhost"
port = 8123
username = "default"
password = "${ENV:CLICKHOUSE_PASSWORD}"

[vars]
environment_name = "dev"

[defaults]
pipeline_mode = "direct"
kafka_broker_list = "${ENV:KAFKA_BROKERS}"
managed_source_ttl = "_replay_landed_at + INTERVAL 14 DAY"
model_ttl = "event_at + INTERVAL 90 DAY"
run_presumed_failed_after = "10m"

[naming]
pipeline_prefix = "pl__"
table_prefix = "tbl__"
view_prefix = "view__"

[targets.dev]
database = "analytics_dev"

[targets.dev.connection]
host = "dev-clickhouse"

[targets.dev.vars]
environment_name = "target-dev"

name and default_target are required. adapter defaults to clickhouse. Classify a production target in committed configuration even when its name is not prod or production:

[targets.live]
database = "analytics_live"
production = true

This immutable classification adds the exact PRODUCTION challenge to target reset. It is not accepted in streambuild.local.toml, so a local override cannot weaken or add production status. Commands that connect to ClickHouse require host, port, username, and password after all configuration layers are merged. StreamBuild supplies no connection defaults. Use password = "" when the ClickHouse account intentionally has an empty password.

Connection settings

settings sends ClickHouse session settings on every statement StreamBuild runs, including replay:

[targets.prod.connection.settings]
max_threads = "8"
max_memory_usage = "16000000000"

Values may be strings, numbers, or booleans, and are sent as strings (true becomes "1"). These override the server’s defaults for StreamBuild’s own connection only; nothing else querying the server is affected. Use this when replay needs a different execution profile from the rest of the warehouse — for example capping max_threads so a large replay does not exhaust max_memory_usage, since read memory scales with thread count. A target’s settings table replaces the project-level one rather than merging key by key. ClickHouse rejects unknown setting names, so a typo fails on connect.

Kafka and retention defaults

Set a broker list once for every managed Kafka source in the project:

[defaults]
kafka_broker_list = "${ENV:KAFKA_BROKERS}"

A source-level broker_list overrides this value. A managed Kafka source must resolve a broker list from one of those levels. The default supports normal project and environment interpolation. Set an optional project-wide TTL for StreamBuild-managed Kafka landing tables:

[defaults]
managed_source_ttl = "_replay_landed_at + INTERVAL 14 DAY"

The value is a ClickHouse TTL expression. _replay_landed_at is the recommended clock because it tracks when StreamBuild persisted the Kafka record, independently of the event timestamp. A managed source can override this default with its own ttl. When neither level defines a value, StreamBuild does not add a TTL clause and retains landing data indefinitely. Set an independent default for table models:

[defaults]
model_ttl = "event_at + INTERVAL 90 DAY"

An explicit ttl in MODEL() overrides model_ttl. When neither level defines a model TTL, StreamBuild creates the table without a TTL clause. The effective TTL is validated against output columns and participates in storage identity and drift detection. Terminal views do not inherit a model TTL.

Pipeline naming

Pipeline names must start with pl__ by default:

[naming]
pipeline_prefix = "custom__" # Use "" to allow unprefixed names.
pipeline_naming_macro = "pipeline_name"

pipeline_naming_macro is optional. It names a Python macro that accepts a PipelineNamingContext and returns the required pipeline directory name. A mismatch stops discovery; StreamBuild never renames project files. Pipeline, source, and model names must be unique across the project.

Model relation naming

Set project-wide prefixes for table and terminal-view relations:

[naming]
table_prefix = "tbl__"
view_prefix = "view__"

These are the built-in defaults. A pipeline can partially override them in its optional pipeline.toml, and a model can set an exact relation_name in MODEL(). Resolution order is exact model name, pipeline prefix, project prefix, then built-in prefix. Logical model identity remains the SQL filename stem, so __ref(), selectors, tests, and audits do not change when warehouse naming changes. kafka__, raw__, and mv__ are reserved for StreamBuild internals. Deployment suffixes are also internal and cannot be configured. Effective relation names must be unique unqualified identifiers.

Pipeline build modes

Direct mode is the project default. Set it explicitly with:

[defaults]
pipeline_mode = "direct"

Use virtual mode as the project default with:

[defaults]
pipeline_mode = "virtual"

An individual pipeline.toml can override the project default:

mode = "virtual"

Direct pipelines may reference direct pipelines, and virtual pipelines may reference virtual pipelines. Model relationships cannot cross the mode boundary in either direction. Multiple pipelines may still consume the same project source. stb plan and stb build accept mixed selections. StreamBuild stages the virtual phase first and starts the immediately-applied direct phase only after virtual staging succeeds.

Build limits

Set an absolute maximum for the distinct pipelines in a build’s expanded downstream scope:

[build]
max_pipelines = 20

[targets.prod.build]
max_pipelines = 5

A target override requires the project default. Builds above the effective limit stop before any warehouse mutation. The limit cannot be set in streambuild_local.toml; local-only targets inherit the project default.

Sensor automation

Enable durable event and polling sensors per target:

[targets.prod.sensors]
enabled = true
tick_retention_days = 30

Sensors are disabled by default. Zero retention keeps all tick history. See Sensors.

Deployment readiness

Virtual deployment audits use configurable project-wide thresholds:

[defaults.deployment_readiness]
maximum_lag = "30s"
minimum_staged_row_ratio = 0.5

maximum_lag accepts an <int><s|m|h|d> duration and defaults to 30s. minimum_staged_row_ratio accepts a number from 0 to 1 and defaults to 0.5. These thresholds contribute to the overall ready, caution, or not_ready assessment reported by stb deployment audit; replay evidence and applicable SQL audits also contribute. The assessment remains advisory and does not block promotion.

Run recovery window

stb dev blocks a new build while a durable run is still running or unresponsive. Configure how long a silent run remains blocking:

[defaults]
run_presumed_failed_after = "10m"

The value accepts an <int><s|m|h|d> duration, defaults to 10m, and must exceed the 45-second unresponsive threshold. Once the window expires, the run is displayed as presumed_failed and a fresh build may start. This setting is committed project policy because overlapping writes affect everyone using the target database.

Local overrides

Use ignored streambuild_local.toml for developer-specific values:

target = "personal"
adapter = "clickhouse"

[defaults]
pipeline_mode = "virtual"

[connection]
host = "localhost"
port = 18123

[targets.personal]
database = "analytics_klonge"

Local configuration can override the target, adapter, default pipeline mode, connection, variables, and target values. It cannot replace the project name or authored resources. Other committed project defaults are not locally overridable.

Precedence

Target

CLI --target
local target
project default_target

Variables, lowest to highest

project [vars]
selected project target vars
selected local target vars
local [vars]
CLI --vars

stb plan --vars '{"environment_name":"ci"}'

Database

CLI --database
selected local target database
selected project target database

Connection, lowest to highest

project connection
selected project target connection
selected local target connection
local connection
STREAMBUILD_CLICKHOUSE_HOST, STREAMBUILD_CLICKHOUSE_PORT, STREAMBUILD_CLICKHOUSE_USERNAME, and STREAMBUILD_CLICKHOUSE_PASSWORD
CLI connection flags

There is no public metadata-database override. Metadata lives in the selected target database.

Interpolation

Use project variables and environment variables in values:

[vars]
database_suffix = "dev"
credentials = {username = "reader", role = "analytics"}

[targets.dev]
database = "analytics_${database_suffix}"

[connection]
password = "${ENV:CLICKHOUSE_PASSWORD}"

Interpolation is recursive through arrays and mappings. A whole token preserves scalar or structured value types. Objects and arrays cannot be embedded inside surrounding text; consume structured variables through a Python macro instead. Connection secrets are expanded only when a command actually connects. stb discover and stb compile remain connection-free.

Getting Started

Core Concepts

Development UI

CLI Reference

Examples

Project Configuration

Connection settings

Kafka and retention defaults

Pipeline naming

Model relation naming

Pipeline build modes

Build limits

Sensor automation

Deployment readiness

Run recovery window

Local overrides

Precedence

Target

Variables, lowest to highest

Database

Connection, lowest to highest

Interpolation

​Connection settings

​Kafka and retention defaults

​Pipeline naming

​Model relation naming

​Pipeline build modes

​Build limits

​Sensor automation

​Deployment readiness

​Run recovery window

​Local overrides

​Precedence

​Target

​Variables, lowest to highest

​Database

​Connection, lowest to highest

​Interpolation

Connection settings

Kafka and retention defaults

Pipeline naming

Model relation naming

Pipeline build modes

Build limits

Sensor automation

Deployment readiness

Run recovery window

Local overrides

Precedence

Target

Variables, lowest to highest

Database

Connection, lowest to highest

Interpolation