ClickHouse Architecture

StreamBuild keeps authored logical identity separate from ClickHouse realization. The current adapter is ClickHouse only.

Managed Kafka landing

For source order_events:

kafka__order_events       Kafka engine table
  |
  +-- mv__order_events    landing materialized view
        |
        +-- raw__order_events   MergeTree landing history

The landing table carries raw kafka_value plus normalized replay columns. The Kafka engine table feeds the landing materialized view, which writes retained history into the raw MergeTree table.

Direct-mode model graph

raw__order_events
  |
  +-- mv__orders
        |
        +-- tbl__orders
              |
              +-- mv__daily_revenue
                    |
                    +-- tbl__daily_revenue

tbl__* names are physical model tables in direct mode. A build drops and recreates selected model tables and MVs while preserving managed source objects. Here the raw source feeds mv__orders, which writes tbl__orders; that table then feeds mv__daily_revenue, which writes tbl__daily_revenue.

Virtual-environment model graph

raw__order_events
  |
  +-- mv__orders__<deployment_id>
        |
        +-- tbl__orders__<deployment_id>

tbl__orders   stable View -> tbl__orders__<published_deployment_id>

In virtual environments, deployment-specific relations hold data and unsuffixed tbl__* names are stable ClickHouse views. Publish replaces those view bindings one relation at a time. The raw source feeds the deployment-specific orders MV and table. The stable tbl__orders view points to the table belonging to the published deployment.

Adopted sources

An adopted stream_table has no Kafka table, raw table, or landing MV:

orders_existing          user-owned source
  |
  +-- mv__orders_enriched
        |
        +-- tbl__orders_enriched

StreamBuild reads and validates the source but does not mutate it. The user-owned orders_existing table feeds mv__orders_enriched, which writes tbl__orders_enriched.

Naming

Pattern	Purpose
`kafka__<source>`	managed Kafka engine table
`raw__<source>`	managed raw landing table
`mv__<model>`	direct model MV or landing MV
`tbl__<model>`	direct table or virtual-environment stable view
`mv__<model>__<deployment_id>`	staged model MV
`tbl__<model>__<deployment_id>`	staged model table

Metadata tables

StreamBuild stores state in the selected target database:

streambuild_object_state_snapshots
streambuild_deployments
streambuild_deployment_watermarks
streambuild_deployment_runtime_details
streambuild_publish_history
streambuild_state_schema_versions
streambuild_target_ownership

These are internal contracts. Do not edit them manually.

Compile boundary

Compilation uses Polyglot as the mandatory SQL parser and analyzer. It produces canonical ClickHouse SQL, exact output types, replay and aggregate facts, references, and a database-qualified adapter template without opening a warehouse connection.

Getting Started

Core Concepts

CLI Reference

Examples

ClickHouse Architecture

Managed Kafka landing

Direct-mode model graph

Virtual-environment model graph

Adopted sources

Naming

Metadata tables

Compile boundary

​Managed Kafka landing

​Direct-mode model graph

​Virtual-environment model graph

​Adopted sources

​Naming

​Metadata tables

​Compile boundary

Managed Kafka landing

Direct-mode model graph

Virtual-environment model graph

Adopted sources

Naming

Metadata tables

Compile boundary