Schema and layouts

A dataspace has a schema independent of its modality and storage. Schema is layout (how columns sit in the engine) plus a catalog of fields (path, type, promotion state). You may pin layout and fields at create. If you omit them, the modality default applies and telemetry infers columns from traffic. → Overview · Storage

Layouts (schema shape)

Layout is the physical column shape inside the engine. It is not the modality and not the storage class.

Layout	Shape	Surprise (undeclared) field	Cold storage?
`jsonb`	One overflow JSON blob	Absorbed into overflow; still readable	No — column set still changing
`hybrid`	Some typed columns plus overflow JSON	Absorbed into overflow	No
`typed`	Every field is a real column; no overflow	Whole batch → orphan table	Yes
`map`	One uniform JSON measurements object	New key in the same object	Yes — already uniform

typed has no overflow. Use it only when you declare fields at create (or have finished promotion) and want undeclared columns rejected. A typed dataspace that still receives surprise keys orphans every such batch, with no automatic path to inventing the column.

Which layouts each modality allows

Modality	Default	Also allowed	Evolves?
Telemetry (`timeseries`)	`jsonb`	`hybrid`, `typed`	Yes — promotion ladder
Point	`map`	`map` only	No — starts and stays uniform
State	`hybrid`	`typed` if you pass `fields`	No auto-promotion. You change schema through the control API.
Mutable	`hybrid`	`typed`	Promotion of overflow paths; `jsonb` and `map` are rejected
Document	`hybrid`	`typed`	No auto-promotion. The platform owns its columns; fields of your own land in overflow.
Vector	`typed`	Nothing else	No — the schema is fixed by the platform

jsonb is the telemetry “start flexible” layout. map is the point “sparse KV” layout. Do not send map on telemetry or jsonb on mutable. Document and vector are the two modalities where the platform owns the column set: the upload and search surfaces read those columns by name, so fields you send at create are replaced rather than merged. The difference is what happens to anything else. A document dataspace is hybrid, so fields of your own — sent as metadata on a registration — are absorbed into overflow and read back. A vector dataspace is typed and has no overflow, because a search filter has to be a real indexed column.

Promotion ladder (telemetry)

Telemetry climbs as hot paths stabilize:

jsonb  →  hybrid  →  typed  →  (then hot_cold is allowed)

jsonb — every user field in overflow. Fast to ingest; queries read JSON paths.
hybrid — frequent, type-stable paths become real columns. The rest stays in overflow. The read view coalesces column with overflow.path so you do not see a seam.
typed — overflow drained, schema frozen. Surprise keys orphan. This is the gate for tiering: hot_cold.

Point does not climb this ladder. State does not auto-promote (the inference job samples ClickHouse overflow only). Mutable can promote overflow → typed the same way as telemetry, but stays hot-only. Trigger promotion with PATCH /projects/{pid}/dataspaces/{id}/schema:

{ "promote": true, "target_layout": "hybrid" }

storage_class: cheap cannot promote — there is no hot MergeTree to ADD COLUMN and backfill. Freeze the schema before choosing cheap, or stay on fast.

Field catalog

Each user column is a field: a path (temperature, location.lat), a type, and a state.

Field types

API `type`	Meaning
`float64`, `float32`	Floating point
`int64`, `int32`	Integers
`bool`	Boolean
`string`	Text
`timestamp`	Event-style timestamp (UTC)
`date`	Calendar date
`uuid`	UUID
`json`	Nested JSON
`bytes`	Binary

Declare them in fields at create:

{
  "name": "line_voltage",
  "modality": "timeseries",
  "fields": [
    { "path": "volts", "type": "float64" },
    { "path": "phase", "type": "string" }
  ]
}

Hard cap: 1000 columns per dataspace. Over-cap rows go to orphan, and the caller sees an error.

Field state

State	Meaning
`overflow`	Value lives in the overflow JSON (or point map). Readable, not a physical column yet.
`promoting`	Control plane is adding the typed column and backfilling. The read view still coalesces overflow.
`typed`	Real column. Source of truth for that path.
`deprecated`	Retired from the live schema; dropped from the query surface.

On a typed layout there is no overflow state — a mismatch is an orphan, not a new overflow field. State hybrid overflow is queryable (the only modality where the overflow blob is a first-class read surface). ClickHouse overflow is exposed through the read view’s JSON/coalesce projection, not as a raw blob you select by default.

Dataspace schema state

The catalog also tracks a whole-dataspace schema_state:

State	Meaning
`inferring`	Sampling traffic; layout still open (`jsonb` / `hybrid`).
`inferred`	Inference has a proposal; waiting for accept (or auto-accept).
`accepted`	You (or auto-accept) locked the current field set.
`evolved`	Schema changed after accept (promotion, add column, deprecate). `schema_version` ticks.

schema_version is monotonic. Every applied schema change bumps it. Producers and the write plane treat it as the generation of the catalog snapshot they normalized against.

Auto vs review

`promotion_policy`	Behavior
`auto`	Stable overflow paths can promote without a human click.
`review`	A schema proposal is raised. You approve or reject (`POST …/schema-proposals/{id}/decision`).

Attaching any edge sync target forces review. You do not want a mirrored SQLite table silently growing typed columns on the device’s cloud replica. List proposals: GET …/dataspaces/{id}/schema-proposals.

What happens to bad rows

Destination	Cause	Lost?
Overflow	Undeclared field on `jsonb` / `hybrid` / `map`	No — kept in JSON
Orphan table (per project)	Undeclared field on `typed`, type mismatch, column cap	No — review queue. Not a DLQ.
Live table	Declared field, matching type	Yes, that’s success

Orphan is a schema problem. Late/future timestamps are a time-window problem and go to dropped-rows, not orphan.

Schema changes at runtime

Applying a schema change (promote, add typed column, freeze layout):

Writes pause briefly for this dataspace (other dataspaces in the project wait in the durable buffer — no loss).
Engine DDL runs (ALTER on Postgres or ClickHouse hot).
Catalog fields, Arrow schema, and JSON schema regenerate; schema_version increments.
Writes resume. Readers pick up the new schema on the next cache miss.

You cannot rename modality. You cannot jump typed → jsonb. Promotion is forward. Natural-key columns should be declared in fields when you set natural_key, especially on state/mutable — those modalities do not invent columns from ClickHouse inference.

Dataspaces overview
Storage — fast / cheap, hot / hot_cold, and which layouts unlock them
Telemetry · Point · State · Mutable · Document · Vector
API introduction

Getting started

Dataspaces

Console

Dashboards

Schema and layouts

Layouts (schema shape)

Which layouts each modality allows

Promotion ladder (telemetry)

Field catalog

Field types

Field state

Dataspace schema state

Auto vs review

What happens to bad rows

Schema changes at runtime

​Layouts (schema shape)

​Which layouts each modality allows

​Promotion ladder (telemetry)

​Field catalog

​Field types

​Field state

​Dataspace schema state

​Auto vs review

​What happens to bad rows

​Schema changes at runtime

​Related

Layouts (schema shape)

Which layouts each modality allows

Promotion ladder (telemetry)

Field catalog

Field types

Field state

Dataspace schema state

Auto vs review

What happens to bad rows

Schema changes at runtime

Related