> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ilyama.golain.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Schema and layouts

> Dataspace layouts (jsonb, hybrid, typed, map), field types, promotion, and how schema evolves.

A dataspace has a **schema** independent of its [modality](/dataspaces/overview) and [storage](/dataspaces/storage). Schema is **layout** (how columns sit in the engine) plus a catalog of **fields** (path, type, promotion state).

You may pin `layout` and `fields` at create. If you omit them, the modality default applies and telemetry infers columns from traffic.

→ [Overview](/dataspaces/overview) · [Storage](/dataspaces/storage)

## Layouts (schema shape)

Layout is the physical column shape **inside** the engine. It is not the modality and not the storage class.

| Layout   | Shape                                         | Surprise (undeclared) field            | Cold storage?                  |
| -------- | --------------------------------------------- | -------------------------------------- | ------------------------------ |
| `jsonb`  | One overflow JSON blob                        | Absorbed into overflow; still readable | No — column set still changing |
| `hybrid` | Some typed columns **plus** overflow JSON     | Absorbed into overflow                 | No                             |
| `typed`  | Every field is a real column; **no overflow** | Whole batch → **orphan** table         | Yes                            |
| `map`    | One uniform JSON measurements object          | New key in the same object             | Yes — already uniform          |

<Warning>
  `typed` has no overflow. Use it only when you declare `fields` at create (or have finished promotion) and want undeclared columns rejected. A `typed` dataspace that still receives surprise keys orphans every such batch, with no automatic path to inventing the column.
</Warning>

### Which layouts each modality allows

| Modality                                           | Default  | Also allowed                 | Evolves?                                                                               |
| -------------------------------------------------- | -------- | ---------------------------- | -------------------------------------------------------------------------------------- |
| [Telemetry](/dataspaces/timeseries) (`timeseries`) | `jsonb`  | `hybrid`, `typed`            | Yes — promotion ladder                                                                 |
| [Point](/dataspaces/point)                         | `map`    | `map` only                   | No — starts and stays uniform                                                          |
| [State](/dataspaces/state)                         | `hybrid` | `typed` if you pass `fields` | **No auto-promotion.** You change schema through the control API.                      |
| [Mutable](/dataspaces/mutable)                     | `hybrid` | `typed`                      | Promotion of overflow paths; `jsonb` and `map` are rejected                            |
| [Document](/dataspaces/document)                   | `hybrid` | `typed`                      | No auto-promotion. The platform owns its columns; fields of your own land in overflow. |
| [Vector](/dataspaces/vector)                       | `typed`  | Nothing else                 | No — the schema is fixed by the platform                                               |

`jsonb` is the telemetry “start flexible” layout. `map` is the point “sparse KV” layout. Do not send `map` on telemetry or `jsonb` on mutable.

Document and vector are the two modalities where the platform owns the column set: the upload and search surfaces read those columns by name, so `fields` you send at create are replaced rather than merged. The difference is what happens to anything else. A [document](/dataspaces/document) dataspace is `hybrid`, so fields of your own — sent as `metadata` on a registration — are absorbed into overflow and read back. A [vector](/dataspaces/vector) dataspace is `typed` and has no overflow, because a search filter has to be a real indexed column.

## Promotion ladder (telemetry)

Telemetry climbs as hot paths stabilize:

```
jsonb  →  hybrid  →  typed  →  (then hot_cold is allowed)
```

1. **`jsonb`** — every user field in overflow. Fast to ingest; queries read JSON paths.
2. **`hybrid`** — frequent, type-stable paths become real columns. The rest stays in overflow. The read view coalesces `column` with `overflow.path` so you do not see a seam.
3. **`typed`** — overflow drained, schema frozen. Surprise keys orphan. **This is the gate for `tiering: hot_cold`.**

Point **does not** climb this ladder. State **does not** auto-promote (the inference job samples ClickHouse overflow only). Mutable can promote overflow → typed the same way as telemetry, but stays hot-only.

Trigger promotion with `PATCH /projects/{pid}/dataspaces/{id}/schema`:

```json theme={null}
{ "promote": true, "target_layout": "hybrid" }
```

`storage_class: cheap` cannot promote — there is no hot MergeTree to `ADD COLUMN` and backfill. Freeze the schema before choosing cheap, or stay on `fast`.

## Field catalog

Each user column is a **field**: a `path` (`temperature`, `location.lat`), a **type**, and a **state**.

### Field types

| API `type`           | Meaning                     |
| -------------------- | --------------------------- |
| `float64`, `float32` | Floating point              |
| `int64`, `int32`     | Integers                    |
| `bool`               | Boolean                     |
| `string`             | Text                        |
| `timestamp`          | Event-style timestamp (UTC) |
| `date`               | Calendar date               |
| `uuid`               | UUID                        |
| `json`               | Nested JSON                 |
| `bytes`              | Binary                      |

Declare them in `fields` at create:

```json theme={null}
{
  "name": "line_voltage",
  "modality": "timeseries",
  "fields": [
    { "path": "volts", "type": "float64" },
    { "path": "phase", "type": "string" }
  ]
}
```

Hard cap: **1000 columns** per dataspace. Over-cap rows go to **orphan**, and the caller sees an error.

### Field state

| State        | Meaning                                                                                           |
| ------------ | ------------------------------------------------------------------------------------------------- |
| `overflow`   | Value lives in the overflow JSON (or point map). Readable, not a physical column yet.             |
| `promoting`  | Control plane is adding the typed column and backfilling. The read view still coalesces overflow. |
| `typed`      | Real column. Source of truth for that path.                                                       |
| `deprecated` | Retired from the live schema; dropped from the query surface.                                     |

On a `typed` layout there is no overflow state — a mismatch is an orphan, not a new `overflow` field.

State `hybrid` overflow **is** queryable (the only modality where the overflow blob is a first-class read surface). ClickHouse overflow is exposed through the read view’s JSON/coalesce projection, not as a raw blob you select by default.

## Dataspace schema state

The catalog also tracks a whole-dataspace **`schema_state`**:

| State       | Meaning                                                                                 |
| ----------- | --------------------------------------------------------------------------------------- |
| `inferring` | Sampling traffic; layout still open (`jsonb` / `hybrid`).                               |
| `inferred`  | Inference has a proposal; waiting for accept (or auto-accept).                          |
| `accepted`  | You (or auto-accept) locked the current field set.                                      |
| `evolved`   | Schema changed after accept (promotion, add column, deprecate). `schema_version` ticks. |

`schema_version` is monotonic. Every applied schema change bumps it. Producers and the write plane treat it as the generation of the catalog snapshot they normalized against.

## Auto vs review

| `promotion_policy` | Behavior                                                                                          |
| ------------------ | ------------------------------------------------------------------------------------------------- |
| `auto`             | Stable overflow paths can promote without a human click.                                          |
| `review`           | A **schema proposal** is raised. You approve or reject (`POST …/schema-proposals/{id}/decision`). |

Attaching any **edge sync target** forces `review`. You do not want a mirrored SQLite table silently growing typed columns on the device’s cloud replica.

List proposals: `GET …/dataspaces/{id}/schema-proposals`.

## What happens to bad rows

| Destination                    | Cause                                                  | Lost?                         |
| ------------------------------ | ------------------------------------------------------ | ----------------------------- |
| **Overflow**                   | Undeclared field on `jsonb` / `hybrid` / `map`         | No — kept in JSON             |
| **Orphan table** (per project) | Undeclared field on `typed`, type mismatch, column cap | No — review queue. Not a DLQ. |
| Live table                     | Declared field, matching type                          | Yes, that’s success           |

Orphan is a **schema** problem. Late/future timestamps are a **time-window** problem and go to [dropped-rows](/dataspaces/storage#late-data-and-dropped-rows), not orphan.

## Schema changes at runtime

Applying a schema change (promote, add typed column, freeze layout):

1. Writes pause **briefly** for this dataspace (other dataspaces in the project wait in the durable buffer — no loss).
2. Engine DDL runs (`ALTER` on Postgres or ClickHouse hot).
3. Catalog fields, Arrow schema, and JSON schema regenerate; `schema_version` increments.
4. Writes resume. Readers pick up the new schema on the next cache miss.

You cannot rename modality. You cannot jump `typed` → `jsonb`. Promotion is forward.

Natural-key columns should be declared in `fields` when you set `natural_key`, especially on state/mutable — those modalities do not invent columns from ClickHouse inference.

## Related

* [Dataspaces overview](/dataspaces/overview)
* [Storage](/dataspaces/storage) — `fast` / `cheap`, `hot` / `hot_cold`, and which layouts unlock them
* [Telemetry](/dataspaces/timeseries) · [Point](/dataspaces/point) · [State](/dataspaces/state) · [Mutable](/dataspaces/mutable) · [Document](/dataspaces/document) · [Vector](/dataspaces/vector)
* [API introduction](/api-reference/introduction)