> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ilyama.golain.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Storage

> Engine, storage class (fast vs cheap), and tiering (hot vs hot_cold) — what you set, what the platform derives, and which combinations are valid.

Storage is a separate axis from [modality](/dataspaces/overview) (what the table *means*) and [schema](/dataspaces/schema) (how columns are shaped). Three knobs exist. You only set two of them; the engine is derived.

| Axis              | API field            | Who sets it             | Values                        |
| ----------------- | -------------------- | ----------------------- | ----------------------------- |
| **Engine**        | `engine` (read-only) | Platform, from modality | `clickhouse` or `postgres`    |
| **Storage class** | `storage_class`      | You, ClickHouse only    | `fast` (default) or `cheap`   |
| **Tiering**       | `tiering`            | You, where eligible     | `hot` (default) or `hot_cold` |

A fourth catalog field, **`engine_type`** (`mergetree` vs `replicated`), is a **deployment** flag for ClickHouse clusters. You do not send it on create. Postgres dataspaces store it as null — it has no meaning there.

→ [Overview](/dataspaces/overview) · [Schema](/dataspaces/schema)

## Engine (derived)

| Modality         | Engine                   | Physical table                                                           |
| ---------------- | ------------------------ | ------------------------------------------------------------------------ |
| Telemetry, point | ClickHouse               | `MergeTree` (or `ReplicatedMergeTree`)                                   |
| Mutable          | ClickHouse               | `ReplacingMergeTree(version, is_deleted)`                                |
| State            | Postgres                 | Ordinary table in the project dataspace schema                           |
| Document         | Postgres                 | Same keyed table shape as state; the file's bytes live in object storage |
| Vector           | Postgres with `pgvector` | Same keyed table shape, plus a vector index and a full-text index        |

You never pass `engine` on create. Sending the wrong mental model (“I want state on ClickHouse”) is a **modality** choice ([mutable](/dataspaces/mutable)), not an engine override.

Edge SQLite is a **sync target**, not an engine. A state dataspace can be Postgres *and* edge-replicated.

## Storage class — `fast` vs `cheap`

ClickHouse only. The Postgres modalities — state, document, and vector — have no storage class.

|                         | `fast` (default)                             | `cheap`                                               |
| ----------------------- | -------------------------------------------- | ----------------------------------------------------- |
| Where data lives        | Hot MergeTree                                | Parquet on object storage **only** — **no hot table** |
| Queries                 | Fast, scans MergeTree parts                  | Slower; reads Parquet                                 |
| Cost                    | Higher                                       | Lower                                                 |
| Layout required         | Any layout the modality allows               | **Frozen/uniform only** — `typed` or `map`            |
| Promotion after create  | Yes (add column + backfill overflow)         | **No** — nothing to `ALTER`                           |
| Combine with `hot_cold` | Yes (that *is* a hot table plus a cold tier) | **No** — cheap already has no hot table to age out of |

`cheap` is “this dataspace never needs a hot store.” [Point](/dataspaces/point) can be cheap from create (`map` is already uniform). Telemetry can be cheap only after the schema is `typed`. Mutable, state, document, and vector cannot be cheap.

## Tiering — `hot` vs `hot_cold`

|                     | `hot` (default)                            | `hot_cold`                                                                   |
| ------------------- | ------------------------------------------ | ---------------------------------------------------------------------------- |
| Hot table           | Yes (if `fast`)                            | Yes — MergeTree keeps the recent window                                      |
| Cold table          | No                                         | Yes — `ENGINE = S3` / Parquet, filled by age-out                             |
| Read view           | Hot only                                   | `UNION ALL` of hot and cold                                                  |
| Layout required     | Any                                        | **`typed` or `map`** — cold Parquet cannot absorb a changing overflow schema |
| Late history        | Accepted (only **future** timestamps drop) | Rows older than the hot window go to **dropped-rows** (`late`)               |
| Eligible modalities | All live ones                              | **Telemetry and point only**                                                 |

`hot_cold` is **not** the same as `cheap`:

* `hot_cold` + `fast` = hot MergeTree **and** a cold Parquet tier (age-out).
* `cheap` = Parquet only, no hot MergeTree, no age-out job.

`hot_cold` requires `storage_class: fast`. Requesting `hot_cold` on state, mutable, document, or vector is rejected — mutable cannot collapse versions and tombstones in Parquet, and the other three are Postgres. A [document](/dataspaces/document) has nowhere to age to: its bytes already sit in object storage. A [vector](/dataspaces/vector) has no time axis to age along, and an aged-out embedding is one that searches would silently stop finding.

### Deferred `hot_cold` on telemetry

Telemetry defaults to `jsonb`, which cannot tier. If you create with `"tiering": "hot_cold"` while the layout is still evolving, the platform stores **`tiering: hot`** plus a request flag. Age-out turns on only after every field is `typed` and overflow is drained. Point (`map`) materializes the cold tier immediately.

## Combination matrix

Valid create-time combinations (live modalities):

| Modality     | Layout                                 | `storage_class` | `tiering`           | Result                                                                         |
| ------------ | -------------------------------------- | --------------- | ------------------- | ------------------------------------------------------------------------------ |
| `timeseries` | `jsonb` / `hybrid` (default `jsonb`)   | `fast`          | `hot`               | Normal telemetry. Can promote later.                                           |
| `timeseries` | `jsonb` / `hybrid`                     | `fast`          | `hot_cold`          | Stored as **hot + requested**. Cold tier waits until `typed`.                  |
| `timeseries` | `typed`                                | `fast`          | `hot` or `hot_cold` | Frozen schema. `hot_cold` builds cold immediately.                             |
| `timeseries` | `typed`                                | `cheap`         | `hot`               | Parquet only. Pass `fields`. No later promotion.                               |
| `point`      | `map`                                  | `fast`          | `hot` or `hot_cold` | Sparse KV. Cold allowed from create.                                           |
| `point`      | `map`                                  | `cheap`         | `hot`               | Parquet-only metrics archive.                                                  |
| `state`      | `hybrid` or `typed`                    | *(none)*        | `hot`               | Postgres. `storage_class` / `hot_cold` rejected.                               |
| `mutable`    | `hybrid` or `typed`                    | `fast`          | `hot`               | ClickHouse RMT. `cheap` / `hot_cold` rejected.                                 |
| `document`   | `hybrid` or `typed` (default `hybrid`) | *(none)*        | `hot`               | Postgres. Platform-owned columns; `storage_class` / `hot_cold` rejected.       |
| `vector`     | `typed` (fixed)                        | *(none)*        | `hot`               | Postgres with `pgvector`. Fixed schema; `storage_class` / `hot_cold` rejected. |

Rejected (400), including:

* `hot_cold` + `cheap`
* `hot_cold` on `state`, `mutable`, `document`, or `vector`
* `cheap` on `jsonb` / `hybrid` (not frozen)
* `cheap` on `state`, `mutable`, `document`, or `vector`
* `map` on anything but `point`
* `jsonb` on `mutable`
* `map` or `jsonb` on `document` or `vector`
* `embedding_*` on anything but `vector`, and `chunk_dataspace_id` on anything but `document`
* `sync_targets` on `document` or `vector`

On `document` and `vector`, `fields` and `natural_key` are not rejected — the platform **replaces** them with its own. Sending them changes nothing.

## Physical objects

For a `fast` ClickHouse dataspace the catalog tracks:

| Catalog field | What it is                                                                                  |
| ------------- | ------------------------------------------------------------------------------------------- |
| `hot_table`   | MergeTree (or ReplacingMergeTree) that takes writes                                         |
| `cold_table`  | Set when `tiering = hot_cold` (or when `cheap`, the only table)                             |
| `read_view`   | What QueryScript / SQL should hit — overflow coalesced, hot∪cold unioned, tombstones hidden |

The Postgres modalities — state, document, and vector — each have a table plus a read view in the project dataspace schema. `hot_table` / `cold_table` / `storage_class` / `engine_type` are unused (null).

Query the dataspace **name**, not these physical names. Naming a physical view in QueryScript is rejected.

## Late data and dropped-rows

The write plane applies a time window to `event_ts`:

| Condition                                               | Where the row goes                                                                       |
| ------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| More than **one day in the future**                     | Per-project **dropped-rows**, reason `future`                                            |
| Older than the **hot window**, and `tiering = hot_cold` | Dropped-rows, reason `late`                                                              |
| Older than the hot window, **hot only**                 | **Accepted** into hot (backfill / offline edge)                                          |
| `cheap` (Parquet-direct)                                | Late appends are typically **accepted** as a new Parquet file (no hot window to protect) |

Dropped-rows are **retained**, not deleted. Late ≠ orphan: orphan is schema, dropped is time.

## Retention and partitioning

Telemetry and point partition by month of `event_ts` unless you override `partition_by`. Mutable uses a **single partition** so versions of one key cannot split (ReplacingMergeTree only collapses inside a partition).

Retention (hot window, cold TTL) lives on the catalog `retention` object. Age-out for `hot_cold` is a platform job: it moves parts older than the hot window to the cold table. Cold is **immutable** — you cannot correct a sample that has aged out.

## Related

* [Dataspaces overview](/dataspaces/overview)
* [Schema and layouts](/dataspaces/schema)
* [Telemetry](/dataspaces/timeseries) · [Point](/dataspaces/point) · [State](/dataspaces/state) · [Mutable](/dataspaces/mutable) · [Document](/dataspaces/document) · [Vector](/dataspaces/vector)