> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ilyama.golain.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Dataspaces

> Project-scoped tables for telemetry, sparse metrics, current state, mutable relational data, documents, and searchable text — how to choose a modality and how storage works.

A **dataspace** is a project-scoped table for device and application data. Devices, APIs, integrations, and edge sync all write into dataspaces. Dashboards, QueryScript, and the HTTP API read from them.

It is described by **independent axes**, not one enum. You pick **modality** (required). You may also set **layout**, **fields**, **storage class**, and **tiering**. The platform derives the engine.

| Axis                         | API                                 | You set?                           | Values                                                                                |
| ---------------------------- | ----------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------- |
| **Modality** (what it means) | `modality`                          | Required                           | `timeseries` · `point` · `state` · `mutable` · `document` · `vector`                  |
| **Layout** (schema shape)    | `layout`                            | Optional                           | `jsonb` · `hybrid` · `typed` · `map` — see [Schema](/dataspaces/schema)               |
| **Storage class**            | `storage_class`                     | Optional, ClickHouse               | `fast` · `cheap` — see [Storage](/dataspaces/storage)                                 |
| **Tiering**                  | `tiering`                           | Optional, where eligible           | `hot` · `hot_cold`                                                                    |
| **Engine**                   | `engine`                            | No                                 | `clickhouse` or `postgres`, from modality                                             |
| **Natural key**              | `natural_key`                       | State and mutable                  | Extra identity besides `source_id`. Document and vector get theirs from the platform. |
| **Sync targets**             | `sync_targets`                      | State and mutable                  | Edge SQLite replica — not an engine                                                   |
| **Embedding pin**            | `embedding_model`, `embedding_dims` | Vector, required                   | Permanent — see [Vector](/dataspaces/vector)                                          |
| **Chunk destination**        | `chunk_dataspace_id`                | Document, optional, at create only | The vector dataspace that files registered with `vectorize: true` are indexed into    |

You cannot change modality later. Create a new dataspace if the contract is wrong.

People often say **telemetry** for the `timeseries` modality. The API value is `timeseries`.

<Note>
  `document` and `vector` are the **content** modalities: files, and the searchable text pulled out of them. Both carry a platform-owned set of columns you cannot change or remove. A `document` dataspace is `hybrid` on top of that, so you can attach fields of your own to a file through a `metadata` object; a `vector` dataspace is `typed`, because every search filter has to be a real indexed column. Pulling text out of a file is opt-in per file. See [Document](/dataspaces/document) and [Vector](/dataspaces/vector).
</Note>

## Guides

<CardGroup cols={2}>
  <Card title="Telemetry" icon="chart-line" href="/dataspaces/timeseries">
    Append-only named fields. ClickHouse MergeTree. API `timeseries`.
  </Card>

  <Card title="Point" icon="location-dot" href="/dataspaces/point">
    Append-only sparse metric bags. Uniform JSON map. Can cold-store from create.
  </Card>

  <Card title="State" icon="toggle-on" href="/dataspaces/state">
    Current document. Postgres, HTTP row CRUD, guaranteed events.
  </Card>

  <Card title="Mutable" icon="table" href="/dataspaces/mutable">
    Current table. ClickHouse ReplacingMergeTree, tombstones.
  </Card>

  <Card title="Document" icon="file-lines" href="/dataspaces/document">
    One row per file. Presigned upload, content-addressed, list and download. Indexing opt-in.
  </Card>

  <Card title="Vector" icon="magnifying-glass" href="/dataspaces/vector">
    One row per chunk plus its embedding. Hybrid semantic and keyword search.
  </Card>

  <Card title="Schema and layouts" icon="sitemap" href="/dataspaces/schema">
    jsonb / hybrid / typed / map, field types, promotion, proposals.
  </Card>

  <Card title="Storage" icon="hard-drive" href="/dataspaces/storage">
    Engine, fast vs cheap, hot vs hot\_cold, and the valid combination matrix.
  </Card>
</CardGroup>

## Choose a modality

Ask these in order:

1. **Is it a file, or prose you want to search by meaning?** A file → [document](/dataspaces/document). The text pulled out of it, searched semantically → [vector](/dataspaces/vector). Pair them when you want files searchable; a document dataspace on its own is a perfectly good file store.
2. **Will this row ever be updated or deleted?** No → [telemetry](/dataspaces/timeseries) or [point](/dataspaces/point).
3. **Is each sample a fixed (or soon-fixed) set of named fields?** Yes → telemetry. A sparse bag of metric names that varies per device or firmware → point.
4. **Is there one current document per source, and do you need HTTP row CRUD, downsync, or a change event that cannot be lost?** Yes → [state](/dataspaces/state).
5. **Is it a many-row current table keyed by a business primary key?** → [mutable](/dataspaces/mutable).

```mermaid theme={null}
flowchart TD
  Start[What is this table?] --> Content{Files or searchable prose?}
  Content -->|the file itself| DOC[document]
  Content -->|its text, searched by meaning| VEC[vector]
  Content -->|neither| Hist{History of samples?}
  Hist -->|yes| Shape{Named columns or sparse KV?}
  Shape -->|named fields| TS["timeseries (telemetry)"]
  Shape -->|varying metric names| PT[point]
  Hist -->|no — current values| Card{One document per source?}
  Card -->|yes + CRUD / events / downsync| ST[state]
  Card -->|many rows per source| MU[mutable]
```

If you need compare-and-set (`UPDATE … WHERE status = 'running'`), leases, or a job state machine, that is ordinary platform data — not a dataspace.

## Modality × layout × storage

Full rules: [Schema](/dataspaces/schema) and [Storage](/dataspaces/storage). Compact matrix:

| Modality     | Default layout              | Engine                        | Storage class                     | Tiering                                                      |
| ------------ | --------------------------- | ----------------------------- | --------------------------------- | ------------------------------------------------------------ |
| `timeseries` | `jsonb` → promote → `typed` | ClickHouse MergeTree          | `fast`; `cheap` only once `typed` | `hot`; `hot_cold` once `typed` (or requested, then deferred) |
| `point`      | `map` (no promotion)        | ClickHouse MergeTree          | `fast` or `cheap` from create     | `hot` or `hot_cold` from create                              |
| `state`      | `hybrid` (`typed` opt-in)   | Postgres                      | —                                 | `hot` only                                                   |
| `mutable`    | `hybrid` or `typed`         | ClickHouse ReplacingMergeTree | `fast` only                       | `hot` only                                                   |
| `document`   | `hybrid` (`typed` opt-in)   | Postgres                      | —                                 | `hot` only                                                   |
| `vector`     | `typed`, fixed              | Postgres with `pgvector`      | —                                 | `hot` only                                                   |

`cheap` is Parquet-only (no hot table). `hot_cold` is a **fast** hot table plus an aged-out Parquet tier. They are not interchangeable. `hot_cold` + `cheap` is rejected. Document and vector take neither: a document's bytes already live in object storage, and an embedding has no time axis to age along.

## At a glance

|                       | [Telemetry](/dataspaces/timeseries) (`timeseries`) | [Point](/dataspaces/point) | [State](/dataspaces/state)                         | [Mutable](/dataspaces/mutable)                       | [Document](/dataspaces/document)        | [Vector](/dataspaces/vector)                      |
| --------------------- | -------------------------------------------------- | -------------------------- | -------------------------------------------------- | ---------------------------------------------------- | --------------------------------------- | ------------------------------------------------- |
| **Contract**          | Append-only named fields                           | Append-only sparse KV      | Last-value-wins document                           | Last-value-wins table                                | One row per file                        | One row per chunk of text                         |
| **Backing store**     | ClickHouse `MergeTree`                             | ClickHouse `MergeTree`     | Postgres table                                     | ClickHouse `ReplacingMergeTree(version, is_deleted)` | Postgres table; bytes in object storage | Postgres table with `pgvector`                    |
| **Sort / identity**   | `(source_id, event_ts)`                            | `(source_id, event_ts)`    | `source_id` (+ `natural_key` if declared)          | `(source_id, natural_key)` — required                | `doc_key`                               | `(document_id, chunk_ordinal)`                    |
| **Updates / deletes** | No                                                 | No                         | Versioned upsert; delete is a tombstone            | New version row; delete is a tombstone               | Re-registering a key is a new revision  | Re-parsing replaces a document's chunks           |
| **Default layout**    | `jsonb` (promotes toward `typed`)                  | `map` (does not promote)   | `hybrid` (`typed` if you declare columns up front) | `hybrid` (`typed` also allowed)                      | `hybrid` (`typed` also allowed)         | `typed`, fixed by the platform                    |
| **HTTP row CRUD**     | No                                                 | No                         | Yes                                                | No                                                   | Yes                                     | Yes                                               |
| **Per-row events**    | None                                               | None                       | Guaranteed with the write                          | Best-effort                                          | Guaranteed with the write               | Guaranteed with the write                         |
| **Read-your-write**   | Yes (append)                                       | Yes (append)               | Yes                                                | Not guaranteed immediately                           | Yes                                     | Yes for the row; embeddings fill in shortly after |
| **Edge sync**         | Device → cloud only                                | Device → cloud only        | Bidirectional                                      | Upsync; downsync if cloud-authoritative              | No                                      | No                                                |
| **Cold tier**         | After fields freeze                                | From create                | No                                                 | No (hot only)                                        | No                                      | No                                                |

## Layouts (summary)

Layout is schema shape, not storage class. Details and field types: [Schema and layouts](/dataspaces/schema).

| Layout   | Shape                               | Surprise field        | Unlocks `hot_cold` / `cheap`? |
| -------- | ----------------------------------- | --------------------- | ----------------------------- |
| `jsonb`  | Overflow JSON only                  | Absorbed              | No                            |
| `hybrid` | Typed columns + overflow            | Absorbed              | No                            |
| `typed`  | Columns only                        | **Orphan** the batch  | Yes                           |
| `map`    | Uniform JSON measurements (`point`) | New key in the object | Yes                           |

Telemetry climbs `jsonb` → `hybrid` → `typed`. Point stays `map`. State does not auto-promote. Mutable and document allow `hybrid` and `typed` only. Vector is `typed` and fixed. On both content modalities the platform owns the column set — the `fields` you send at create are replaced, not merged — but a `hybrid` document dataspace still absorbs fields of your own into overflow.

<Warning>
  A `typed` dataspace **orphans** batches with undeclared columns. Declare `fields` at create if you start `typed`.
</Warning>

## Identity and meta columns

Every row is associated with a **source** (usually a device). The platform adds meta columns; you do not declare them as user fields.

| Column        | ClickHouse (telemetry, point, mutable)         | Postgres (state, document, vector)             |
| ------------- | ---------------------------------------------- | ---------------------------------------------- |
| `source_type` | Producer class (`device`, `api`, `edge`, …)    | Same                                           |
| `source_id`   | UUID of the producer                           | UUID of the producer                           |
| `event_ts`    | Event time from the payload (UTC, millisecond) | Event time                                     |
| `ingest_ts`   | Server receive time                            | **Not present**                                |
| `version`     | Mutable only — last-write-wins token           | Last-write-wins token                          |
| `natural_key` | Mutable — row identity besides `source_id`     | Present when you declare a key                 |
| `updated_at`  | —                                              | Last change time                               |
| `is_deleted`  | Mutable — tombstone flag                       | Tombstone flag (filtered out of the read view) |

`source_id` is **which producer wrote the row**, not your business key. A SKU, machine id, or OPC-UA node belongs in `natural_key` (or in a user field). Hashing a business key into `source_id` makes the row unaddressable by the thing it describes.

## Writes

All row writes go through **one write plane**. Producers encode a batch (MQTT, edge sync, HTTP, workers). A durable per-project buffer is the no-loss commit point. The write plane then normalizes against the catalog and persists to ClickHouse or Postgres.

* Known fields land in typed columns.
* Surprise fields land in **overflow** when the layout has one.
* Surprise fields on a **typed** layout go to the per-project **orphan** table (kept, not deleted) for review.
* `event_ts` too far in the **future** (more than one day ahead) goes to the per-project **dropped-rows** table.
* `event_ts` older than the hot window is dropped as **late** only when **tiering is `hot_cold`**. Hot-only dataspaces accept arbitrarily late history.

Retries of the same batch are safe: append tables use native block deduplication; state and mutable ignore a `version` that is not newer than what is already stored.

[Document](/dataspaces/document) files are the exception to the shape, not to the plane: you upload bytes straight to object storage through a presigned URL, and the row that describes them is written the same way every other keyed row is. Chunking and embedding a file is opt-in per registration (`vectorize: true`) and produces ordinary rows in the paired [vector](/dataspaces/vector) dataspace.

## Reads

* **QueryScript** and the query gateway use the dataspace **name**. Do not put physical table names in documents — those are rejected.
* **SQL** (`POST /projects/{project_id}/dataspaces/query`) is read-only and runs against the dataspace **read view**. The catalog field `read_view` is informational; copy the name from the catalog if you use raw SQL.
* **HTTP row CRUD** (`…/dataspaces/{id}/rows`) covers the keyed Postgres modalities: **state, document, and vector**. Telemetry and point have no current-row identity. Mutable does not offer GET-after-PUT.
* **Document reads** (`…/dataspaces/{id}/documents`) list a document dataspace, read one file's record, and mint a short-lived download link. **Document only**, and the surface to prefer over raw rows. See [Document](/dataspaces/document#reading-a-document-dataspace).
* **Semantic search** (`POST …/dataspaces/{id}/search`) is **vector only**. See [Vector](/dataspaces/vector#search).
* Authorization is per dataspace (`can_read_data` / `can_write_data`). A caller without read grant does not see the table.

You can **share** a dataspace into another project for reads (`POST …/dataspaces/{id}/shares`).

## Change events

| Modality         | Per-row events                                                                                                                                                                                                                     |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Telemetry, point | **None.** High-volume history is queried, not subscribed as one event per insert.                                                                                                                                                  |
| State            | **Guaranteed.** The row and the change event commit in the same Postgres transaction. A crash cannot persist the row without the event. Delivery is at-least-once; consumers must be idempotent on `(dataspace, source, version)`. |
| Mutable          | **Best-effort.** The ClickHouse row and the Postgres event are not one transaction. Fine for bulk mirrors; not for safety-adjacent state.                                                                                          |
| Document, vector | **Guaranteed**, same as state. Filling in a chunk's embedding is deliberately silent — it is a derived value, and a re-index of a whole corpus would otherwise drown the events that matter.                                       |

Hard-real-time safety (for example an e-stop) must still trip **on the device**. The platform records and propagates the change; it is not a substitute for a local safety loop.

## Edge SQLite sync

Attaching a sync policy provisions an ordinary dataspace — one per device table, shared across devices in the project. Rows are keyed by `source_id` = device id.

| Policy / capture strategy                    | Dataspace modality |
| -------------------------------------------- | ------------------ |
| `timeseries`, `telemetry`, `append`          | `timeseries`       |
| `state`, `key_value`, `kv`                   | `state`            |
| `rows`, `mutable`, `relational`, `row_batch` | `mutable`          |

`point` is a cloud create choice. Edge attach does not auto-map to it.

Only **state** and **mutable** may carry an `edge_sqlite` sync target. Telemetry and point are upsync-only.

Omega's on-device `table_strategies` (`rows` / `telemetry` / `ignore`) is how data **leaves the device**. The table above is how the **cloud replica** is stored. `strategy: rows` in Omega becomes a **mutable** dataspace on attach, not `state`. Use a `state` / `kv` policy strategy when the table is one document per device.

See [Capture strategies](/edge/data-sync/capture-strategies) and [Edge data sync](/edge/data-sync/overview).

## Create a dataspace

```bash theme={null}
curl -sS -X POST "$API/projects/$PROJECT_ID/dataspaces" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{"name":"device_readings","modality":"timeseries"}'
```

`$API` is the [platform base URL](/api-reference/introduction) (production: `https://api.ilyama.golain.io/core/api/v1`).

Optional body fields: `storage_class` (`fast` / `cheap`), `tiering` (`hot` / `hot_cold`), `layout`, `fields`, `natural_key`, `sync_targets`. Which combinations are legal is the [storage matrix](/dataspaces/storage#combination-matrix).

Vector requires `embedding_model`, `embedding_dims`, and one credential field. Document optionally takes `chunk_dataspace_id`, and needs it only if you intend to register files with `vectorize: true`. All of them are fixed at create.

Interactive create, list, query, schema, shares, and row CRUD live under **HTTP API → Dataspaces** in the [API reference](/api-reference/introduction).

## Related

* [Telemetry](/dataspaces/timeseries)
* [Point](/dataspaces/point)
* [State](/dataspaces/state)
* [Mutable](/dataspaces/mutable)
* [Document](/dataspaces/document)
* [Vector](/dataspaces/vector)
* [Schema and layouts](/dataspaces/schema)
* [Storage](/dataspaces/storage)
* [Platform concepts](/getting-started/concepts)
* [API introduction](/api-reference/introduction)