Dataspaces

A dataspace is a project-scoped table for device and application data. Devices, APIs, integrations, and edge sync all write into dataspaces. Dashboards, QueryScript, and the HTTP API read from them. It is described by independent axes, not one enum. You pick modality (required). You may also set layout, fields, storage class, and tiering. The platform derives the engine.

Axis	API	You set?	Values
Modality (what it means)	`modality`	Required	`timeseries` · `point` · `state` · `mutable` · `document` · `vector`
Layout (schema shape)	`layout`	Optional	`jsonb` · `hybrid` · `typed` · `map` — see Schema
Storage class	`storage_class`	Optional, ClickHouse	`fast` · `cheap` — see Storage
Tiering	`tiering`	Optional, where eligible	`hot` · `hot_cold`
Engine	`engine`	No	`clickhouse` or `postgres`, from modality
Natural key	`natural_key`	State and mutable	Extra identity besides `source_id`. Document and vector get theirs from the platform.
Sync targets	`sync_targets`	State and mutable	Edge SQLite replica — not an engine
Embedding pin	`embedding_model`, `embedding_dims`	Vector, required	Permanent — see Vector
Chunk destination	`chunk_dataspace_id`	Document, optional, at create only	The vector dataspace that files registered with `vectorize: true` are indexed into

You cannot change modality later. Create a new dataspace if the contract is wrong. People often say telemetry for the timeseries modality. The API value is timeseries.

document and vector are the content modalities: files, and the searchable text pulled out of them. Both carry a platform-owned set of columns you cannot change or remove. A document dataspace is hybrid on top of that, so you can attach fields of your own to a file through a metadata object; a vector dataspace is typed, because every search filter has to be a real indexed column. Pulling text out of a file is opt-in per file. See Document and Vector.

Guides

Telemetry

Append-only named fields. ClickHouse MergeTree. API timeseries.

Point

Append-only sparse metric bags. Uniform JSON map. Can cold-store from create.

State

Current document. Postgres, HTTP row CRUD, guaranteed events.

Mutable

Current table. ClickHouse ReplacingMergeTree, tombstones.

Document

One row per file. Presigned upload, content-addressed, list and download. Indexing opt-in.

Vector

One row per chunk plus its embedding. Hybrid semantic and keyword search.

Schema and layouts

jsonb / hybrid / typed / map, field types, promotion, proposals.

Storage

Engine, fast vs cheap, hot vs hot_cold, and the valid combination matrix.

Choose a modality

Ask these in order:

Is it a file, or prose you want to search by meaning? A file → document. The text pulled out of it, searched semantically → vector. Pair them when you want files searchable; a document dataspace on its own is a perfectly good file store.
Will this row ever be updated or deleted? No → telemetry or point.
Is each sample a fixed (or soon-fixed) set of named fields? Yes → telemetry. A sparse bag of metric names that varies per device or firmware → point.
Is there one current document per source, and do you need HTTP row CRUD, downsync, or a change event that cannot be lost? Yes → state.
Is it a many-row current table keyed by a business primary key? → mutable.

If you need compare-and-set (UPDATE … WHERE status = 'running'), leases, or a job state machine, that is ordinary platform data — not a dataspace.

Modality × layout × storage

Full rules: Schema and Storage. Compact matrix:

Modality	Default layout	Engine	Storage class	Tiering
`timeseries`	`jsonb` → promote → `typed`	ClickHouse MergeTree	`fast`; `cheap` only once `typed`	`hot`; `hot_cold` once `typed` (or requested, then deferred)
`point`	`map` (no promotion)	ClickHouse MergeTree	`fast` or `cheap` from create	`hot` or `hot_cold` from create
`state`	`hybrid` (`typed` opt-in)	Postgres	—	`hot` only
`mutable`	`hybrid` or `typed`	ClickHouse ReplacingMergeTree	`fast` only	`hot` only
`document`	`hybrid` (`typed` opt-in)	Postgres	—	`hot` only
`vector`	`typed`, fixed	Postgres with `pgvector`	—	`hot` only

cheap is Parquet-only (no hot table). hot_cold is a fast hot table plus an aged-out Parquet tier. They are not interchangeable. hot_cold + cheap is rejected. Document and vector take neither: a document’s bytes already live in object storage, and an embedding has no time axis to age along.

At a glance

	Telemetry (`timeseries`)	Point	State	Mutable	Document	Vector
Contract	Append-only named fields	Append-only sparse KV	Last-value-wins document	Last-value-wins table	One row per file	One row per chunk of text
Backing store	ClickHouse `MergeTree`	ClickHouse `MergeTree`	Postgres table	ClickHouse `ReplacingMergeTree(version, is_deleted)`	Postgres table; bytes in object storage	Postgres table with `pgvector`
Sort / identity	`(source_id, event_ts)`	`(source_id, event_ts)`	`source_id` (+ `natural_key` if declared)	`(source_id, natural_key)` — required	`doc_key`	`(document_id, chunk_ordinal)`
Updates / deletes	No	No	Versioned upsert; delete is a tombstone	New version row; delete is a tombstone	Re-registering a key is a new revision	Re-parsing replaces a document’s chunks
Default layout	`jsonb` (promotes toward `typed`)	`map` (does not promote)	`hybrid` (`typed` if you declare columns up front)	`hybrid` (`typed` also allowed)	`hybrid` (`typed` also allowed)	`typed`, fixed by the platform
HTTP row CRUD	No	No	Yes	No	Yes	Yes
Per-row events	None	None	Guaranteed with the write	Best-effort	Guaranteed with the write	Guaranteed with the write
Read-your-write	Yes (append)	Yes (append)	Yes	Not guaranteed immediately	Yes	Yes for the row; embeddings fill in shortly after
Edge sync	Device → cloud only	Device → cloud only	Bidirectional	Upsync; downsync if cloud-authoritative	No	No
Cold tier	After fields freeze	From create	No	No (hot only)	No	No

Layouts (summary)

Layout is schema shape, not storage class. Details and field types: Schema and layouts.

Layout	Shape	Surprise field	Unlocks `hot_cold` / `cheap`?
`jsonb`	Overflow JSON only	Absorbed	No
`hybrid`	Typed columns + overflow	Absorbed	No
`typed`	Columns only	Orphan the batch	Yes
`map`	Uniform JSON measurements (`point`)	New key in the object	Yes

Telemetry climbs jsonb → hybrid → typed. Point stays map. State does not auto-promote. Mutable and document allow hybrid and typed only. Vector is typed and fixed. On both content modalities the platform owns the column set — the fields you send at create are replaced, not merged — but a hybrid document dataspace still absorbs fields of your own into overflow.

A typed dataspace orphans batches with undeclared columns. Declare fields at create if you start typed.

Identity and meta columns

Every row is associated with a source (usually a device). The platform adds meta columns; you do not declare them as user fields.

Column	ClickHouse (telemetry, point, mutable)	Postgres (state, document, vector)
`source_type`	Producer class (`device`, `api`, `edge`, …)	Same
`source_id`	UUID of the producer	UUID of the producer
`event_ts`	Event time from the payload (UTC, millisecond)	Event time
`ingest_ts`	Server receive time	Not present
`version`	Mutable only — last-write-wins token	Last-write-wins token
`natural_key`	Mutable — row identity besides `source_id`	Present when you declare a key
`updated_at`	—	Last change time
`is_deleted`	Mutable — tombstone flag	Tombstone flag (filtered out of the read view)

source_id is which producer wrote the row, not your business key. A SKU, machine id, or OPC-UA node belongs in natural_key (or in a user field). Hashing a business key into source_id makes the row unaddressable by the thing it describes.

Writes

All row writes go through one write plane. Producers encode a batch (MQTT, edge sync, HTTP, workers). A durable per-project buffer is the no-loss commit point. The write plane then normalizes against the catalog and persists to ClickHouse or Postgres.

Known fields land in typed columns.
Surprise fields land in overflow when the layout has one.
Surprise fields on a typed layout go to the per-project orphan table (kept, not deleted) for review.
event_ts too far in the future (more than one day ahead) goes to the per-project dropped-rows table.
event_ts older than the hot window is dropped as late only when tiering is hot_cold. Hot-only dataspaces accept arbitrarily late history.

Retries of the same batch are safe: append tables use native block deduplication; state and mutable ignore a version that is not newer than what is already stored. Document files are the exception to the shape, not to the plane: you upload bytes straight to object storage through a presigned URL, and the row that describes them is written the same way every other keyed row is. Chunking and embedding a file is opt-in per registration (vectorize: true) and produces ordinary rows in the paired vector dataspace.

Reads

QueryScript and the query gateway use the dataspace name. Do not put physical table names in documents — those are rejected.
SQL (POST /projects/{project_id}/dataspaces/query) is read-only and runs against the dataspace read view. The catalog field read_view is informational; copy the name from the catalog if you use raw SQL.
HTTP row CRUD (…/dataspaces/{id}/rows) covers the keyed Postgres modalities: state, document, and vector. Telemetry and point have no current-row identity. Mutable does not offer GET-after-PUT.
Document reads (…/dataspaces/{id}/documents) list a document dataspace, read one file’s record, and mint a short-lived download link. Document only, and the surface to prefer over raw rows. See Document.
Semantic search (POST …/dataspaces/{id}/search) is vector only. See Vector.
Authorization is per dataspace (can_read_data / can_write_data). A caller without read grant does not see the table.

You can share a dataspace into another project for reads (POST …/dataspaces/{id}/shares).

Change events

Modality	Per-row events
Telemetry, point	None. High-volume history is queried, not subscribed as one event per insert.
State	Guaranteed. The row and the change event commit in the same Postgres transaction. A crash cannot persist the row without the event. Delivery is at-least-once; consumers must be idempotent on `(dataspace, source, version)`.
Mutable	Best-effort. The ClickHouse row and the Postgres event are not one transaction. Fine for bulk mirrors; not for safety-adjacent state.
Document, vector	Guaranteed, same as state. Filling in a chunk’s embedding is deliberately silent — it is a derived value, and a re-index of a whole corpus would otherwise drown the events that matter.

Hard-real-time safety (for example an e-stop) must still trip on the device. The platform records and propagates the change; it is not a substitute for a local safety loop.

Edge SQLite sync

Attaching a sync policy provisions an ordinary dataspace — one per device table, shared across devices in the project. Rows are keyed by source_id = device id.

Policy / capture strategy	Dataspace modality
`timeseries`, `telemetry`, `append`	`timeseries`
`state`, `key_value`, `kv`	`state`
`rows`, `mutable`, `relational`, `row_batch`	`mutable`

point is a cloud create choice. Edge attach does not auto-map to it. Only state and mutable may carry an edge_sqlite sync target. Telemetry and point are upsync-only. Omega’s on-device table_strategies (rows / telemetry / ignore) is how data leaves the device. The table above is how the cloud replica is stored. strategy: rows in Omega becomes a mutable dataspace on attach, not state. Use a state / kv policy strategy when the table is one document per device. See Capture strategies and Edge data sync.

Create a dataspace

curl -sS -X POST "$API/projects/$PROJECT_ID/dataspaces" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{"name":"device_readings","modality":"timeseries"}'

$API is the platform base URL (production: https://api.ilyama.golain.io/core/api/v1). Optional body fields: storage_class (fast / cheap), tiering (hot / hot_cold), layout, fields, natural_key, sync_targets. Which combinations are legal is the storage matrix. Vector requires embedding_model, embedding_dims, and one credential field. Document optionally takes chunk_dataspace_id, and needs it only if you intend to register files with vectorize: true. All of them are fixed at create. Interactive create, list, query, schema, shares, and row CRUD live under HTTP API → Dataspaces in the API reference.

Getting started

Console

Dashboards

Dataspaces

Guides

Telemetry

Point

State

Mutable

Document

Vector

Schema and layouts

Storage

Choose a modality

Modality × layout × storage

At a glance

Layouts (summary)

Identity and meta columns

Writes

Reads

Change events

Edge SQLite sync

Create a dataspace

​Guides

Telemetry

Point

State

Mutable

Document

Vector

Schema and layouts

Storage

​Choose a modality

​Modality × layout × storage

​At a glance

​Layouts (summary)

​Identity and meta columns

​Writes

​Reads

​Change events

​Edge SQLite sync

​Create a dataspace

​Related

Guides

Choose a modality

Modality × layout × storage

At a glance

Layouts (summary)

Identity and meta columns

Writes

Reads

Change events

Edge SQLite sync

Create a dataspace

Related