State dataspaces

A state dataspace is a keyed table of what is true now: device shadow, settings, e-stop, enrollment, one live document per source (or per source plus a business key). Writes are last-write-wins on a monotonic version. Deletes are tombstones, not physical removal. A stale write is a no-op in storage and a 409 on the HTTP API. The API modality is state. → All modalities

When to use it

Use state when

You have one current document per device (or a small keyset), not thousands of child rows.
Cloud-authored changes must reach the device (downsync).
You need GET after PUT, If-Match / ETag, or a change event that cannot be lost if the process crashes.
Safety-adjacent values (shadow, interlock, “armed”) must be recorded and propagated. The local device loop still owns hard-real-time trip; the platform owns durable record + fan-out.

Use mutable instead when each device holds a table of current rows (work orders, SKUs, child records). Postgres will hold high cardinality; it is the wrong engine for it. Use telemetry instead when the data is history you will not edit. Do not put a compare-and-set job (UPDATE … WHERE status = 'running') in a dataspace. That belongs in ordinary platform tables.

Backing store

State lives in Postgres as an ordinary table in the project dataspace schema. It is never ClickHouse and never tiered to object storage. You cannot set hot_cold or storage_class on state (rejected).

Physical detail	Behavior
Engine	Postgres
Identity	`source_id` only, or `UNIQUE (source_id, natural_key)` when you declare a key
Upsert	Insert on conflict, update only when the stored `version` is lower than the incoming `version`
Stale write	Zero rows updated — skipped, no event
Delete	Version-guarded tombstone (`is_deleted`). The read view hides tombstones. A newer write undeletes.
Transactions	The row change and the outbox event commit together

That shared transaction is why state stays on Postgres. A ClickHouse-backed “state” would put the row in one system and the event in another. For values you will act on, that regression is not acceptable.

Identity

source_id is the producer (device id on the edge path, user/connection on the API path). It is a UUID. A business key — machine_id, (line_id, station_id), SKU — belongs in natural_key, declared at create as an ordered list of field paths. Order is significant; reordering is a different identity.

No natural_key → one live row per source_id (classic shadow).
With natural_key → one live row per (source_id, natural_key). HTTP {row_key} is the natural key, not the UUID.

You can rekey only a dataspace that currently has no key (POST …/dataspaces/{id}/rekey). Changing an existing key can merge rows that are distinct today; that is refused. Create a new dataspace instead.

Meta columns

Column	Meaning
`source_type`	Producer class
`source_id`	Producer UUID
`event_ts`	Event time
`version`	Monotonic last-write-wins token. Exposed as `ETag` on HTTP. Send it as `If-Match` on update/delete.
`updated_at`	Last change time
`natural_key`	Present when declared

State does not expose ingest_ts. Overflow JSON is readable on a hybrid state dataspace (the only modality where the overflow blob is a query surface). version must be stable across retries. Use the producer’s logical version (for example edge commit_seq). Do not default to wall-clock now: a redelivery would look newer and last-write-wins would become last-arrival-wins.

Schema and layouts

Default layout is hybrid: declared columns are typed; undeclared keys land in overflow and stay readable. That default exists because Postgres schema does not auto-grow. Nothing infers new columns from overflow the way ClickHouse telemetry promotion does.

Layout	When to use it
`hybrid` (default)	You do not have a full field list yet. Surprises are kept.
`typed`	You pass `fields` at create and want undeclared columns rejected (HTTP `400` on row CRUD; firehose batches orphan).

Schema changes go through the control API (proposals / review when sync is attached). The write plane will not ALTER state tables on its own. Column cap is 1000. Over-cap or typed mismatch → orphan + error to the caller.

HTTP row CRUD

State is the only modality with synchronous JSON row APIs:

GET    /projects/{pid}/dataspaces/{dsid}/rows
GET    /projects/{pid}/dataspaces/{dsid}/rows/{row_key}
PUT    /projects/{pid}/dataspaces/{dsid}/rows/{row_key}
PATCH  /projects/{pid}/dataspaces/{dsid}/rows/{row_key}
DELETE /projects/{pid}/dataspaces/{dsid}/rows/{row_key}

POST   /projects/{pid}/dataspaces/{dsid}/rows/batch
PATCH  /projects/{pid}/dataspaces/{dsid}/rows          # predicate update
DELETE /projects/{pid}/dataspaces/{dsid}/rows          # predicate tombstone

{row_key} is the declared natural key, or source_id when there is none.

Behavior	Contract
Stale `version` / `If-Match`	`409`, never a silent `200`
Undeclared column on `typed`	`400` (not an orphan row — wrong for a synchronous request)
Delete	Tombstone; a later higher-version PUT restores the row
Reads	Served from the state read view with `can_read_data` in SQL
Writes	Executed by the write plane. The HTTP API does not write Postgres itself.

Predicate PATCH/DELETE use a structured filter (equality, in, ranges on declared columns). Raw SQL WHERE is not accepted. An empty filter is rejected; whole-dataspace updates must set "where": {"all": true}. Each changed row gets its own version bump and its own outbox event. max_rows caps fan-out.

Change events and edge

On a successful commit the write plane emits a row-changed event with the full row (deletes carry identity + version). Delivery is at-least-once. Idempotency key for consumers: dataspace + source + version (and natural key when present). Edge:

Upsync and downsync. A cloud write to a sync-tagged state dataspace fans out to devices.
Authority is per table: device (cloud is a replica) or cloud (cloud is source of truth).
Offline upsync is safe: an older version is a no-op.
Attaching any sync target forces schema promotion policy = review.

Joining state into analytics queries

State lives in Postgres and your history lives in the analytics engine, but an analytics query can read a state dataspace directly, in the same query, without copying anything. That closes the gap that used to force you to duplicate current values into a telemetry dataspace just so a report could see them. You can now write the rollups people actually ask for:

Average reading per machine, labelled by that machine’s current mode rather than by an id.
Yesterday’s throughput per line, restricted to the lines whose current state is running.
A daily report that pairs history with the settings in force when it runs.

Three things to know:


Read-only	Analytics queries read state; they never write it. Every change still goes through row CRUD, the edge path, or the write plane, so versioning, tombstones, and change events cannot be bypassed.
Always the live rows	The query sees the same read view your API reads. Tombstoned rows are excluded, and rows quarantined for schema or timestamp problems are not visible at all.
No copy, no lag	There is no mirror to keep in sync. The values a rollup reads are the values in the table at the moment it runs.

Reference the dataspace by name, as you would anywhere else. Physical table names are not a supported query surface.

On a self-hosted deployment this connection is configured once at install time. If a query that joins state against history returns rows from the history side and nothing from the state side, that connection is the first thing to check.

Create

One row per device (shadow-style):

curl -sS -X POST "$API/projects/$PROJECT_ID/dataspaces" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "device_shadow",
    "modality": "state"
  }'

Business key, declared up front:

curl -sS -X POST "$API/projects/$PROJECT_ID/dataspaces" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "machine_status",
    "modality": "state",
    "natural_key": ["machine_id"],
    "fields": [
      {"path": "machine_id", "type": "string"},
      {"path": "mode", "type": "string"}
    ]
  }'

Dataspaces overview
Schema and layouts
Mutable — many current rows per device, ClickHouse
Document · Vector — the two content modalities built on this same keyed-row shape
Edge data sync
API introduction

Getting started

Dataspaces

Console

Dashboards

State dataspaces

When to use it

Backing store

Identity

Meta columns

Schema and layouts

HTTP row CRUD

Change events and edge

Joining state into analytics queries

Create

​When to use it

​Backing store

​Identity

​Meta columns

​Schema and layouts

​HTTP row CRUD

​Change events and edge

​Joining state into analytics queries

​Create

​Related

When to use it

Backing store

Identity

Meta columns

Schema and layouts

HTTP row CRUD

Change events and edge

Joining state into analytics queries

Create

Related