Skip to main content
A lineage is the cloud’s record of syncing one SQLite table from one device. Staging is where row data waits when the lineage cannot materialize yet.

​
Lineage definition

Examples:
  • Device warehouse-gw-01, table device_state → one lineage
  • Same device, table order_events → a second lineage
Each lineage has independent status, schema versions, reviews, and materialization counters.

​
Two ingest pipelines, one lineage

Every enrolled table maps to a single lineage, but ingest follows one of two MQTT paths: Schema governance, staging, replay, and backpressure are lineage-scoped — both pipelines share the same lineage row and review ticket for a given (device, source_table).

​
Lineage status

Check status in platform-tui → Edge Sync → Lineages or GET /projects/{id}/edge/lineages.

​
Lifecycle

​
First batch

  1. Edge worker creates lineage row (usually active briefly).
  2. If schema is unseen and policy requires review → classification ambiguous.
  3. Lineage flips to paused, review queued, rows go to edge_staged_rows.
This applies to both pipelines:
  • Pipeline A: telemetry batches stage via stageTelemetryRowsTSDB, then governance queues the review.
  • Pipeline B: row batches on a paused lineage stage with staging_reason=schema_review.
The device keeps publishing during review; only volume backpressure sends pause_lineage downlink.

​
After schema approval

  1. Materialization target provisioned (edge_state_* for relational rows, edge_ts_* for telemetry).
  2. Replay intent scheduled — staged rows drain (telemetry-first when both kinds exist).
  3. Lineage active — subsequent live batches materialize directly.

​
Volume backpressure

When staged row count or bytes exceed policy caps (defaults: 100,000 rows / 1 GiB per lineage):
  1. Lineage paused.
  2. Edge worker stages EdgeIngestControlRequestedV2 (pause_lineage); mqtt-broker delivers on sync/ingest/control.
  3. Device stops publishing that table and buffers locally.
  4. When backlog drops below 80% of caps, server sends resume_lineage.
→ Backpressure

​
Materialization circuit breaker

After 5+ permanent materialization errors on an active lineage:
  1. Lineage paused, pause_lineage downlink sent.
  2. Operator fixes root cause (DDL, bindings, TSDB issue).
  3. POST .../lineages/{id}/reset-materialization clears error state and may resume.

​
Staging

​
What is staged?

Full row payloads (insert/update/delete for Pipeline B; timestamped samples for Pipeline A) as JSON, keyed by:
  • device_id, source_table, source_commit_seq or event_ts
  • staging_reason
  • Inline payload or compressed blob (server-internal)

​
Why rows stage

​
Inspecting staged rows

platform-tui:
  1. Edge Sync → Lineages → select lineage
  2. Press d for staged data (or w for written/mirror)
HTTP API:
Paginate with page and limit query params.
Staged rows prove the device is sending data even when mirror tables are empty — always check staged view before debugging MQTT connectivity. After edge v2 cutover, confirm the edge worker is running; batches no longer flow through the integration worker.
Each lineage links to:

​
Deleting a lineage

DELETE .../edge/lineages/{lineage_id} removes cloud sync state for that table on that device. It does not delete the edge SQLite database or Omega triggers. Use when retiring a table sync or resetting a botched enrollment. Requires PROJECT_CAN_MANAGE_DEVICES. platform-tui: lineage detail → D (confirm).

​
Edge-side state (Omega)

Separate from cloud lineage, Omega persists in state.db:
  • Last published commit_seq / batch_seq per table
  • journal_epoch per table
  • Pause watermarks from downlink controls
  • Local backpressure buffer
Cloud lineage status and edge pause state can diverge briefly during schema review (cloud paused, device still sending). → Configuration — state database