Skip to main content
Edge SQLite sync moves data from a local SQLite database on a device into Golain-managed storage in the cloud. The edge agent captures changes; the cloud governs schema and materializes approved rows into relational state tables or telemetry hypertables.

​
What gets replicated

The cloud does not store a replica of your SQLite file. It stores governed projections you approve column-by-column.

​
Pipeline stages

​
1. Application writes (device)

Your application reads and writes a normal SQLite database (source.db). Omega’s module does not require application code changes beyond using SQLite.

​
2. Capture (device)

On startup the sqlite-replication module:
  1. Discovers user tables (excludes sqlite_* and internal __omega_* tables).
  2. Installs AFTER INSERT/UPDATE/DELETE triggers that append to __omega_journal (Pipeline B tables).
  3. Computes a schema fingerprint (SHA-256 of canonical column metadata).
For telemetry tables (Pipeline A), the agent maintains a cursor on the highest event_ts included in an accepted batch. For state tables (Pipeline B), each journal entry records commit_seq, operation, primary key JSON, full row JSON (except deletes), and schema_hash.

​
3. Publish (device → MQTT)

Every 5 seconds (or when batch size reaches 512 KiB), the module publishes at QoS 1:
  • Pipeline A: sync/telemetry/batch with batch_id (UUIDv7) and monotonic batch_seq
  • Pipeline B: sync/rows/batch with batch_id, journal_epoch, and row operations
The device never embeds org_id, project_id, or device_id in payloads — the broker derives tenant scope from the mTLS session.

​
4. Broker ingest (cloud)

The MQTT broker:
  • Authenticates the device (mTLS or username/password).
  • Enforces topic ACLs (devices cannot publish on sync/ingest/control).
  • Strips any untrusted tenant fields from JSON.
  • Publishes to edge_ingress_exchange with routing keys for the target pipeline:
Wire details: ilyama edge-client-guide.

​
5. Edge worker intake (cloud)

The edge worker (not the integration worker) owns all ingest and governance:
  1. Dedupes on (device_id, batch_id).
  2. Records segment metadata in the batch ledger.
  3. Upserts a lineage — one row per (device_id, source_table).
Pipeline A batches with an approved schema publish to the telemetry ready stream for the telemetry ingester. Pipeline B batches on an active lineage materialize to edge_state_* directly.

​
6. Schema classification (cloud)

When a batch carries a new schema_hash: Default production policy treats the first schema for a table as ambiguous → schema review required. On schema miss, both pipelines stage first, then advance governance:
  • Pipeline A (telemetry): rows land in edge_staged_rows, then lineage classification and review queue run.
  • Pipeline B (rows/batch): paused lineages stage incoming batches with staging_reason=schema_review; active lineages follow the live materialization path.
The device does not receive pause_lineage for review-only pauses — it keeps publishing while the cloud holds rows.

​
7. Staging (cloud)

While lineage is paused or materialization is blocked, row bodies are stored in edge_staged_rows (Timescale) with a reason: Staging is server-side. The device may continue publishing during schema review; the cloud holds rows until you approve.

​
8. Schema review (operator)

An operator:
  1. Claims the review (queued → in_review).
  2. Inspects staged sample rows.
  3. Approves with column actions (mirror, map, ignore, …) or rejects.
On approve the worker provisions the target DDL, creates column bindings, inserts a replay intent, and sets lineage active.

​
9. Replay and materialization (cloud)

  • Replay drains staged rows into the provisioned telemetry or state target.
  • Live batches on an active lineage write directly to edge_ts_* or edge_state_*.
For the recommended project-coalesced flow, telemetry tables materialize into edge_ts_p{project8}_{source_table}. Golain enables Timescale compression on these hypertables with a 1 day compression policy. When the edge worker needs to pause ingestion, grant a presigned URL, or push cloud-authoritative state, it stages EdgeIngestControlRequestedV2 on the MQTT downlink exchange. The broker delivers the JSON body on sync/ingest/control at QoS 1. → Downlink control

​
11. Query (operator / product)

Query materialized data via:
  • GET .../edge/lineages/{id}/mirror-rows
  • platform-tui Written view
  • QueryScript against the lineage’s edge_ts_* hypertable for telemetry
→ Querying synced data

​
Two kinds of “pause”

Operators often confuse these — they behave differently: Schema approval sets lineage active in the database but does not automatically send resume_lineage unless the device was paused by a downlink control message. → Backpressure

​
Data stores (cloud)