Device RPC troubleshooting

Work top-down when a call fails: device online → schema bound → permissions → correct URL → invoke. Set these for the copyable commands:

export API=https://api.ilyama.golain.io/core/api/v1
export RPC_PROXY=https://rpc.ilyama.golain.io
export TOKEN=…
export ORG_ID=…
export PROJECT_ID=…
export DEVICE_ID=…

Quick diagnostic flow

Symptom index

Symptom	Likely cause	Section
Empty `services` from list services	Schema not bound	Schema not bound
`device descriptor not found` on invoke	Schema not bound	Schema not bound
`forbidden` (403)	Missing permission	Permissions
`unauthorized` (401)	Invalid or missing token	Permissions
502 / timeout / can’t reach device	Device offline or wrong root topic	Device offline
`service not found in descriptor`	Wrong service name in URL	Wrong URL
`method not found in descriptor`	Wrong method name in URL	Wrong URL
`streaming rpc not supported` (501)	Server-stream or client/bidi on sync path	Streaming on wrong path
`only server-streaming rpc supported` (501)	Unary or client/bidi on `/streams/` path	Streaming on wrong path
Invoke works locally, fails from cloud	Root topic mismatch	Root topic mismatch
Outdated method list after OTA	Stale schema binding	Stale schema after OTA
Queued call stuck or expired	Device offline past TTL	Queued stuck or duplicates
Duplicate side effects from queued RPC	Handler not idempotent	Queued idempotency
Schema never appears after device connects	Self-hosted stack misconfiguration	Self-hosted operators

Schema not bound

Symptoms:

GET …/rpc/services returns "services": [] with no descriptor_digest
Invoke returns 404 with "device descriptor not found"

Check:

device-rpc module enabled — in modules.required and security.capabilities. See Omega setup.
Device online — Console shows the device connected over MQTT.
Device restarted after enabling RPC — the module publishes a schema announce on start and reconnect.

Fix:

# List services (should show descriptor_digest + services when bound)
curl -sS \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  "$API/projects/$PROJECT_ID/devices/$DEVICE_ID/rpc/services"

# Force reflection (device must be online; requires can_write_device_metadata)
curl -sS -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{}' \
  "$API/projects/$PROJECT_ID/devices/$DEVICE_ID/rpc/reflect"

Re-run list services. You should see a non-empty services array before invoking.

Device offline / can’t reach device

Symptoms:

Schema is bound but invoke returns 502, 503, or times out
Reflect succeeds but invoke never completes
Error text mentions the device cannot be reached

Check:

Device online in Console — MQTT heartbeat recent.
device-rpc running — Omega logs show the gRPC listener started (not memory transport only).
MQTT credentials and TLS — cert not expired; broker reachable from the device network.

Fix:

Restart the device with MQTT transport and device-rpc enabled.
Reconnect MQTT (restart Omega or power-cycle) so the platform sees the device again.
Probe with a unary Echo call:

curl -sS -X POST \
  "$RPC_PROXY/projects/$PROJECT_ID/devices/$DEVICE_ID/golain.device.v1.EchoService/Echo" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -H "Connect-Protocol-Version: 1" \
  -d '{"message":"diagnostic ping"}'

If list services works but invoke still fails, check Root topic mismatch.

Root topic mismatch

Symptoms:

Device shows online; schema bound; invoke still fails with reachability or bridge errors
Works with a local standalone echo server but not when registered in Golain

Cause: The cloud dials your device using the topic slug and device name from the Console. Omega connection.root_topic must match that assignment. Check:

Console assignment	Omega `root_topic`
Topic slug `acme`, device name `sensor-01`	`acme/sensor-01` or `/acme/sensor-01`
No topic slug, device name `sensor-01`	`sensor-01`

Confirm in Console device MQTT details or with golain omega scaffold. Fix: Update Omega client YAML or OMEGA_ROOT_TOPIC, redeploy, and verify list services still returns methods. Then retry invoke.

Permissions

Symptoms:

HTTP 403 with "message": "forbidden"
HTTP 401 with "message": "unauthorized"

Check:

Authorization: Bearer header present and token not expired
ORG-ID set to a valid organization UUID
Token is a user OIDC token — not a device MQTT credential

Fix: Grant can_invoke_rpc on the target device for your user or permission group. Reflect additionally requires can_write_device_metadata. Update permissions in Console, then retry.

Wrong URL

Symptoms:

404 service not found in descriptor
404 method not found in descriptor
Invoke hits the management API host instead of the RPC host

Fix:

List services and copy exact services[].name and methods[].name:

curl -sS \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  "$API/projects/$PROJECT_ID/devices/$DEVICE_ID/rpc/services"

Use the full service name (for example golain.device.v1.EchoService), not a short alias.
POST to the RPC host with no /core/api/v1 prefix:

POST https://rpc.ilyama.golain.io/projects/{project_id}/devices/{device_id}/golain.device.v1.EchoService/Echo

Only unary methods use the sync path. Server-stream methods use /streams/{Service}/{Method}. See Invoking methods.

Streaming on wrong path

Symptoms:

HTTP 501 with "streaming rpc not supported" — you POSTed a server-stream, client-stream, or bidi method to the sync path (/{Service}/{Method})
HTTP 501 with "only server-streaming rpc supported" — you POSTed a unary or non-server-stream method to the /streams/ path

Fix:

Method type	Correct cloud path
`unary`	`POST $RPC_PROXY/.../{Service}/{Method}` with Connect JSON headers
`server_stream`	`POST $RPC_PROXY/.../streams/{Service}/{Method}` with `Accept: text/event-stream` and `curl -N`
`client_stream`, `bidi_stream`	Device only — not available from cloud

Example — move EchoServerStream from sync to streams:

curl -N -X POST \
  "$RPC_PROXY/projects/$PROJECT_ID/devices/$DEVICE_ID/streams/golain.device.v1.EchoService/EchoServerStream" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -H "Accept: text/event-stream" \
  -d '{"message":"stream me"}'

For client-stream and bidi methods (EchoClientStream, EchoBidiStream), use device-local tooling — the cloud does not expose these shapes.

Stale schema after OTA

Symptoms:

New firmware deployed; invoke hits old methods or schema errors
List services shows an outdated method list

Check:

Device restarted after OTA so it publishes a fresh schema announce.
descriptor_digest from list services matches what the device logs after restart.

Fix:

curl -sS -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{}' \
  "$API/projects/$PROJECT_ID/devices/$DEVICE_ID/rpc/reflect"

Re-check list services. After breaking proto changes, expect a new digest — plan OTA rollouts so old and new firmware can coexist during migration.

Queued stuck or duplicates

Symptoms:

Poll shows pending or failed for a long time
Call becomes expired — device was offline past expires_at
Call shows succeeded but the device ran the handler twice

Check:

Device online when the platform attempts delivery (queued calls retry on reconnect).
expires_at not passed — default TTL is 900 s; extend with ?ttl_seconds= on enqueue (max 86400).
attempts and last_error on the poll response.

Fix — poll status:

curl -sS \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  "$API/projects/$PROJECT_ID/devices/$DEVICE_ID/rpc/queued/$CALL_ID"

Bring the device online and wait for delivery. For long outages, enqueue again with a higher ttl_seconds. Fix — enqueue with longer TTL:

curl -sS -X POST \
  "$RPC_PROXY/projects/$PROJECT_ID/devices/$DEVICE_ID/enqueue/golain.device.v1.EchoService/Echo?ttl_seconds=3600" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -H "Connect-Protocol-Version: 1" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"message":"retry when online"}'

→ Full behavior: Queued invoke

Queued idempotency

Symptoms:

Motor started twice, counter incremented twice, or duplicate writes after one enqueue
succeeded in poll but duplicate real-world effects

Cause: Queued RPC is at-least-once. The platform may invoke your device more than once for the same logical call. Fix (device firmware):

Add an idempotency key field to request protos; cache results per key on the device.
Prefer naturally idempotent handlers where possible.
Do not rely on call_id — it is not sent to the device today.

→ Omega setup — Idempotency

Self-hosted operators

If the device connects and publishes data but RPC schema never binds or invoke never reaches devices on a self-hosted stack, verify apis, mqtt-broker, and worker processes are running and topology is current. See Self-hosted deployment.

Operator checklist

Run in order for a single device:

# 1. Device online? — Console or: golain devices list

# 2. Schema bound?
curl -sS -H "Authorization: Bearer $TOKEN" -H "ORG-ID: $ORG_ID" \
  "$API/projects/$PROJECT_ID/devices/$DEVICE_ID/rpc/services"

# 3. Reflect if empty (requires can_write_device_metadata)
curl -sS -X POST -H "Authorization: Bearer $TOKEN" -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" -d '{}' \
  "$API/projects/$PROJECT_ID/devices/$DEVICE_ID/rpc/reflect"

# 4. Probe unary Echo
curl -sS -X POST \
  "$RPC_PROXY/projects/$PROJECT_ID/devices/$DEVICE_ID/golain.device.v1.EchoService/Echo" \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -H "Connect-Protocol-Version: 1" \
  -d '{"message":"diagnostic ping"}'

Local dev: set API=https://localhost:19090/core/api/v1 and RPC_PROXY=http://localhost:8082.

Still stuck?

Omega setup — enable device-rpc on the device
Invoking methods — sync invoke, SSE streaming, and error codes
API reference — all endpoints
Queued invoke — async delivery

This guide covers Device RPC only. For legacy shell command RPC (exec:… over signed control topics), see Remote control — Legacy shell RPC.

Omega edge runtime

Device RPC

Edge SQLite sync

Edge agent (Omega)

Operate & query

Wire protocol reference

Device RPC troubleshooting

Quick diagnostic flow

Symptom index

Schema not bound

Device offline / can’t reach device

Root topic mismatch

Permissions

Wrong URL

Streaming on wrong path

Stale schema after OTA

Queued stuck or duplicates

Queued idempotency

Self-hosted operators

Operator checklist

Still stuck?

​Quick diagnostic flow

​Symptom index

​Schema not bound

​Device offline / can’t reach device

​Root topic mismatch

​Permissions

​Wrong URL

​Streaming on wrong path

​Stale schema after OTA

​Queued stuck or duplicates

​Queued idempotency

​Self-hosted operators

​Operator checklist

​Still stuck?

Quick diagnostic flow

Symptom index

Schema not bound

Device offline / can’t reach device

Root topic mismatch

Permissions

Wrong URL

Streaming on wrong path

Stale schema after OTA

Queued stuck or duplicates

Queued idempotency

Self-hosted operators

Operator checklist

Still stuck?