Alarms troubleshooting

Work top-down when an alarm misbehaves: definition saved → definition active → rule evaluates → instance opens → notification delivered. Set these for the copyable commands:

export API=https://api.ilyama.golain.io/core/api/v1
export TOKEN=…
export ORG_ID=…
export PROJECT_ID=…
export ALARM_DEF_ID=…

Errors use the envelope {"ok": 0, "message": "…", "detail": "…"}. The message field carries the reason quoted in this guide.

Quick diagnostic flow

Symptom index

Symptom	Likely cause	Section
`400` with a `kind`, `on_no_data`, or `staleness_window_seconds` message	Invalid definition fields	Rejected definitions
`400` mentioning `query_spec.sql` and a relative window	SQL is not bounded by a relative `event_ts` window	Query budget rejections
`400` “query_spec.sql was rejected by the dataspace gateway”	SQL does not plan against the dataspace	Query rejected by the gateway
`503` “query_spec.sql could not be validated”	Validation could not run	Query rejected by the gateway
`409` “alarm definition key already exists in project”	Duplicate `key`	Duplicate key
`400` “_keys is required”, or a PATCH changes nothing	Missing or incomplete `_keys`	PATCH did not change a field
Definition switched to Deactivated on its own	Rule was quarantined after repeated slow evaluations	Rule quarantined
Rule saved but never fires	Inactive, bad CEL, wrong dataspace, shelved, or re-alarm delay	Rule never fires
Device-offline rule never fires for a device	Device is not an MQTT device	Staleness rule and non-MQTT devices
Device-offline alarm for a device that was never deployed	Never-connected devices count as offline	Never-connected devices
Alarm stays on the Board after the condition clears	Unacknowledged, latching, or `keep_firing` on no data	Alarm does not go away
Alarm is on the Board but nobody was notified	Silence, inhibition, or subscription filter	No notification
Console Test shows not available	Condition dry-run is not available yet	Console Test shows not available
`403` “forbidden”	Missing `can_manage_assets`	Permissions and rate limits
`429` “rate limit exceeded”	Too many requests per second	Permissions and rate limits

Rejected definitions

Create (POST …/alarm-definitions) and update (PATCH …/alarm-definitions/{alarm_def_id}) check the definition before saving it. An update is checked as the merged result of your change and the stored definition.

`message`	Cause	Fix
`invalid kind "…": must be one of event, threshold, staleness`	Unknown `kind`	Use `event`, `threshold`, or `staleness`. Omitting `kind` means `event`.
`invalid on_no_data "…": must be one of resolve, keep_firing, alert`	Unknown no-data policy	Use `resolve`, `keep_firing`, or `alert`, or omit it.
`source_dataspace_id is required for threshold rules`	Threshold rule without a source	Set `source_dataspace_id` to the dataspace the query reads.
`query_spec is required for threshold rules`	Threshold rule without a query	Add a `query_spec` with SQL.
`query_spec.sql is required for threshold rules …`	`query_spec` has no SQL	Put native SQL against the dataspace read view in `query_spec`.
`invalid query_spec: …`	`query_spec` could not be parsed	Send valid JSON in the same shape the Console editor saves.
`staleness_window_seconds must be > 0 for staleness rules`	Missing or zero window	Set `staleness_window_seconds` to a positive number of seconds.
`Failed to parse request body`	Unknown field or bad JSON	Remove fields the endpoint does not accept. Request bodies are strict.

Missing key, name, or condition_cel on create also returns 400.

Query budget rejections

Symptoms: 400 on create or update with one of:

query_spec.sql must bound event_ts with a relative window (e.g. event_ts >= now() - INTERVAL 4 HOUR); an unbounded scan stalls the shared evaluator
query_spec.sql must not pin an absolute time bound; use a relative window (e.g. event_ts >= now() - INTERVAL 4 HOUR)
query_spec.sql scans a … window, over the 24h0m0s cap; narrow the relative window

Cause: Every scheduled query — threshold rules, and event rules that carry SQL — must scan a bounded, recent slice of data. The query must:

Filter event_ts with a window relative to now().
Use a window of 24 hours or less.
Contain no absolute time floor — no date literal such as '2026-01-01' and no toDateTime(…), parseDateTime…(…), or fromUnixTimestamp…(…) call.

Fix: Rewrite the filter in one of the accepted forms:

Form	Example
`INTERVAL`	`WHERE event_ts >= now() - INTERVAL 15 MINUTE`
`toInterval…`	`WHERE event_ts >= now() - toIntervalHour(4)`
`subtract…`	`WHERE event_ts >= subtractMinutes(now(), 30)`
Bare seconds	`WHERE event_ts >= now() - 900`
Reversed	`WHERE now() - INTERVAL 1 HOUR <= event_ts`

In the Console, the Test panel shows not bounded and does not run the query until the SQL has a relative event_ts window.

Query rejected by the gateway

After the budget check, the platform plans your SQL against the dataspace without running it.

Status	`message`	Cause	Fix
`400`	`query_spec.sql: …` (the planner’s reason)	Unknown column, unknown function, or invalid SQL	Fix the SQL. Run it in the Console query editor first.
`403`	`query_spec.sql: …`	The rule’s author cannot read the dataspace	Grant read access to the dataspace, or have someone with access author the rule.
`503`	`query_spec.sql could not be validated: dataspace gateway unavailable`	The query service was unreachable, overloaded, or timed out	Retry later. The rule is not saved until validation succeeds.

An update only re-validates the SQL when you change query_spec or kind. Changing other fields works even while validation is unavailable.

Duplicate key

Symptom: 409 with alarm definition key already exists in project. Fix: Each definition key must be unique in the project. Choose a new key, or update the existing definition instead.

PATCH did not change a field

Symptoms:

400 with _keys is required
200 response, but a field you sent kept its old value

Cause: PATCH on definitions, silences, and inhibitions only applies the fields named in _keys. Fields in the body but not in _keys are ignored. Fix:

curl -sS -X PATCH \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{"priority": 1, "on_no_data": "resolve", "_keys": ["priority", "on_no_data"]}' \
  "$API/projects/$PROJECT_ID/alarm-definitions/$ALARM_DEF_ID"

Rule quarantined

Symptoms:

A threshold or event rule shows Deactivated in Alarms → Definitions, but nobody deactivated it
GET …/alarm-definitions/{alarm_def_id} returns "is_active": false

Cause: Each scheduled evaluation has a time budget equal to the rule’s evaluation interval, with a floor of 5 seconds and a cap of 60 seconds. An evaluation counts as an overrun when it takes longer than that budget, times out, or is turned away because the query service is overloaded. After 3 consecutive overruns the platform deactivates the rule so it cannot slow down evaluation for everyone else. Fix:

Make the query cheaper: narrow the event_ts window, filter earlier, and aggregate less.
Raise eval_interval_seconds if the rule does not need to run that often.
Reactivate the rule. Reactivating clears the quarantine and resets the overrun count.

curl -sS -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  "$API/projects/$PROJECT_ID/alarm-definitions/$ALARM_DEF_ID/activate"

If the query is still too slow, the rule is quarantined again after three more overruns.

Rule never fires

Check these in order.

Check	What to look for	Fix
Definition active	Alarms → Definitions shows Evaluating or Listening, not Deactivated. The API returns `"is_active": true`.	Click Activate, or `POST …/activate`. See Rule quarantined.
CEL compiles	`condition_cel`, `resolve_cel`, and `clear_cel` are only compiled when the rule evaluates, not when you save it.	Make sure each expression returns a `bool` and only uses the variables `event`, `asset`, `alarm`, and `now`.
Right dataspace	An event rule reacts to rows written to its `source_dataspace_id`.	Point `source_dataspace_id` at the dataspace that receives the data.
Condition held long enough	With `for_seconds` set, the instance stays `pending` until the condition holds that long. A pending instance whose condition drops closes without firing.	Lower `for_seconds`, or confirm the condition holds continuously.
Not shelved	An open instance with `shelved_until` in the future is not evaluated at all until the shelve ends.	Unshelve it: `POST …/alarm-instances/{alarm_instance_id}/unshelve`.
Re-alarm delay	With `re_alarm_delay_seconds` set, a new instance does not open until that long after the previous one for the same labels.	Wait out the delay, or lower `re_alarm_delay_seconds`.
Query returns rows	A threshold rule’s SQL returns rows in the current window.	Run the SQL in Test or the Console query editor.

A CEL expression that does not compile, or does not return a bool (for example event.temperature + 1), is still saved. It then fails on every evaluation and the rule never fires. Check expressions carefully before saving.

Staleness rule and non-MQTT devices

Symptom: A staleness (device-offline) rule never fires for LoRaWAN, HTTP, or OPC UA devices. Cause: “Offline” means the MQTT broker recorded a disconnect. Staleness rules only cover the project’s enabled MQTT devices. Devices that never hold a broker connection are never “offline” in this sense. Fix: Use a threshold rule that alarms when telemetry stops arriving, based on max(event_ts) in the device’s dataspace. For example, compare max(event_ts) per device against now() inside a window of 24 hours or less. Timing to expect:

Offline devices are checked once a minute, so firing lands up to about a minute after staleness_window_seconds elapses. eval_interval_seconds has no effect on staleness rules.
A reconnect resolves the device’s offline alarms immediately, without waiting for the next check.

Never-connected devices

Symptom: A device-offline alarm opens for a device that was created but never deployed. Cause: A device that has never connected counts as offline since the moment it was created. Fix: Disable devices that are not deployed yet, or connect them once. Use a silence if you only need to stop notifications while you roll out.

Alarm does not go away

Symptom	Cause	Fix
`condition_state` is `resolved` and `ack_state` is `unacknowledged`	A resolved alarm stays open for operator review until someone acknowledges it	Acknowledge it. It closes on the next evaluation while the condition stays clear.
`condition_state` is `latched`	The rule has `latching: true`, so a cleared alarm latches instead of resolving	A latched instance stays open; acknowledging it does not close it, and there is no reset action yet. Set `latching` to `false` if you do not want this behavior.
Alarm keeps `firing` after the data stops	`on_no_data` defaults to `keep_firing`: a series that disappears does not resolve the alarm	Set `on_no_data` to `resolve` to treat missing data as a clear.
Alarm stays `firing` for a while after the value recovers	`keep_firing_for_seconds` holds the alarm until the recovery lasts that long	Lower `keep_firing_for_seconds`.
Alarm stays `firing` with the value just below the limit	`resolve_cel`, `clear_cel`, or `deadband` requires a bigger recovery	Adjust the resolve expression or the deadband.

curl -sS -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  "$API/projects/$PROJECT_ID/alarm-instances/$ALARM_INSTANCE_ID/acknowledge"

→ States and timing: Alarms overview and Kinds and conditions

No notification

Symptom: The instance appears on the Board, but no message arrives. Silences and inhibitions suppress notifications only. The alarm instance still opens, shows on the Board, and can be acknowledged.

Check	What to look for	Fix
Silences	An active silence in Alarms → Silences whose matchers match the alarm’s labels, with the current time between `starts_at` and `ends_at`	Delete the silence, end it early, or narrow its matchers.
Inhibitions	An inhibition whose source matchers match another alarm that is currently firing	Clear the source alarm, or narrow the inhibition’s matchers or `equal_labels`.
Subscription events	The project’s notification subscription lists the alarm event keys in `subscribed_events`: `alarm.alarm_instance.activated.v1` (fired), `alarm.alarm_instance.cleared.v1` (resolved), `alarm.alarm_instance.acknowledged.v1`	Add the missing event key.
Subscription filter	`alarm_def_ids` is empty (every alarm) or includes this definition	Add the definition ID, or clear the list.
Subscription enabled	`enabled` is `true`	Enable the subscription.
Repeat notifications	Repeats need `repeat_interval_seconds` on the definition, and stop once the alarm is acknowledged or resolves	Set `repeat_interval_seconds`.

Changes to silences and inhibitions can take up to about 15 seconds to apply.

List the project’s subscriptions:

curl -sS \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  "$API/projects/$PROJECT_ID/notification-subscriptions"

→ Full setup: Notifications and Silences and shelving

Shelve rejected

`message`	Cause	Fix
`only active alarm instances can be shelved`	The instance is not active	Shelve only alarms that are currently active.
`shelved_until must be in the future`	`shelved_until` is in the past	Send a future RFC 3339 time.
`shelved_until exceeds shelve_max_hours (N)`	Shelve is longer than the definition allows (default 24 hours)	Shorten the shelve, or raise `shelve_max_hours` on the definition.

Console Test shows not available

Symptom: In the definition editor, Test shows the Condition check as not available. Cause: The platform cannot dry-run the CEL condition yet. The query check still runs your SQL against the dataspace. Fix: Confirm the query returns the rows you expect, save the rule, and watch the Board for the result. Event rules without a query have no dry-run either — write a matching row to the dataspace instead.

Permissions and rate limits

Status	`message`	Cause	Fix
`401`	`unauthorized`	Missing or expired token	Send a valid `Authorization: Bearer` token and `ORG-ID`.
`403`	`forbidden`	Creating, updating, deleting, activating, or deactivating definitions, silences, or inhibitions requires `can_manage_assets` on the project	Grant `can_manage_assets` to your user or permission group.
`429`	`rate limit exceeded`	Too many requests per second for your organization	Back off and retry.

Per-second limits for alarm routes:

Operation	Limit
List and get	64 req/s
Create definitions, silences, inhibitions	12 req/s
Update, delete, activate, deactivate, acknowledge, shelve, unshelve	32 req/s

Still stuck?

API reference — every endpoint, body, and response
Kinds and conditions — event, threshold, and staleness rules, SQL, CEL, timing, and no-data behavior
Console guide — Board, Definitions, and Silences

Getting started

Dataspaces

Console

Dashboards

Alarms

Ticketing

Alarms troubleshooting

Quick diagnostic flow

Symptom index

Rejected definitions

Query budget rejections

Query rejected by the gateway

Duplicate key

PATCH did not change a field

Rule quarantined

Rule never fires

Staleness rule and non-MQTT devices

Never-connected devices

Alarm does not go away

No notification

Shelve rejected

Console Test shows not available

Permissions and rate limits

Still stuck?

​Quick diagnostic flow

​Symptom index

​Rejected definitions

​Query budget rejections

​Query rejected by the gateway

​Duplicate key

​PATCH did not change a field

​Rule quarantined

​Rule never fires

​Staleness rule and non-MQTT devices

​Never-connected devices

​Alarm does not go away

​No notification

​Shelve rejected

​Console Test shows not available

​Permissions and rate limits

​Still stuck?

Quick diagnostic flow

Symptom index

Rejected definitions

Query budget rejections

Query rejected by the gateway

Duplicate key

PATCH did not change a field

Rule quarantined

Rule never fires

Staleness rule and non-MQTT devices

Never-connected devices

Alarm does not go away

No notification

Shelve rejected

Console Test shows not available

Permissions and rate limits

Still stuck?