> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ilyama.golain.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Alarms troubleshooting

> Diagnose rejected alarm definitions, rules that never fire or never resolve, missing notifications, and API errors.

Work top-down when an alarm misbehaves: definition saved → definition active → rule evaluates → instance opens → notification delivered.

Set these for the copyable commands:

```bash theme={null}
export API=https://api.ilyama.golain.io/core/api/v1
export TOKEN=…
export ORG_ID=…
export PROJECT_ID=…
export ALARM_DEF_ID=…
```

Errors use the envelope `{"ok": 0, "message": "…", "detail": "…"}`. The `message` field carries the reason quoted in this guide.

## Quick diagnostic flow

```mermaid theme={null}
flowchart TD
  A[Alarm misbehaves] --> B{Create or update rejected?}
  B -->|Yes| C[Fix the definition - see Rejected definitions]
  B -->|No| D{Definition active?}
  D -->|No| E[Activate it - check for quarantine]
  D -->|Yes| F{Instance on the Board?}
  F -->|No| G[Rule never fires]
  F -->|Yes| H{Notification received?}
  H -->|No| I[Check silences, inhibitions, subscriptions]
  H -->|Yes| J{Alarm stays after condition clears?}
  J -->|Yes| K[Acknowledge, latching, or on_no_data]
```

## Symptom index

| Symptom | Likely cause | Section |
| - | - | - |
| `400` with a `kind`, `on_no_data`, or `staleness_window_seconds` message | Invalid definition fields | [Rejected definitions](#rejected-definitions) |
| `400` mentioning `query_spec.sql` and a relative window | SQL is not bounded by a relative `event_ts` window | [Query budget rejections](#query-budget-rejections) |
| `400` "query\_spec.sql was rejected by the dataspace gateway" | SQL does not plan against the dataspace | [Query rejected by the gateway](#query-rejected-by-the-gateway) |
| `503` "query\_spec.sql could not be validated" | Validation could not run | [Query rejected by the gateway](#query-rejected-by-the-gateway) |
| `409` "alarm definition key already exists in project" | Duplicate `key` | [Duplicate key](#duplicate-key) |
| `400` "\_keys is required", or a PATCH changes nothing | Missing or incomplete `_keys` | [PATCH did not change a field](#patch-did-not-change-a-field) |
| Definition switched to **Deactivated** on its own | Rule was quarantined after repeated slow evaluations | [Rule quarantined](#rule-quarantined) |
| Rule saved but never fires | Inactive, bad CEL, wrong dataspace, shelved, or re-alarm delay | [Rule never fires](#rule-never-fires) |
| Device-offline rule never fires for a device | Device is not an MQTT device | [Staleness rule and non-MQTT devices](#staleness-rule-and-non-mqtt-devices) |
| Device-offline alarm for a device that was never deployed | Never-connected devices count as offline | [Never-connected devices](#never-connected-devices) |
| Alarm stays on the Board after the condition clears | Unacknowledged, latching, or `keep_firing` on no data | [Alarm does not go away](#alarm-does-not-go-away) |
| Alarm is on the Board but nobody was notified | Silence, inhibition, or subscription filter | [No notification](#no-notification) |
| Console **Test** shows **not available** | Condition dry-run is not available yet | [Console Test shows not available](#console-test-shows-not-available) |
| `403` "forbidden" | Missing `can_manage_assets` | [Permissions and rate limits](#permissions-and-rate-limits) |
| `429` "rate limit exceeded" | Too many requests per second | [Permissions and rate limits](#permissions-and-rate-limits) |

***

## Rejected definitions

Create (`POST …/alarm-definitions`) and update (`PATCH …/alarm-definitions/{alarm_def_id}`) check the definition before saving it. An update is checked as the merged result of your change and the stored definition.

| `message` | Cause | Fix |
| - | - | - |
| `invalid kind "…": must be one of event, threshold, staleness` | Unknown `kind` | Use `event`, `threshold`, or `staleness`. Omitting `kind` means `event`. |
| `invalid on_no_data "…": must be one of resolve, keep_firing, alert` | Unknown no-data policy | Use `resolve`, `keep_firing`, or `alert`, or omit it. |
| `source_dataspace_id is required for threshold rules` | Threshold rule without a source | Set `source_dataspace_id` to the dataspace the query reads. |
| `query_spec is required for threshold rules` | Threshold rule without a query | Add a `query_spec` with SQL. |
| `query_spec.sql is required for threshold rules …` | `query_spec` has no SQL | Put native SQL against the dataspace read view in `query_spec`. |
| `invalid query_spec: …` | `query_spec` could not be parsed | Send valid JSON in the same shape the Console editor saves. |
| `staleness_window_seconds must be > 0 for staleness rules` | Missing or zero window | Set `staleness_window_seconds` to a positive number of seconds. |
| `Failed to parse request body` | Unknown field or bad JSON | Remove fields the endpoint does not accept. Request bodies are strict. |

Missing `key`, `name`, or `condition_cel` on create also returns `400`.

***

## Query budget rejections

**Symptoms:**

`400` on create or update with one of:

* `query_spec.sql must bound event_ts with a relative window (e.g. event_ts >= now() - INTERVAL 4 HOUR); an unbounded scan stalls the shared evaluator`
* `query_spec.sql must not pin an absolute time bound; use a relative window (e.g. event_ts >= now() - INTERVAL 4 HOUR)`
* `query_spec.sql scans a … window, over the 24h0m0s cap; narrow the relative window`

**Cause:**

Every scheduled query — threshold rules, and event rules that carry SQL — must scan a bounded, recent slice of data. The query must:

1. Filter `event_ts` with a window **relative to `now()`**.
2. Use a window of **24 hours or less**.
3. Contain **no absolute time floor** — no date literal such as `'2026-01-01'` and no `toDateTime(…)`, `parseDateTime…(…)`, or `fromUnixTimestamp…(…)` call.

**Fix:**

Rewrite the filter in one of the accepted forms:

| Form | Example |
| - | - |
| `INTERVAL` | `WHERE event_ts >= now() - INTERVAL 15 MINUTE` |
| `toInterval…` | `WHERE event_ts >= now() - toIntervalHour(4)` |
| `subtract…` | `WHERE event_ts >= subtractMinutes(now(), 30)` |
| Bare seconds | `WHERE event_ts >= now() - 900` |
| Reversed | `WHERE now() - INTERVAL 1 HOUR <= event_ts` |

> [!TIP]
>
> In the Console, the **Test** panel shows **not bounded** and does not run the query until the SQL has a relative `event_ts` window.

***

## Query rejected by the gateway

After the budget check, the platform plans your SQL against the dataspace without running it.

| Status | `message` | Cause | Fix |
| - | - | - | - |
| `400` | `query_spec.sql: …` (the planner's reason) | Unknown column, unknown function, or invalid SQL | Fix the SQL. Run it in the Console query editor first. |
| `403` | `query_spec.sql: …` | The rule's author cannot read the dataspace | Grant read access to the dataspace, or have someone with access author the rule. |
| `503` | `query_spec.sql could not be validated: dataspace gateway unavailable` | The query service was unreachable, overloaded, or timed out | Retry later. The rule is not saved until validation succeeds. |

An update only re-validates the SQL when you change `query_spec` or `kind`. Changing other fields works even while validation is unavailable.

***

## Duplicate key

**Symptom:** `409` with `alarm definition key already exists in project`.

**Fix:** Each definition `key` must be unique in the project. Choose a new `key`, or update the existing definition instead.

***

## PATCH did not change a field

**Symptoms:**

* `400` with `_keys is required`
* `200` response, but a field you sent kept its old value

**Cause:** PATCH on definitions, silences, and inhibitions only applies the fields named in `_keys`. Fields in the body but not in `_keys` are ignored.

**Fix:**

```bash theme={null}
curl -sS -X PATCH \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  -H "Content-Type: application/json" \
  -d '{"priority": 1, "on_no_data": "resolve", "_keys": ["priority", "on_no_data"]}' \
  "$API/projects/$PROJECT_ID/alarm-definitions/$ALARM_DEF_ID"
```

***

## Rule quarantined

**Symptoms:**

* A threshold or event rule shows **Deactivated** in **Alarms → Definitions**, but nobody deactivated it
* `GET …/alarm-definitions/{alarm_def_id}` returns `"is_active": false`

**Cause:**

Each scheduled evaluation has a time budget equal to the rule's evaluation interval, with a floor of **5 seconds** and a cap of **60 seconds**. An evaluation counts as an overrun when it takes longer than that budget, times out, or is turned away because the query service is overloaded. After **3 consecutive overruns** the platform deactivates the rule so it cannot slow down evaluation for everyone else.

**Fix:**

1. Make the query cheaper: narrow the `event_ts` window, filter earlier, and aggregate less.
2. Raise `eval_interval_seconds` if the rule does not need to run that often.
3. Reactivate the rule. Reactivating clears the quarantine and resets the overrun count.

```bash theme={null}
curl -sS -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  "$API/projects/$PROJECT_ID/alarm-definitions/$ALARM_DEF_ID/activate"
```

If the query is still too slow, the rule is quarantined again after three more overruns.

***

## Rule never fires

Check these in order.

| Check | What to look for | Fix |
| - | - | - |
| **Definition active** | **Alarms → Definitions** shows **Evaluating** or **Listening**, not **Deactivated**. The API returns `"is_active": true`. | Click **Activate**, or `POST …/activate`. See [Rule quarantined](#rule-quarantined). |
| **CEL compiles** | `condition_cel`, `resolve_cel`, and `clear_cel` are only compiled when the rule evaluates, not when you save it. | Make sure each expression returns a `bool` and only uses the variables `event`, `asset`, `alarm`, and `now`. |
| **Right dataspace** | An event rule reacts to rows written to its `source_dataspace_id`. | Point `source_dataspace_id` at the dataspace that receives the data. |
| **Condition held long enough** | With `for_seconds` set, the instance stays `pending` until the condition holds that long. A pending instance whose condition drops closes without firing. | Lower `for_seconds`, or confirm the condition holds continuously. |
| **Not shelved** | An open instance with `shelved_until` in the future is not evaluated at all until the shelve ends. | Unshelve it: `POST …/alarm-instances/{alarm_instance_id}/unshelve`. |
| **Re-alarm delay** | With `re_alarm_delay_seconds` set, a new instance does not open until that long after the previous one for the same labels. | Wait out the delay, or lower `re_alarm_delay_seconds`. |
| **Query returns rows** | A threshold rule's SQL returns rows in the current window. | Run the SQL in **Test** or the Console query editor. |

> [!WARNING]
>
> A CEL expression that does not compile, or does not return a `bool` (for example `event.temperature + 1`), is still saved. It then fails on every evaluation and the rule never fires. Check expressions carefully before saving.

***

## Staleness rule and non-MQTT devices

**Symptom:** A `staleness` (device-offline) rule never fires for LoRaWAN, HTTP, or OPC UA devices.

**Cause:** "Offline" means the MQTT broker recorded a disconnect. Staleness rules only cover the project's **enabled MQTT devices**. Devices that never hold a broker connection are never "offline" in this sense.

**Fix:** Use a `threshold` rule that alarms when telemetry stops arriving, based on `max(event_ts)` in the device's dataspace. For example, compare `max(event_ts)` per device against `now()` inside a window of 24 hours or less.

**Timing to expect:**

* Offline devices are checked **once a minute**, so firing lands up to about a minute after `staleness_window_seconds` elapses. `eval_interval_seconds` has no effect on staleness rules.
* A reconnect resolves the device's offline alarms **immediately**, without waiting for the next check.

***

## Never-connected devices

**Symptom:** A device-offline alarm opens for a device that was created but never deployed.

**Cause:** A device that has never connected counts as offline since the moment it was created.

**Fix:** Disable devices that are not deployed yet, or connect them once. Use a [silence](/alarms/silences-and-shelving) if you only need to stop notifications while you roll out.

***

## Alarm does not go away

| Symptom | Cause | Fix |
| - | - | - |
| `condition_state` is `resolved` and `ack_state` is `unacknowledged` | A resolved alarm stays open for operator review until someone acknowledges it | Acknowledge it. It closes on the next evaluation while the condition stays clear. |
| `condition_state` is `latched` | The rule has `latching: true`, so a cleared alarm latches instead of resolving | A latched instance stays open; acknowledging it does not close it, and there is no reset action yet. Set `latching` to `false` if you do not want this behavior. |
| Alarm keeps `firing` after the data stops | `on_no_data` defaults to `keep_firing`: a series that disappears does not resolve the alarm | Set `on_no_data` to `resolve` to treat missing data as a clear. |
| Alarm stays `firing` for a while after the value recovers | `keep_firing_for_seconds` holds the alarm until the recovery lasts that long | Lower `keep_firing_for_seconds`. |
| Alarm stays `firing` with the value just below the limit | `resolve_cel`, `clear_cel`, or `deadband` requires a bigger recovery | Adjust the resolve expression or the deadband. |

```bash theme={null}
curl -sS -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  "$API/projects/$PROJECT_ID/alarm-instances/$ALARM_INSTANCE_ID/acknowledge"
```

→ States and timing: [Alarms overview](/alarms/overview) and [Kinds and conditions](/alarms/kinds-and-conditions)

***

## No notification

**Symptom:** The instance appears on the **Board**, but no message arrives.

Silences and inhibitions **suppress notifications only**. The alarm instance still opens, shows on the Board, and can be acknowledged.

| Check | What to look for | Fix |
| - | - | - |
| **Silences** | An active silence in **Alarms → Silences** whose matchers match the alarm's labels, with the current time between `starts_at` and `ends_at` | Delete the silence, end it early, or narrow its matchers. |
| **Inhibitions** | An inhibition whose source matchers match another alarm that is currently firing | Clear the source alarm, or narrow the inhibition's matchers or `equal_labels`. |
| **Subscription events** | The project's notification subscription lists the alarm event keys in `subscribed_events`: `alarm.alarm_instance.activated.v1` (fired), `alarm.alarm_instance.cleared.v1` (resolved), `alarm.alarm_instance.acknowledged.v1` | Add the missing event key. |
| **Subscription filter** | `alarm_def_ids` is empty (every alarm) or includes this definition | Add the definition ID, or clear the list. |
| **Subscription enabled** | `enabled` is `true` | Enable the subscription. |
| **Repeat notifications** | Repeats need `repeat_interval_seconds` on the definition, and stop once the alarm is acknowledged or resolves | Set `repeat_interval_seconds`. |

> [!NOTE]
>
> Changes to silences and inhibitions can take up to about 15 seconds to apply.

List the project's subscriptions:

```bash theme={null}
curl -sS \
  -H "Authorization: Bearer $TOKEN" \
  -H "ORG-ID: $ORG_ID" \
  "$API/projects/$PROJECT_ID/notification-subscriptions"
```

→ Full setup: [Notifications](/alarms/notifications) and [Silences and shelving](/alarms/silences-and-shelving)

***

## Shelve rejected

| `message` | Cause | Fix |
| - | - | - |
| `only active alarm instances can be shelved` | The instance is not active | Shelve only alarms that are currently active. |
| `shelved_until must be in the future` | `shelved_until` is in the past | Send a future RFC 3339 time. |
| `shelved_until exceeds shelve_max_hours (N)` | Shelve is longer than the definition allows (default **24** hours) | Shorten the shelve, or raise `shelve_max_hours` on the definition. |

***

## Console Test shows not available

**Symptom:** In the definition editor, **Test** shows the **Condition** check as **not available**.

**Cause:** The platform cannot dry-run the CEL condition yet. The query check still runs your SQL against the dataspace.

**Fix:** Confirm the query returns the rows you expect, save the rule, and watch the **Board** for the result. Event rules without a query have no dry-run either — write a matching row to the dataspace instead.

***

## Permissions and rate limits

| Status | `message` | Cause | Fix |
| - | - | - | - |
| `401` | `unauthorized` | Missing or expired token | Send a valid `Authorization: Bearer` token and `ORG-ID`. |
| `403` | `forbidden` | Creating, updating, deleting, activating, or deactivating definitions, silences, or inhibitions requires **`can_manage_assets`** on the project | Grant `can_manage_assets` to your user or permission group. |
| `429` | `rate limit exceeded` | Too many requests per second for your organization | Back off and retry. |

Per-second limits for alarm routes:

| Operation | Limit |
| - | - |
| List and get | 64 req/s |
| Create definitions, silences, inhibitions | 12 req/s |
| Update, delete, activate, deactivate, acknowledge, shelve, unshelve | 32 req/s |

## Still stuck?

* [API reference](/alarms/api-reference) — every endpoint, body, and response
* [Kinds and conditions](/alarms/kinds-and-conditions) — event, threshold, and staleness rules, SQL, CEL, timing, and no-data behavior
* [Console guide](/alarms/console-guide) — Board, Definitions, and Silences


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.