Documentation

Appearance

Respond to an incident ​

This playbook connects alerts, incidents, logs, monitor context, remote tools, and audit history. Adapt the technical checks to your product.

1. Establish ownership and impact ​

Open the alert or incident. Confirm organisation, customer or site, affected resource, severity, source, and first-observed time. Acknowledge an incident before making changes so another responder does not duplicate work.

Check the source:

  • HTTP result: status, response time, required text, and error
  • user impact: failed flows, metric thresholds, poll error, and freshness
  • heartbeat: last ping and expected interval
  • device monitor: selected devices, offline count or ratio, and minimum-impact guard
  • log alert: query, grouping key, throttle window, and matching records

2. Gather evidence ​

Open Logs with the resource and time already scoped where possible. Otherwise build a query such as:

text
device:abc123 since:"30 minutes ago" level:ERROR

Inspect context before and after representative errors. Compare another healthy device or a pre-failure window. Preserve the query and concise findings in the incident timeline.

3. Choose the least invasive action ​

Prefer a known fleet job or product action when it gives target preview, confirmation, and per-target history. Use remote shell for one-device diagnosis or a carefully bounded fix. Use Wormhole when the relevant response tool is a device-hosted web application.

Before a command:

  1. verify the device identity and customer or site
  2. state the expected result and rollback
  3. avoid commands that remove logs or restart unrelated services
  4. use the smallest target set
  5. obtain approval required by your organisation

Use monitors, alerts, and incidents as the health model. Remote terminal and Wormhole are response tools for diagnosis and recovery.

4. Verify recovery ​

Re-run an independent check. Wait for the monitor's configured successful evaluations, confirm the customer journey or endpoint, and check that error volume returns to normal. A successful shell command alone is not recovery.

Resolve the incident with cause, action, verification, and remaining risk. If customer impact was public, publish a short resolved update to the selected status pages.

5. Close the operational loop ​

Review the incident event timeline, fleet-job history, alert state, and organisation or device event history. Record any missing evidence or unaudited manual step. Convert repeated diagnosis into a monitor, saved log query, runbook, or guarded fleet job.