Documentation

Appearance

Incidents and on-call ​

Use incident response when a monitor failure needs ownership, acknowledgement, escalation, delivery tracking, or public communication. Internal incidents currently support service-endpoint, user-impact, heartbeat, and custom source types.

Current Alerts page showing active connectivity events across several customer sites
Alerts provide device, customer or network, status, time, and message context for triage.
Current Incident management page with open and resolved incidents
Incident management separates active response work from resolved events.

Lifecycle ​

An incident opens with a severity, source identifier, affected resource, and stable deduplication key. Its internal states are:

  • Open: waiting for acknowledgement
  • Acknowledged: owned by a responder
  • Resolved: recovery or a manual resolution was recorded
  • Final tier unacknowledged: every configured tier elapsed without acknowledgement

Acknowledgement stops further escalation for that incident. Resolution records an event and can also assign the resolving user if nobody acknowledged it. Auto-resolution can be scheduled, cancelled by renewed failure, and recorded in the timeline.

Configure on-call schedules ​

Schedules support three tiers. Each tier selects a group, timezone, active days, optional local start and end times, escalation delay, grace period, and notification rotation.

Use Notify entire group when every eligible responder should receive the event. Use Round robin to distribute incidents. Overnight windows are supported when the start time is later than the end time.

Delivery records can use web, mobile push, or email and show queued, dispatching, sent, delivered, failed, or cancelled. A sent state does not prove a human saw the notification. Test every channel and keep at least one break-glass path outside the affected system.

Bind monitors with automation ​

Incident automation binds monitor signals to incident creation and settlement. Configure trigger and settle windows, severity, cooldown, and source selection. Use a stable source set to avoid duplicate incidents during a continuing outage.

Test three paths:

  1. enough failures open one incident
  2. renewed failures do not create duplicates during cooldown
  3. enough successful samples settle or resolve the incident as configured

Use device and fleet offline monitors for alerts. For an incident that enters the on-call workflow, use service, heartbeat, or user-impact monitoring.

Publish safely ​

Internal incident visibility and public status-page visibility are separate. Select the intended status pages and review the summary before publishing. Public updates should state customer impact, current action, and next update time without exposing hostnames, logs, identities, or security details.