From signal to verified recovery

The objective of incident response is not to run a successful command. It is to restore the affected service, verify the result independently, and leave enough context for the next responder.

Operating connected products series

1. After devices ship
2. One device record from factory to field ticket
3. Dataplicity beside your existing stack
4. From signal to verified recovery
5. Trust on someone else's network

This method connects alerts, incidents, logs, monitor context, fleet actions, remote tools, and product history around the affected device or cohort. Adapt the technical checks and approval policy to your product.

Why shell-first response fails

A terminal is a powerful response tool, but a poor incident record. Starting with an interactive session can hide the questions that determine whether the action is safe:

• Is the report about one unit or a cohort?
• Which customer, site, product class, or network is affected?
• Did the application fail, the network disappear, or the monitor misclassify a transient state?
• Has another responder already taken ownership?
• What independent signal will prove recovery?

The operational loop keeps those questions visible:

scope and ownership → evidence → controlled action → verification → review

1. Establish scope and ownership

Start from the alert, incident, customer report, or device record. Confirm:

• organisation
• customer or site where mapped
• affected device, class, tags, or other selected cohort
• severity and observed customer impact
• source and first-observed time
• current owner and escalation state

Dataplicity alerts and incidents serve different purposes. Device and fleet connectivity monitors create alerts. Use an incident where enabled when the response requires acknowledgement, escalation, on-call ownership, or selected public communication.

If an incident exists, acknowledge it before making changes so another responder does not duplicate the intervention. If the organisation owns the wider response in ITSM or another on-call system, preserve the Dataplicity device and incident links there.

Read the source at the right altitude

Check what the source can actually establish:

Source	Review
HTTP or service monitor	Status, response time, required text, evaluation history, and reported error
User-impact monitor	Failed flow, threshold, poll error, and data freshness
Heartbeat	Last ping and expected interval
Device monitor	Selector, included devices, offline count or ratio, and minimum-impact guard
Log alert	Query, grouping key, throttle window, and representative matching records
Customer report	Affected journey, unit or site description, observed time, and reproducibility

A hostname or error message is not yet scope. Resolve it to the device record and compare it with its class, customer, site, rollout, or network cohorts before concluding that the product is broadly affected.

2. Gather evidence

Open Logs with the affected resource and time already scoped where possible. Otherwise start narrowly:

device:abc123 since:"30 minutes ago" level:ERROR

Inspect context before and after representative records. Compare:

• the affected device with a healthy peer in the same class
• the failure window with a known healthy window
• one customer or site with another
• an affected network or ISP cohort with the rest of the fleet

Widen the time range only after validating the query. A single error line may be a symptom, retry, or unrelated background event.

Preserve a concise evidence note:

Observed:
Scope:
Evidence:
Working hypothesis:
Next safe check:

Do not copy secrets, unnecessary personal data, or unrestricted log exports into a ticket or public status update. Product log availability and retention depend on the organisation's configuration; preserve required evidence according to the organisation's own policy.

3. Choose the least invasive action

Choose an action whose target and expected result can be stated before execution.

Prefer a known product action or guarded fleet job when it provides target preview, confirmation, progress, and per-device results. Use Remote Shell for one-device diagnosis or a carefully bounded fix. Use Wormhole when the relevant response tool is a device-hosted web application with its own authorisation.

Before a command or fleet action:

1. verify the device identity, customer or site, and target selector
2. state the expected result and how it will be measured
3. identify rollback or the safe stop condition
4. avoid deleting logs or restarting unrelated services
5. use the smallest target set
6. obtain the approval required by the organisation

The platform authorisation and device-side Linux permission boundary are separate. A command inherits the agent's effective OS identity; Dataplicity does not inherently elevate it. Read the security model before defining production response permissions.

Escalate instead of improvising

Stop and escalate when:

• the device identity or ownership is uncertain
• the proposed action cannot be bounded to the intended target
• evidence suggests physical damage, safety risk, or site work
• recovery would require credentials or permissions outside the approved support boundary
• the action could destroy evidence needed for investigation
• a customer, regulatory, or internal approval is required

Remote access makes a device reachable; it does not make every remote action appropriate.

4. Verify recovery independently

Re-run a check that is independent of the action result:

• wait for the monitor's configured successful evaluations
• confirm the customer journey or endpoint
• check that error volume and connection state return to the expected range
• compare the repaired device with a healthy peer
• confirm that no adjacent cohort regressed

A zero exit code, completed fleet job, or responsive terminal proves that the action ran. It does not by itself prove that the product or customer journey recovered.

Record:

• action performed
• observed result
• independent verification
• remaining risk or follow-up

If customer impact was published, update only the selected customer-safe facts on the relevant status surface. Internal logs, device identifiers, credentials, and remote-tool details stay out of public communication.

5. Close the operational loop

Review the evidence available for the response:

• alert state and monitor evaluation
• incident timeline and ownership changes
• fleet-job target and per-device results
• relevant organisation or device event history
• external ticket or communication record

Coverage and retention differ across features. Record any manual step or missing evidence rather than implying that one universal audit trail contains the entire response.

Turn repeated work into a safer next response:

• save the validated log query
• create or refine a monitor
• write a product runbook
• replace a repeated interactive command with a guarded fleet job
• improve device, customer, site, or cohort metadata
• revise escalation and customer-communication criteria

The loop is complete when the service is verified, ownership is clear, and the next occurrence can be handled with less reconstruction.

Rehearse the method

Run the process against one staging fault before depending on it in production:

1. trigger or simulate a known monitor condition
2. confirm the alert retains enough resource context
3. practise acknowledgement and handoff
4. gather evidence without opening a shell first
5. execute a reversible action under the production device identity
6. verify through the monitor or customer journey
7. review what the history did and did not retain

The rehearsal tests the joins between systems and teams, not only whether the device is reachable.

Continue the series

Next, read Trust on someone else's network for the connection, identity, access, and device-side privilege boundaries behind the response path.

Related

• Search logs
• Incidents and on-call
• Fleet jobs
• Remote shell
• Wormhole
• Integrations
• Audit trails
• Fix a remote site without a truck roll