Documentation

Appearance

How do offline datasets work? #

Offline datasets let a connected product carry a bounded copy of cloud-managed data locally so the device can continue making product decisions when Dataplicity is unavailable.

Typical examples include:

  • authorised tags or credentials;
  • site configuration;
  • approved product records;
  • schedules;
  • customer-specific reference data needed for offline operation.

Define what authority the local copy has, how stale it may become, and how the product reconciles changes after reconnect.

Create an offline dataset #

Create the customer data model and its fields first. Then open the Product Application under Developer > Product Applications, go to Build > Offline datasets, and select Add offline dataset.

The creation wizard has two steps.

1. Choose the dataset and source #

In Source:

  1. Enter a customer-readable Name.
  2. Review the dataset Key. The wizard generates it from the name in lowercase underscore form and limits it to 64 characters. Edit it before saving only when the device contract requires a different stable identifier.
  3. Choose the Source model. Each offline dataset projects exactly one customer data model.

The wizard initially selects the first available model. It derives the name from that model when the name is empty and selects all of its fields. A source model can have only one offline dataset in the application.

2. Choose projected fields #

In Fields, select at least one field to copy to devices. Select all and Clear help with larger models. The field list shows the customer label, stable field key, and field type.

The live Projection preview updates as the selection changes. It states the field count, source model, and dataset key. This preview describes the projection contract; it does not contain customer records.

Select Save to add the definition to the editable draft. The wizard fixes new definitions to device targeting and JSON format.

Edit the projection safely #

Editing opens at Fields, because changing the projection is the common task. You can go back to change the Name or Source model, and you can change the projected fields. The dataset Key is shown as read-only and the stable dataset identity is retained. Device targeting, JSON format, expiry, size, and delta compaction settings are not editable in this wizard.

An edit changes only the application draft. Publish a new Product Application version before generating data from the revised definition. Deleting a dataset also removes it only from the editable draft; previously published versions are unchanged.

Treat the source model and projected field keys as a device-facing schema. Before publishing:

  • keep the dataset key stable;
  • retain fields that deployed software still reads;
  • add consumer support before changing a field's meaning or type;
  • verify that every deployed reader tolerates added fields;
  • plan a migration when changing the source model.

Product Application publish compatibility rejects removal of model fields while instances still use prior versions and rejects field type changes. Those checks do not replace compatibility tests in the device application.

Publish, generate, and activate #

After publishing the Product Application version, use Preview offline dataset against a sandbox instance to generate and inspect the next payload. Generate the desired dataset for the intended Product instance and device. Generation produces an immutable base or an ordered delta from an immutable base. Identical projected content reuses the existing base instead of rewriting it.

The Product Runtime fetches the complete base and delta chain, verifies each content SHA-256, stages every generation, and atomically switches the active marker. The previous active marker becomes the last-known-good version. A failed download, hash check, chain check, or activation leaves the current usable version in place and reports the apply failure.

On reconnect and after a device or application restart:

  1. read the active dataset through the Product Runtime rather than a staging path;
  2. confirm the reported applied generation matches the desired generation;
  3. confirm the runtime reports current, rather than stale, failed, or expired;
  4. exercise one safe local lookup against known fixture data;
  5. confirm corrupt or hard-expired data is rejected and that a still-valid last-known-good version is used when available.

Dataplicity delivers, verifies, and activates the generation through the Product Runtime. The device application still owns its schema checks, use of the records, persistence requirements beyond that runtime contract, and safe behaviour when no usable generation exists.

Project secret fields deliberately #

A projected field with type Secret is stored as a one-way hash. Customer and public record responses expose only whether the value is set. An offline dataset that includes the field delivers the stored hash, never the plaintext, so device software can verify a presented credential without receiving a readable secret.

Hash comparison belongs in the trusted device application. Do not treat a secret hash as an encryption format or attempt to recover the original value. Exclude the field entirely when offline verification is not required.

Use offline data only for decisions the device is allowed to make locally #

A dataset is useful when the device already has a local decision it can safely make.

For example:

text
cloud/customer system
        |
        | published dataset version
        v
device local copy
        |
        | local lookup while online or offline
        v
product decision

The device should still enforce its own safety and domain rules.

Define the dataset contract #

For every offline dataset specify:

  • record schema;
  • stable record identifiers;
  • owning customer/site/product scope;
  • dataset version;
  • freshness/expiry policy;
  • what happens when the dataset is missing;
  • what happens when it is stale;
  • whether the device can create local state derived from it;
  • how updates are applied atomically;
  • how much storage is required.

Without those rules, an offline cache can quietly become an uncontrolled second database.

Staleness is a product decision #

Different products tolerate different ages of offline data.

Examples:

  • an HVAC weekly schedule may remain useful for days;
  • a list of service technicians may tolerate a moderate delay;
  • available prepaid credit may require a much tighter model;
  • revoked access credentials may need explicit expiry or revocation handling.

Do not present “works offline” as if it means “cloud data can be stale forever.”

Credit and entitlement need more than a copied number #

Suppose a device receives a balance of 100 units and then spends 40 units while offline. Another device may also be operating offline against related state.

A copied balance alone does not solve distributed credit accounting.

For credit, prepaid access, fuel, or other scarce entitlement, you may need a domain-specific model such as:

  • per-device reservations;
  • bounded offline allowances;
  • monotonic transaction IDs;
  • local durable journals;
  • expiry windows;
  • reconciliation after reconnect;
  • conflict handling and reversal rules.

Offline datasets can distribute the approved input to that model. They do not automatically make distributed financial state safe.

Apply updates atomically #

A device should not observe a half-written dataset.

A common local pattern is:

  1. receive the new version into temporary storage;
  2. validate schema/checksum/signature as required by your product;
  3. fsync/persist according to durability needs;
  4. atomically switch the active version;
  5. keep or remove the previous version according to rollback policy.

The exact implementation depends on the device application and storage constraints.

Be mindful of flash wear #

Large or frequently changing datasets can create unnecessary writes on embedded flash.

Design for the actual update pattern:

  • send meaningful changes rather than rewriting continuously;
  • batch updates where acceptable;
  • avoid persisting volatile fields that are not required offline;
  • use an appropriate local filesystem/database strategy;
  • measure write amplification on constrained hardware;
  • reserve durable writes for data whose loss actually matters.

Dataplicity can deliver data; your device application still owns how it persists and consumes that data safely.

Report which version is active #

For operational confidence, have the device report the active dataset version or another verification signal.

Then an operator can distinguish:

  • cloud version 42 published;
  • device online but still using version 41;
  • device offline using version 41;
  • device rejected version 42;
  • device successfully activated version 42.

That is more useful than merely knowing that a download request was issued.

Scope datasets correctly #

Customer data should not leak between tenants.

If a dataset is customer/site scoped:

  • generate/distribute only the records authorised for that scope;
  • remove or replace cached data when a device is reassigned;
  • test cross-customer isolation;
  • preserve audit evidence of version/allocation changes where required.

See Customer, site, and device isolation.

Offline-first workflow pattern #

A healthy architecture is often:

text
cloud defines bounded policy/data
        |
        v
device caches approved version
        |
        +----> product continues during outage
        |
        v
local journal records important events
        |
        v
reconnect -> upload/reconcile -> receive newer policy

This keeps cloud convenience without making connectivity a prerequisite for correct local operation.