Device connectivity disruption

Incident Report for HyperTrack

Resolved

Summary

On July 9, 2026, beginning at 10:00 UTC, devices attempting to establish or re-establish live connections to the HyperTrack platform experienced failures and delays. Devices that were already connected and tracking were largely unaffected, but newly starting devices experienced delayed or missing location data until service was fully restored at 19:25 UTC.

What happened

An unexpected surge in load caused a brief database disruption that disconnected a large number of devices at once. The resulting wave of simultaneous reconnection attempts exceeded our connection-handling capacity, which delayed new device connections for the remainder of the window. We deployed a fix at 19:14 UTC, and connectivity was fully restored by 19:25 UTC.

Customer impact

* Devices starting sessions during the window experienced delayed first location updates — in the worst cases by several hours — and a portion never delivered location data for orders created during the window.
* Some orders completed during the window are missing arrival or on-site time data.
* Location data for devices that remained connected was captured normally.

What we are doing to prevent recurrence

* Fixed the configuration that allowed the initial disruption.
* Hardened connection handling — bounded requests, timeouts, and load-shedding — so mass reconnections can no longer create a sustained backlog.
* Added safeguards that cap and pace surges in load so they can no longer cascade.
* Added monitoring and alerting tuned to detect this failure pattern well before customer impact.
Posted Jul 09, 2026 - 10:00 UTC