Platform outage

Summary

On Friday, September 5 at 23:33 UTC (16:33 PDT), one of our AWS accounts was accessed using a set of unauthorized keys. The intrusion was quickly detected, and the compromised keys were revoked within 30 minutes.

During this window, the attackers provisioned a large number of expensive compute resources for cryptocurrency mining, which in turn triggered AWS to lock the account.

Importantly, there was no unauthorized access to platform resources or customer data.

Our investigation determined the root cause: a misconfiguration in our JavaScript build process inadvertently exposed CI/CD environment variables. This led to the leak of a pipeline key, which the attackers exploited.

We immediately cleaned up all malicious resources and engaged with AWS in real time to fully restore account functionality.

Full traffic was restored at September 6, 7:07 UTC (12:07 am PDT)

Timeline

Sep 06 2025 00:13 UTC: The outage started at as AWS locked down account‌

Sep 06 2025 00:18 UTC: AWS alerts HyperTrack engineering

Sep 06 2025 04:00 UTC:

• HyperTrack engineering completed actions to remove over 3000 EC2 instances and associated resources
• HyperTrack engineering reiterates to AWS urgency of reopening full access to the account and turning traffic back on

Sep 06 2025 05:33 UTC:

• AWS confirms case was escalated to Service team to reinstate account.
• HyperTrack explains urgency: HyperTrack SDK used in millions of apps, supporting nurses and essential workers

Sep 06 2025 07:07 UTC:

• AWS turned account back on

Sep 06 2025 10:30 UTC:

• HyperTrack engineers continued working with the production resources and handling scale as mobile devices were coming back online after the outage. HyperTrack SDK caches tracking data and thus core pipeline inbound traffic scaled up by a couple orders of magnitude

Sep 06 2025 17:15 UTC:

• HyperTrack engineers continued investigation to understand the source of the leak. The source of the leak was identified to be a misconfiguration in our JavaScript build process.‌

Next steps

Our team is conducting an in-depth analysis of the incident and is enhancing our security policies and procedures. These updates are designed to reinforce safeguards and ensure the continued safety of customer data.