System outage

Incident Report for HyperTrack

Postmortem

Summary

During the incident on Friday, September 5th, attackers gained access to our AWS account using a leaked key that had been inadvertently exposed in a JavaScript build. Within minutes, the attackers created concealed backdoor access resources. AWS and HyperTrack cleanup operations removed them, though a few resources initially went undetected.

On September 10, attackers used the remaining backdoor resources to access the AWS account used by the HyperTrack platform and launched large-scale crypto-mining operations across multiple non-default AWS regions. The attack used compromised third-party AWS accounts unrelated to HyperTrack. This triggered AWS automated abuse detection, which temporarily restricted account access at 11:56 UTC. After investigation and remediation, AWS restored full account functionality by 18:10 UTC.

Importantly, no customer data or platform systems were accessed. The incident was contained to crypto-mining activity.

Timeline

September 10, 2025 (Wednesday)

  • 10:51 UTC - Attacker accessed account resources from third-party compromised AWS accounts.
  • 10:53 UTC - Crypto-mining workloads launched in new regions.
  • 10:57–11:00 UTC - Multiple regions disabled again to hide activity.
  • 11:43–11:44 UTC - Attacker switched between external AWS accounts from a new proxy IP address.
  • 11:56 UTC - AWS issued compute resources abuse alert and started shutdown process.
  • 12:08 UTC - Services were temporarily restricted.
  • 12:10 UTC - HyperTrack engineers started responding to the incident and engaged with the AWS security team.
  • 15:07 UTC - HyperTrack engineers removed all crypto mining resources, including all hidden non-default regions.
  • 18:10 UTC - AWS security team confirmed restrictions lifted and account access restored. HyperTrack platform started receiving events and processing API requests.

Next Steps

We recently detected and contained unauthorized access attempts related to leaked CI/CD credentials. Our investigation confirmed that no customer data was accessed and build pipelines for platform code repositories have been hardened.

In response to the two incidents, we have taken immediate actions to strengthen our security posture in the coming weeks. Since the first incident, we implemented a code deployment freeze to address necessary steps to harden platform environments.

These steps include the following:

  • Improve Incident Containment & Remediation

    • Multiple meetings with AWS security team members for account status and configuration review.
    • Continue to monitor CloudTrail for anomalies with alerting enabled.
  • Continue Strengthening Identity & Access Measures

    • Apply tighter Service Control Policies to limit usage to approved regions.
  • Expanded Security Monitoring

    • Review and update GuardDuty, Inspector, Security Hub, and IAM Access Analyzer configurations across all accounts and regions.
    • Configure additional alerting for high-severity findings and account activity to ensure immediate response.
  • Implement Governance & Ongoing Improvements

    • Aggregate all security findings centrally for faster triage and remediation.
    • Review Cognito and network configurations to ensure no hidden backdoors exist.
    • Evaluate AWS Control Tower and additional protections (WAF, Firewall Manager) for consistent org-wide governance.

Our focus remains on delivering reliable, outstanding value to our customers. We are hardening our platform, strengthening comprehensive monitoring, and implementing industry best practices to prevent the recurrence of incidents.

Thank you for your continued trust and patience.

Posted Sep 12, 2025 - 17:17 UTC

Resolved

This incident has been resolved.
Posted Sep 10, 2025 - 18:15 UTC

Identified

We resolved outstanding issues in the platform and are working with AWS to go through the steps of unlocking our AWS account required to resume platform operations.
Posted Sep 10, 2025 - 15:07 UTC

Update

We are continuing to investigate this issue.
Posted Sep 10, 2025 - 12:09 UTC

Investigating

We are currently investigating this issue.
Posted Sep 10, 2025 - 12:08 UTC
This incident affected: Cloud service, Orders, Nearby, Geofences, Geotags, Dashboard, Ops Dashboard, Order tracking views, and Webhooks.