Skip to content
Hitrak logo-1
 

August 1st, 2024 Microsoft and 2Degrees Outage

Incident Details

INCIDENT DATE:   1 August 2024
INCIDENT TIME:   8:24 am 
INCIDENT DURATION:  Approximately 90 minutes
SEVERITY LEVEL:   Level 1 outage for affected customer
 
Incident Description
At approximately 8:24am on 1 August 2024, New Zealand 2 Degrees Broadband and Mobile customers lost connectivity to multiple Microsoft online services following an earlier Microsoft Azure outage.

This loss of connectivity resulted in Hitrak customers on the 2Degrees network being unable to access Hitrak online applications.

The Hitrak suite of applications remained fully operational for all customers on other networks, this incident was limited to 2Degrees customers. 

Following a configuration change at 2Degrees, connectivity was restored by 9.53am.

During the outage, the Hitrak team provided electronic copies of run manifests to affected customers to minimise disruption by allowing drivers to continue delivering freight.

Systems and Data Affected
All Hitrak applications were temporarily unavailable for affected clients.
No application data was compromised.

Root Cause Analysis
This incident was caused by a wider disruption to Microsoft services for 2Degrees customers.
Hitrak internal root cause analysis was not required.

Lessons Learnt
This issue was not a Hitrak outage, it was a (limited) interruption of connectivity to the Hitrak system for 2Degrees customers.  
The Hitrak system remained fully operational, and the Hitrak team were to relay information and documents as necessary to assist affected businesses to continue operating.   

Lesson 1:  Implement independent storage for disaster recovery purposes 
In order to provide additional disaster recovery capabilities, and limit exposure to an individual cloud provider, Hitrak has implemented a process by which key operational documents are created and stored in an independent cloud environment.
In the event of a connectivity issue such as this one, or in the unlikely event of a full outage, the Hitrak team will distribute the files necessary for customers to continue operating. 

 
Lesson 2: Extend Offline Capabilities
The current offline capability of the Driver application is designed to deal with intermittent network disruptions, Hitrak is now extending this capability to better support longer periods without application connectivity. 
This will allow the driver to continue using the Driver App to deliver whilst offline (with some caveats).
 
Having Issues?