You had a rough day at the office did you. Nope. Nothing compared to the poor chap at Telstra who pushed the wrong button or patched the wrong cable forcing millions of Telstra Mobile customers off their network in an extraordinary three-hour outage today.
From around midday until just after 3pm, Telstra Mobile services went from bad to worse and back again after a problem with a “node” went horribly wrong.
Normally there are backups and backups to backups, but in an “embarrassing Human Error” a Telstra employee patched everyone onto the faulty network instead of the backups and backups of the backups.
Life goes on I guess, but geez there were some upset humans on Twitter today.
That being said, there’s a lot of genuine impact, and if your business or life was impacted in a meaningful way I’d encourage you to reach out to Telstra.
For the rest of you – who couldn’t post an Instagram pic of your lunch or snapchat a silly face – chill out, Telstra is offering a full day of free data this Sunday. Knock yourselves out with the downloads folks.
Here’s the full text of Telstra Operations, Chief Operations Officer Kate McKenzie’s explination and apology to customers:
Earlier today we experienced an outage on our mobile network that impacted voice and data services for some of our customers. I wanted to let you know what occurred, what we did to fix it and what we’re doing for our customers as a result.
First let me say we place the highest priority on the quality of our networks and we’re continually investing to ensure we give our customers the best and most reliable mobile network in the country. That is why we’re incredibly disappointed the outage occurred and deeply sorry for the inconvenience we caused.
Our mobile network is set up with a number of major connection points (what we call nodes) around the country, which our customers connect to. These nodes are the equipment that essentially manage the flow of voice and data traffic across our mobile network. The outage was triggered when one of these nodes experienced a technical fault and was taken offline to fix. This normally wouldn’t impact services as we have processes in place to make sure any customers currently connected to a node are transferred to another node before it is taken offline. Unfortunately on this occasion the right procedures were not followed and this resulted in customers being disconnected and consequent heavy congestion on other nodes as customers attempted to reconnect to the network.
As soon as we identified what had occurred we worked to address the fault and take action to bring customers back online as quickly possible. In doing so we prioritised voice services, and when these were back online we started adding data services. While the outage was short in duration we fully realise the impact it had on our customers, which is why we are offering all of our customers a day of free mobile data this Sunday. Customers don’t need to do anything to receive the free data, it will happen automatically for all of our mobile customers.
I apologise again on behalf of the company and thank everyone for their patience while we restored services.
