Census 2016: The $10 million online census – What went wrong? – EFTM
Screen Shot 2016-08-09 at 8.33.58 PM

Census 2016: The $10 million online census – What went wrong?

Poor planning is the basic problem

Well, hate to say I told you so, but – this was always going to happen.  It’s as if Click Frenzy didn’t happen in 2012 and the ABS just assumed they had it all under control.  Nope.  Tonight, millions of Australians were ready to follow instructions and complete the census online – but the website failed.

So here’s the thing.  A website is just a bunch of files sitting on a computer.  Any one computer can only really handle so many people accessing it at one time.

As the internet has grown, some great things have been put in place to account for this.  Tonight, we saw how even the best laid plans can come unstuck.

Screen Shot 2016-08-09 at 8.33.58 PM

The real issue for Australians, and the Australian Bureau of Statistics (ABS) is the $469,367.50 spent on “Load Testing” to ensure this very thing didn’t happen.

Through open tenders, the Australian Bureau of Statistics awarded the contract to “Design, development and implementation of eCensus Solution 2016” to IBM.  They built a pretty darn slick solution that created a platform by which all Australians could fill out the 2016 Census – at a cost of $9,606,725.00

Screen Shot 2016-08-09 at 9.11.31 PM

When someone raised the prospect of issues with people connecting, the ABS wisely undertook to tender for services to “Load Test” the site and system.

Load Testing is a process by which a system is tested to see how much traffic it can cope with.

The ABS paid an Aussie company “Revolution IT Pty Ltd” $469,367.50 across three tenders to undertake “Load Testing Services for Census 2016”.  $325,000 of that was for software licences to get the equipment required to do the testing, the rest we can assume was for the teams to actually undertake the testing.

Screen Shot 2016-08-09 at 8.53.09 PM

Screen Shot 2016-08-09 at 8.35.29 PM
Screen Shot 2016-08-09 at 8.35.15 PM

 

Here’s the issue.  We can’t yet “blame” Revolution IT for tonight’s outage.  In fact, they may well have tested it exactly as requested.

The ABS themselves have been quick to respond to concerns of the website’s capacity in recent days:

Screen Shot 2016-08-09 at 9.05.46 PM

 

So, let’s assume for the moment that in fact, 1,000,000 form submissions per hour was exactly what the site was tested for.  The above message from the ABS assumes they expected 500,000 submissions per hour.

Aren’t there 10,000,000 homes in Australia, so probably many more than that when you consider other institutions like hospitals etc?

Yep, 1,000,000 or more completed the census before Tuesday August 9.  Yep, lets say another 2,000,000 completed it during the day on Tuesday.

Let’s also assume another 2,000,000 are doing paper forms.  Heck, lets put another 2,000,000 into the “in the days after Tuesday” category.

That leaves 3,000,000 to complete it on Census night.  Did the ABS really think that number of submissions would be evenly spread over 6 hours?

What a joke.  Of course 7-9pm was going to be peak time.  Of course they were going to get millions of people each hour.  They simply didn’t expect it as the rest of us did.

This is an utter failure of planning and process.

And it reflects poorly on anything being “online” in the future.  Australian’s will be scarred by this experience, forced to be sceptical of any such concept in the future.

What could they have done though?

Spread the load.  Sure there is probably a server based solution that would handle the millions in the short space of time.  God knows what that would have cost.

But if the ABS had actually encouraged people to complete the forms early, or late, then perhaps the load would have been more evenly spread with reasonable and manageable traffic spikes.

Not to be.

An investigation will be required, and likely an assessment of just how accurate and complete the data really is after it’s all said and done.

Screen Shot 2016-08-09 at 7.50.11 PM

 

Categories
Tech

Trevor produces two of the most popular technology podcasts in Australia, Your Tech Life and Two Blokes Talking Tech. He has a weekly radio show on 2UE, as well as appearances across the country and regularly provides Technology Commentary to Channel 9’s Today Show and A Current Affair. Father of three, he is often found down in his Man Cave. Like this post? Buy Trev a drink!
9 Comments on this post.
  • Laughing Too Much
    9 August 2016 at 10:15 pm
    Leave a Reply

    You would think that mathematical statisticians understand such concepts as “peak hours” e.g electrical demand during after work cooking times. I am sorry to have to reveal that my experience in the Australian work place is one of repeated failures and incompetence followed by back slapping and cover ups.

    Stop trying to run and swim fast and jump over high objects and learn about mathematics, technology, physics, science and electronics etc etc.

    Fail.. No Gold Medal here LOL..

  • Lothar
    9 August 2016 at 10:42 pm
    Leave a Reply

    Well ABS, you are an odd Bureau but I must say that you steam a good Census.

  • ZDC
    10 August 2016 at 1:08 am
    Leave a Reply

    In this day and age of stateless web application design and IAAS you would think they could just provision a datacentre or two worth of capacity for a few hours and be done with it.

  • tmlc
    10 August 2016 at 8:28 am
    Leave a Reply

    there are plenty of cloud services that can spin up a LOT of capacity for short peaks. Maybe just not in the IBM cloud??

  • Jason
    10 August 2016 at 11:50 am
    Leave a Reply

    To true. 10 mil households, 2/3 will do on line 6.6 mil at a cost $470mil, lets just say it goes to IBM thats $470,000,000 / 6,600,000 households cost $71.21 per house hold. And remember we are the 1st along with the government to bitch that these are company’s that pay very little to no tax here onshore.
    Shit do it over 2 nights Fri and Sat. Give me a slab of beer or 3 bottles of vino at our local Dan’s and I’ll drop it in person by hand. Save 1/3 say $150,000,000 gets me out of the house and as we are all heading to the bottle shop on one of those nights anyway. I’ve seen the numbers on these nights go thru Dan’s, a walk in the park for them.
    Win Win – no DDOS, keep money on shore and jobs. I’ll drink to that.
    Shit I can’t I still need to do my 2016 Census.
    PS “We can’t yet “blame” Revolution IT for tonight’s outage. In fact, they may well have tested it exactly as requested.”
    At this price point it is their job to advise the ABS about DDOS and the target it was going to be. Not just the loads. Hell if Microsoft and Sony and other net works with 40 mil + users are taken off line for days and unable to keep systems up at times what was ABS thinking, not told or “CHOSE TO OVER LOOK”.

  • Will
    10 August 2016 at 12:46 pm
    Leave a Reply

    Great article. Well reasoned and sensible analysis.

  • Simon
    10 August 2016 at 2:34 pm
    Leave a Reply

    Well written post, voicing what the rest of us were thinking about 1,000,000 submissions per hour.

  • Que
    17 August 2016 at 1:45 pm
    Leave a Reply

    the article doesn’t really say much new… this is not even about being wrong planning for peak traffic.

    They just need to turn the error message to say, ‘Our registration server is currently full, please come back another time.’ instead of that pathetic ‘unable to open page’ rubbish.

    redirecting all capped traffic to a static info page that says something nice. cost 50 cents.

    this is about managing user experience. and obviously, that is too hard.

    • Trevor Long
      17 August 2016 at 4:51 pm
      Leave a Reply

      Again, this is perfect, because you point out the most fundamental simple thing that was overlooked – despite the millions of dollars. Amazing

    Leave a Reply

    *

    *