A story of systemic meltdown – Part II

On Wed evening, as the chaos at Heathrow was unfoldoing in front of me, the “still calm” part of my brain was thinking about what might have triggered the seeming breakdown of the entire system…

Not that the snow or the severe weather was a surprise…

And not that things were somehow smooth until 7pm before suddenly unravelling…

As early as 11am on that day, flights were being cancelled and rescheduled. Even while this was happening, online departure schedules were blithely showing “Scheduled” for flights later in the day; albeit with a disclaimer: “Please check with your airline before flying” – which was a bit pointless becuase you could not get through to most airlines’ telephone lines (which is why people were online).

I can imagine that the phone lines must be busy – what with tens of flights already cancelled and hundreds (perhaps even thousands) of passengers calling in for more information. I can understand that you cannot order phone lines on demand – much less staff them to deal with “peak demands” such as this.

But this is where a bit of calm thinking could have helped – instead of what my wife calls “unnecessary heroics” – as in “we will run the flights come what may“.

Calm thinking would have suggested that it would be futile to expect the phone lines to cope with the demand.  Calm thinking would have directed people online, switched on additional servers and taken out every bandwidth-hogging application and cute image(s) offline to help pages load faster.

Calm thinking would have suggested a minimalist page that would display flight schedules along side a message asking people to email (not call) if they wanted to re-book or cancel the booking. The email server could have been easily configured to respond with an acknowledgement and a reference number to track subsequently.

Calm thinking would have cancelled all flights when it was clear that things were not looking up instead of letting optimistic (and naive) fools like me to check-in their baggage at 7pm even as other passengers were being asked to de-plane and their bags were being off-loaded.

I can go on and on but the point is draw some lessons from this…

What follows is a modest list of actions that might have helped avoid – or at least mitigate – the consequences of system-wide failure(s) and avoid things going from bad to worse…

• Empower front-line staff to take decisions up to at least two levels above them
• Avoid excessive regulations and consider suspending some of them in such times (e.g. the restriction on staff to handle baggage)
• Avoid un-ncessary heroics
• Keep processes and systems flexible
• Be open and honest in your communication(s)
• Let the staff – in different functions – talk directly to each other without having to go via “The Boss”

…and I am tempted to add: Develop a “jugaad” mentality!

What do others think? Feel free to share your own experiences of being in any such situation (although I sincerely hope you never have to encounter it) and what you think could have been done to avoid a meltdown.

Related Post: A story of systemic meltdown

P.S. Don’t forget the Plan “B” in the event of this blog going down!

Somewhat Related:

Summary of the White House review [pdf file] of the attempted terrorist attack on 25th Dec ’09 which President Obama referred to as “systemic failure”. Do read George Pataki’s take on it.