Slashdot Mirror
Ask Slashdot: How Transparent Should Companies Be When Operational Technology Failures Happen?
New submitter supernova87a writes: Last week, Southwest Airlines had an epic crash of IT systems across their entire business when "a router failure caused the airlines' systems to crash [...] and all backups failed, causing flight delays and cancellations nationwide and costing the company probably $10 million in lost bookings alone." Huge numbers of passengers, crew, and airplanes were stranded as not only reservations systems, but scheduling, dispatch, and other critical operational systems had to be rebooted over the course of 12 hours. Passenger delays, which directly attributable to this incident, continued to trickle down all the way from Wednesday to Sunday as the airline recovered. Aside from the technical issues of what happened, what should a public-facing company's obligation be to discuss what happened in full detail? Would publicly talking about the sequence of events before and after failure help restore faith in their operations? Perhaps not aiming for Google's level of admirable disclosure (as in this 18-minute cloud computing outage where a full post-mortem was given), should companies aim to discuss more openly what happened and how they recovered from system failures?
The companies understand one thing: profit.
It depends on the volume of business and a variety of factors. For example, I was recently considering the purchase of a new automobile. There was one make which I ended up removing from consideration because their infotainment was not open for me to hack on. I felt like this was important and so I told the salesman why it was important to me and that this single factor resulted in my no longer considering any models from this manufacturer.
In another instance, a specific dealership had two different sales people contact me by phone, essentially competing with each other. I didn't like that so I didn't bother calling back either one. Several days later I received a form inquiry from the general manager (certainly an automated message). I took the time to respond, explaining that I wouldn't be doing business with them because of the poor coordination of their salesmen's activities. If I already talked with one and explained what I needed in a vehicle, why was another going to call me and try to make me go through all that again?
Granted, these are different examples, but I make this small effort in the hopes that it will either improve the situation for the person who comes along after me or for myself the next time. Of course, the larger the organization, the less likely this is to have an effect. I expect that the GM of the dealership with two salesmen could possibly do something based on my feedback. I fully expect nothing to change from the manufacturer of the car with the closed infotainment system. However, if 10,000 customers all told different dealers the same thing or bothered to write to the manufacturer directly, then something might change.
Southwest and other airlines are by necessity very large companies. If you tell a booking agent something it is almost certain no manager will hear of it. But, if you contact the execs directly, perhaps if there is a VP of customer service or an ombudsman, contact that person and let them know that you value openness and that you are specifically avoiding giving them your business because of their lack of it. If they hear this from enough people, the will get the message: we are losing out on business because of our approach to blah blah blah.
So, bottom line: companies should be as transparent as their customers demand. If you, the customer, don't demand then they won't know and won't make any change.
i've been delayed because of weather, engine troubles, etc and i'm still alive and happy.
Many (most?) can accept that things happen but, there are limits. Yet, most can't accept a lack of information and being outright lied to. Air line in general have a very poor public perception of telling the truth about why delays are occurring. Weather and "Acts of God" are one thing, we can check the weather, many can sort out things like "Oh, there's a thunderstorm in Ohio, why is that affecting my flight in Colorado... Oh, my plane in trying to leave Ohio..."
With air lines implementing various cost cutting measure (which rarely seem to improve service or cut out prices) that have meant fewer, fuller flights and less convenience for travelers the system seems wholly unable to deal with interruptions above a certain threshold. In light of all this, I fully understand someone's ire at a lack of transparency from any air line.
Southwest, and others, should (as in be required to) provide details on what went wrong.
the only thing public pressure does is cause the company to spend more money in redundant hardware which mostly sits unused and raises prices
It's likely that the cost of redundant hardware pales in comparison to what the device failure cost them..
I worked IT in the airline industry for over 20 years and that happening does not surprise me.
In many cases the systems are old, the software is not well maintained, and management does not understand how critical it is to the operation of the company. Many airline/aircraft companies have outsourced their IT to Managed Service Providers under the guise that "We are an airline, not an IT company." In doing so management negotiated the contracts, not IT, and the contracts are crap. No clauses for upgrading systems, no clauses for management of software patching, and one such contract, that I have read, guaranteed a 98% uptime. Yes, it really was 98% and not 99.999%.
In almost all cases once IT was outsourced, they not only eliminated their IT department, the added rules that stated they could not hire IT people as it was all outsourced and they had no need of them. The companies I have worked for have haired me with odd titles to avoid such rules.
Redundancy is, in many cases, non-existent. Equipment is aging and starting to fail, and there is no plans or projects in the works to update them. Heck, one company I know of is still running on computers that were purchased in 1995.
When projects are put forward with proper HA, network fail over, SAN, etc. They get cut in cost cutting measures to the point that they are unrecognizable. A great example is an upgrade to an Oracle server that I was working on. The original upgrade plan was to deploy an HA pair with back end SAN on a dual 10g fail over connection. After it was cut it ended up being a single dual proc windows system with internal drives running on a 1g connection. It has already crashed multiple times and each time has brought the company to a standstill.
In this day and age, companies need to realize that they run on IT. If your IT infrastructure fails, your company comes to a halt and you loose money!
The companies understand one thing: profit.
That's not true. Companies and the people that run them understand more than just profit. I defy you to find a single person in a company who cannot comprehend something other than profit. To claim that profit is all they can understand is absurdly untrue. But there is a nugget of truth in what you say. What is true is that companies and some (not all) of those who run them have a strong tendency to focus on profits excessively, particularly short term profits. They do this to the detriment of all else including the long term health of the company sometimes. It's too glib to say that companies only understand profit but it is fair to say that companies tend to focus on it too hard at times and make bad decisions as a result.
A well managed company has to consider things like the health of their community, the well being of their suppliers, the trust of their customers, etc. All these things sooner or later will impact profits so if company focuses excessively on near term profits then in the long term they will likely be worse off and so will all those who depend on the company - customers, suppliers, community, shareholders and employees.
Yes! I think airlines and all companies exposing the public to potential life and death situations should definitely give a post mortem when critical systems fail, regardless of whether they are mechanical or not. However, if your local supermarket had a crash of their inventory management system, would you really care? No you probably would not because you will still be able to pay with cash and take your goods anyway. I think the line should be drawn somewhere near exposure to mortal danger. Therefore every company offering some sort of transportation service should be as transparent as possible and should have near-zero privacy.
"Outsourcing partner" in Bangalore must have screwed up.
On Indian outsourcing, here's a war story. When working with Fokker, the Dutch aerospace company, I was sent to Bangalore to emit a final judgment on an outsourcing firm there. On the second day, needing to go to the toilet, I lost my way in the building. Trying to find the loo, I walked by an empty cubicle (the cubicles had large glass panes in them). On the table lay a blueprint. Being an engineer, I couldn't refrain from looking at it. The name "Areva" was printed all over it, Areva being a French constructor of nuclear power plants. It soon became clear to me that those st***d Indians had left the blueprint of an import safety valve in a current nuclear reactor design, unsupervised, on a table in an empty cubicle, and that anyone could walk in on it. I took a picture with my cell phone and sent it to Areva - after having stood there, for a test, for about 10 minutes. Nobody turned up. Anyways - some high-up security guy there went ballistic; on the phone, he thanked me and explained to me the kind of mayhem that blueprint falling in the wrong hands could have caused. (Needless to say we at Fokker immediately cut ties with that Bangalore company.)
Religous speak to God. Insane are spoken to by God. When all shut up, one can finally hear Shostakovich in peace
You haven't worked in enterprise IT for long, have you? An embarrassment like this will make them flog their existing staff harder, insist on more metrics to measure performance, more boxes on the audit form to tick, more mandatory unpaid overtime. But little chance they'll actually spend more money on the IT cost center.
Only those that are required to by some laws, regulations, or an external body, such as financial or health care institutions. Everybody else cheats on infrastructure and recovery equipment. They figure the odds of 2 or more related apps going down and then combine recovery/fail over systems and equipment into one. The cell phone industry is one of the worst, they barely have enough equipment to handle 50% of their KNOWN customer load and just figure that not everyone is going to try and make a call at the same time. When we were primarily a land line market the governing FCC require them to handle at least 80% before a failure, but as we switched to mobile devices they did away with the requirements for workload almost entirely, arriving at the basic priority restoration system we have now. In the event of a emergency the cell networks are going to fail almost instantly, leaving texting as the only, unreliable alternative.
errr....umm...*whooosh* *whoosh* Is this thing on ?