Slashdot Mirror
Bank's IT Failure Loses 600,000 Payments
An anonymous reader writes: The Royal Bank of Scotland had an IT glitch last night that prevented some 600,000 payments from reaching the accounts of its customers. This included bill payments, wages, tax credits, and benefits payments. RBS apologized for the delay, and claims to have fixed the underlying problem. They hope to have all the missing payments sorted by the weekend. This isn't the first major IT screwup for RBS; in 2012, the company was fined £56 million after a software upgrade prevented about 6.5 million customers from logging into their accounts.
Don't be an ass. Its not in any employees interest including the CEO to run it into the ground when a lot of them are on performance related pay. And its certainly not in the governments interest.
I suggest you put your tin foil hat away. This is just plain old fashioned incompetance.
So, I used to write banking software for a living, and that included being in the damage control team when things went south. Which they did. A lot. Which were brought to the technical staff's attention, without fail, every friday at 4 PM - but I digress.
This isn't really a big deal.
This sort of thing happens on almost a daily basis. I'd say that 1 in every 500 banks "loses" a day's transactions each week. From hardware failure to networking problems, to someone entering bad data to simple bugs in the software (usually data-dependent and hard to source), and even an occasional overwrite from a backup. Something breaks.
Once we had an FI lie about their data center setup - it wasn't redundant, offsite, certified, or even a datacenter. They just had some machines running in their basement. Imagine our surprise when they called up wanting us to (remotely) fix their servers which were under 6 feet of water during a flood. ... again, I digress.
The thing about these transactions is they're not really lost. Nothing really goes missing.
See, all financial software is super keen on accounting. I don't mean that in a strictly 'add up the numbers' way, but in an auditing way. There are logs upon logs. Using your ATM card to withdraw $20 probably generates around, oh, I dunno, 30-40 log messages, depending on how it's routed. There's a log of the transmission and response in each node along the way, using a standardized protocol. It's a severe pain in the butt, but even if a system goes down forever, we can regenerate those logs from the other systems. It's a great deal of effort, especially to preserve the sequence order, and it's tedious, but it can be done.
This could easily account for the initial delay in fixing things.
Not only that, all these sorts of systems are very keen on sequence-order processing, so if it's just the case that the end of day processing (clearance) system went down/had a bug/etc and none of the transactions were finalized, then they'll just stack up. Once they get that system up again, it'll start processing them again. It might take longer due to a large backload, but it'll eventually complete.
98% of the time, the only people that notice are companies waiting on a payroll to go out, and most of them are happy enough to accept the financial institution's admission of fault. For a day or two at least. Why this one made the paper, I can't tell you. Probably a customer was savvy enough with social media and decided to burn the bank. I guess they should just be happy the public in general doesn't realize how many problems and how much actual work goes into making banking seem reliable and secure.