Slashdot Mirror
Bank's IT Failure Loses 600,000 Payments
An anonymous reader writes: The Royal Bank of Scotland had an IT glitch last night that prevented some 600,000 payments from reaching the accounts of its customers. This included bill payments, wages, tax credits, and benefits payments. RBS apologized for the delay, and claims to have fixed the underlying problem. They hope to have all the missing payments sorted by the weekend. This isn't the first major IT screwup for RBS; in 2012, the company was fined £56 million after a software upgrade prevented about 6.5 million customers from logging into their accounts.
What madness is this??
When my bank makes a mistake, they blame me and penalize me for the problem, then when I complain, they charge me for investigating their error.
Since the taxpayer took a large stake in RBS, it has been deliberately run into the ground to lower it's share price ready to be sold back into private ownership. It ends up looking like a windfall for the current chancellor, when in fact it's transferred a big pile of taxpayer's money into private hands while wrecking a financial institution. Banksters.
He must have misplaced a decimal point. He always screws up some mundane detail.
If you post as Anonymous Coward, don't expect a reply.
Never worked for RBS, so I'm not claiming to have any knowledge into their IT dept, but I've seen this trend in IT locally too many times. Execs don't see the need paying top dollar for IT staff and wind up getting people that do "just enough." That's when snafus like this one happen, and it costs the company more in the long run than paying top $$ for good talent.
I wonder if some lead IT whizz kid decided it would be really "kool" and look good on his CV to replace all that crufty old cobol running on the cobweb covered mainframe in the corner with this months latest hot language?
I'm exaggerating a bit, but it does seem to me having worked in the dev industry for > 20 years that a lot of the younger devs really don't understand the serious reasons behind the phrase If It Ain't Broke... If you mention it they either think you're being ironic or just too old to "get it". Sadly its them who don't get it but they're too arrogant/naive to realise it.
I experienced the same chicanery with my credit union recently. A system upgrade took everything (offices, ATM's, network transactions, credit card access and online sites) offline for 3 days. On monday the issues still werent resolved. Credit cards didnt work, online transfers failed, networks that normally recognized the payment cards suddenly didnt, and the on-hold wait time ballooned to nearly 40 minutes. My direct deposits stopped working due to the fact that my accounts had, without explanation, been suffixed with an additional integer. No one cared.
No one cared for a very important reason: there are no repercussions. Banks dont fail in america as im sure they dont fail in europe. theyre secured by the FDIC which raids and reassigns bank ownership quickly and quietly to prevent runs because the very concept of a bank operates on the principle of too big to fail. During the 2008 financial crisis hundreds of banks across the country failed, but were quietly never reported and instead gobbled up by larger institutions. most of the staff, management, and customers just changed letterhead. no one lost a job and the old bank fines became a pittance; a frosting the larger banks kindly paid.
so if a run happened after 600,000 payments were bungled here in the states, the bank manager would keep his job and just assume a new name like 'ally bank.'
Good people go to bed earlier.
I installed some transaction processing software at RBS in the late 1990s. Here's hoping they retired it a while back...I haven't heard from these guys in over a decade.
By far the most interesting statement on this comes from Iain Martin at the Daily Telegraph, who says that at the time RBS (a fairly small bank) took over the much-larger and more established NatWest bank:
> His team worked out quickly that the NatWest system was superior to the RBS computer system.... Then they crunched the numbers and
> confirmed that sticking with plan A and migrating NatWest's customers onto RBS's inferior and cheaper to run system would save more money.
http://blogs.telegraph.co.uk/news/iainmartin1/100248741/why-the-rbs-computer-keeps-saying-no/
The rest, as they say, is history...
So, I used to write banking software for a living, and that included being in the damage control team when things went south. Which they did. A lot. Which were brought to the technical staff's attention, without fail, every friday at 4 PM - but I digress.
This isn't really a big deal.
This sort of thing happens on almost a daily basis. I'd say that 1 in every 500 banks "loses" a day's transactions each week. From hardware failure to networking problems, to someone entering bad data to simple bugs in the software (usually data-dependent and hard to source), and even an occasional overwrite from a backup. Something breaks.
Once we had an FI lie about their data center setup - it wasn't redundant, offsite, certified, or even a datacenter. They just had some machines running in their basement. Imagine our surprise when they called up wanting us to (remotely) fix their servers which were under 6 feet of water during a flood. ... again, I digress.
The thing about these transactions is they're not really lost. Nothing really goes missing.
See, all financial software is super keen on accounting. I don't mean that in a strictly 'add up the numbers' way, but in an auditing way. There are logs upon logs. Using your ATM card to withdraw $20 probably generates around, oh, I dunno, 30-40 log messages, depending on how it's routed. There's a log of the transmission and response in each node along the way, using a standardized protocol. It's a severe pain in the butt, but even if a system goes down forever, we can regenerate those logs from the other systems. It's a great deal of effort, especially to preserve the sequence order, and it's tedious, but it can be done.
This could easily account for the initial delay in fixing things.
Not only that, all these sorts of systems are very keen on sequence-order processing, so if it's just the case that the end of day processing (clearance) system went down/had a bug/etc and none of the transactions were finalized, then they'll just stack up. Once they get that system up again, it'll start processing them again. It might take longer due to a large backload, but it'll eventually complete.
98% of the time, the only people that notice are companies waiting on a payroll to go out, and most of them are happy enough to accept the financial institution's admission of fault. For a day or two at least. Why this one made the paper, I can't tell you. Probably a customer was savvy enough with social media and decided to burn the bank. I guess they should just be happy the public in general doesn't realize how many problems and how much actual work goes into making banking seem reliable and secure.
Not quite. A large Indian outsourcer won a large part of work to maintain the CA7 schedule. Anyone has worked at a bank knows that the batch schedule is somewhat mystical and was put together by guys who had been at the bank for 20 yrs - replacing them with bright, but fresh out of uni, offshore outsourcers was a large part of the cause of the 2012 outage during an upgrade.
This outage hit the NatWest batches. The NatWest batches are far bigger than the RBS, Ulster(s) and IoM/Courts and are the first to run over and the hardest to rerun before the next batch window. So it looks like to me that a file was blocked causing the backlog (and no one noticed quick enough).