Slashdot Mirror
Bank's IT Failure Loses 600,000 Payments
An anonymous reader writes: The Royal Bank of Scotland had an IT glitch last night that prevented some 600,000 payments from reaching the accounts of its customers. This included bill payments, wages, tax credits, and benefits payments. RBS apologized for the delay, and claims to have fixed the underlying problem. They hope to have all the missing payments sorted by the weekend. This isn't the first major IT screwup for RBS; in 2012, the company was fined £56 million after a software upgrade prevented about 6.5 million customers from logging into their accounts.
What madness is this??
When my bank makes a mistake, they blame me and penalize me for the problem, then when I complain, they charge me for investigating their error.
I imagine they can be forgiven for something like this as long as they make it right. So for interest bearing accounts, the interest of those deposits needs to start from when they were supposed to credit the account. For anyone who spent money and incurred an overdraft or the like - no fees should be assessed as long as the late deposit would have covered the amount. If they don't do those things - well then maybe they should get fined again.
It's waiting for the day that a solar storm disrupts grid power world-wide for a few days, and digital money will cease to exist. Although personally i'd bet bitcoin would survive such event.
Since the taxpayer took a large stake in RBS, it has been deliberately run into the ground to lower it's share price ready to be sold back into private ownership. It ends up looking like a windfall for the current chancellor, when in fact it's transferred a big pile of taxpayer's money into private hands while wrecking a financial institution. Banksters.
He must have misplaced a decimal point. He always screws up some mundane detail.
If you post as Anonymous Coward, don't expect a reply.
Never worked for RBS, so I'm not claiming to have any knowledge into their IT dept, but I've seen this trend in IT locally too many times. Execs don't see the need paying top dollar for IT staff and wind up getting people that do "just enough." That's when snafus like this one happen, and it costs the company more in the long run than paying top $$ for good talent.
The angry mob wants to know!
I wonder if some lead IT whizz kid decided it would be really "kool" and look good on his CV to replace all that crufty old cobol running on the cobweb covered mainframe in the corner with this months latest hot language?
I'm exaggerating a bit, but it does seem to me having worked in the dev industry for > 20 years that a lot of the younger devs really don't understand the serious reasons behind the phrase If It Ain't Broke... If you mention it they either think you're being ironic or just too old to "get it". Sadly its them who don't get it but they're too arrogant/naive to realise it.
Ummmmmmmmmmmmmmmmm.........
Oops...?
I experienced the same chicanery with my credit union recently. A system upgrade took everything (offices, ATM's, network transactions, credit card access and online sites) offline for 3 days. On monday the issues still werent resolved. Credit cards didnt work, online transfers failed, networks that normally recognized the payment cards suddenly didnt, and the on-hold wait time ballooned to nearly 40 minutes. My direct deposits stopped working due to the fact that my accounts had, without explanation, been suffixed with an additional integer. No one cared.
No one cared for a very important reason: there are no repercussions. Banks dont fail in america as im sure they dont fail in europe. theyre secured by the FDIC which raids and reassigns bank ownership quickly and quietly to prevent runs because the very concept of a bank operates on the principle of too big to fail. During the 2008 financial crisis hundreds of banks across the country failed, but were quietly never reported and instead gobbled up by larger institutions. most of the staff, management, and customers just changed letterhead. no one lost a job and the old bank fines became a pittance; a frosting the larger banks kindly paid.
so if a run happened after 600,000 payments were bungled here in the states, the bank manager would keep his job and just assume a new name like 'ally bank.'
Good people go to bed earlier.
I'll put money on it that this is IBM's fault / incompetence again, not the bank.
Extremely dated infrastructure and mainframes that handle all those payments / systems, are all IBM. Combine that with their
incompetence and I'm surprised this doesn't happen more often.
ISM messages, MQ, COBOL, Java stacks that just make my brain hurt thinking about.
I have a migraine from thinking about that, thank fuck I don't work at HSBC anymore.
If you ever want the worst environment in the world to work in as a programmer, go work for a big bank.
- Dan
they could have processed the 600,000 transactions to the wrong account
Posting anon, because customer data... Circa 5 years ago I worked in support for a commercial software vendor. A big bank in US used several of our software, in a somewhat customized way. One of those products was for network file transfer, one for data manipulation. They called with an issue of a small percentage of inbound transfers consistently failing and after manually reprocessing them a few minutes later they always succeeded. Long story short - they've customized the transfer product to drop a file on a nfs, and send the path and file name via rest to the manipulation product. When the manipulation product tried to pick a file it was not always there, but always appeared in a few minutes. Long story short - turns out one of the lower admins switched their NFS to asynchronous for performance reasons. Five years later I still use the same bank as my primary (but not as my only)
I installed some transaction processing software at RBS in the late 1990s. Here's hoping they retired it a while back...I haven't heard from these guys in over a decade.
By far the most interesting statement on this comes from Iain Martin at the Daily Telegraph, who says that at the time RBS (a fairly small bank) took over the much-larger and more established NatWest bank:
> His team worked out quickly that the NatWest system was superior to the RBS computer system.... Then they crunched the numbers and
> confirmed that sticking with plan A and migrating NatWest's customers onto RBS's inferior and cheaper to run system would save more money.
http://blogs.telegraph.co.uk/news/iainmartin1/100248741/why-the-rbs-computer-keeps-saying-no/
The rest, as they say, is history...
There are some guys on Stackoverflow right now trying to get a fix.
I bet they used H1B visa holders that created this system that had a failure.
Royal Bank of Screwups.
"Cock Up Your Beaver" does not mean what you think. This sig is intended to clog filters and annoy do-gooders
One of the things about banks is that most of their core transaction processing is done on mainframes. There's tons of stuff layered on top of it to do fancier things, but the day to day moving of funds from account to account is usually batched and run after business hours on a mainframe. One of the reasons for this is the sheer amount of business logic tied up in these systems, the massive transaction volume. and the fact that you can't easily swap out a working process with something untested.
The problem is that all the mainframers are starting to retire, and no one is stepping up to fill the spots. So, basic supply and demand kicks in, and mainframe customers have to start paying more for expertise. And we all know what happens when labor rates go up in IT..... You would think younger people would jump at this opportunity -- an environment with a low volume of change that, while important, has safeguards built in that other commodity x86 based things don't. However, there is a persistent "mainframer == old curmudgeon stuck in a dead end job" mentality, so I can see why people might not want this on their resume.
One other site I read a lot is The Register, and they have an interesting UK take on these bank IT failures, including the recent NatWest and other RBS failures, which were pretty big. Their reports indicated that (surprise, surprise) the bank offshored mainframe support and was having quality issues. The banks in question replaced staff with ages of real-world experience on these systems. In my personal experience, these folks know where all the bodies are buried, the stuff that isn't easily laid out in a runbook for a disinterested third party. Root cause for the NatWest failure was someone not knowing how to safely stop CA's batch processing software and dropping tons of messages they thought were backed up, if I recall correctly.
I wonder what it will take for companies to realize that not every IT task is run of the mill, and having some people who know the entire system on staff is a good thing.
So, I used to write banking software for a living, and that included being in the damage control team when things went south. Which they did. A lot. Which were brought to the technical staff's attention, without fail, every friday at 4 PM - but I digress.
This isn't really a big deal.
This sort of thing happens on almost a daily basis. I'd say that 1 in every 500 banks "loses" a day's transactions each week. From hardware failure to networking problems, to someone entering bad data to simple bugs in the software (usually data-dependent and hard to source), and even an occasional overwrite from a backup. Something breaks.
Once we had an FI lie about their data center setup - it wasn't redundant, offsite, certified, or even a datacenter. They just had some machines running in their basement. Imagine our surprise when they called up wanting us to (remotely) fix their servers which were under 6 feet of water during a flood. ... again, I digress.
The thing about these transactions is they're not really lost. Nothing really goes missing.
See, all financial software is super keen on accounting. I don't mean that in a strictly 'add up the numbers' way, but in an auditing way. There are logs upon logs. Using your ATM card to withdraw $20 probably generates around, oh, I dunno, 30-40 log messages, depending on how it's routed. There's a log of the transmission and response in each node along the way, using a standardized protocol. It's a severe pain in the butt, but even if a system goes down forever, we can regenerate those logs from the other systems. It's a great deal of effort, especially to preserve the sequence order, and it's tedious, but it can be done.
This could easily account for the initial delay in fixing things.
Not only that, all these sorts of systems are very keen on sequence-order processing, so if it's just the case that the end of day processing (clearance) system went down/had a bug/etc and none of the transactions were finalized, then they'll just stack up. Once they get that system up again, it'll start processing them again. It might take longer due to a large backload, but it'll eventually complete.
98% of the time, the only people that notice are companies waiting on a payroll to go out, and most of them are happy enough to accept the financial institution's admission of fault. For a day or two at least. Why this one made the paper, I can't tell you. Probably a customer was savvy enough with social media and decided to burn the bank. I guess they should just be happy the public in general doesn't realize how many problems and how much actual work goes into making banking seem reliable and secure.
"This isn't the first major IT screwup for RBS; in 2012, the company was fined £56 million after a software upgrade prevented about 6.5 million customers from logging into their accounts"
I read somewhere that RBS downsized their UK CA-7 batch processing department and then imported the one man from India to take over. Not being experienced enough he botched an overnight job and in the attempt to roll back a days worth of transactions accidentally rolled it back a months worth of transactions. Soon after he removed all references to RBS from his linkedin profile.
That would have been a different sorry l story altogether and maybe one worth reading.
When you cut corners with the very people responsible for your IT infrastructure, things like this will happen. Treat them well, even hire them directly, things like this tend not to happen.
Unfortunately, the UK is rife with this kind of corner-cutting.
Twitter supports and protects racists - by smearing their critics with the "Hate Speech" label.