Slashdot Mirror
Faulty Patch Freezes Millions of UK Bank Accounts
frisket writes with news from The Register about ongoing problems for some UK banks: "'RBS and Natwest have failed to register inbound payments for up to three days, customers have reported, leaving people unable to pay for bills, travel and even food. The banks — both owned by RBS Group — have confirmed that technical glitches have left bank accounts displaying the wrong balances and certain services unavailable. There is no fix date available.' Customers of NatWest subsidiary Ulster Bank in Ireland have also been left without banking services. RTE reports that 'the problem had arisen within the systems of parent bank RBOS when an incorrect patch was applied.'"
Where are those cleanmypc.com ads when you need them?
... getting rid of all the expensive people with experience in the mainframe backend system...
I always mess up some mundane detail.
They're doing tests on locking down funds transfer & electronic payments systems. This is probably much harder than you'd think, because they're designed to, well, just work. A few weeks ago (5th June, to be precise) a similar thing happened in Belgium. Caused chaos on the railways.
If anyone thinks it isn't a rehearsal for when Greece drops out of the Euro then I've got a nice bridge for sale.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
I hate it when I can't food. We all need to food sometime.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
To prevent another run on UK banks.
A feeling of having made the same mistake before: Deja Foobar
Always have one. Test it in your testing environment and if it works correctly, deploy on live servers. server idiots guide 101 available in all stores right now
As a customer, I'm annoyed that a) A major high street bank doesn't have enough failsafes/testing to prevent this and b) That there is so little communication as to the cause and expected time to fix the problem.
Thankfully I don't live week to week off my wage like some people do, but if I did I'd be having major problems as evidenced by some of the BBC stories.
When you remove a 1000 members of IT staff [many of which were probably your best] and replace them with 500 offshore workers combined with the need to support *legacy* systems, you are asking for trouble.
http://www.computerweekly.com/news/1280093677/Royal-Bank-of-Scotland-cuts-1000-IT-jobs
Regardless of the technical problems, the root cause of this seems to be management......
The summary says there's no fix date available, and I know that's what it says in the Register article, but the second article, linking to the bank's site, has this to say:
The bank says the issue has now been fixed but it will take the weekend to clear the backlog which amounts to millions of euro in transactions.
It would appear that an update to CA-7 resulted in the actual schedule for these being corrupted or deleted. Therefore they do not know how much any customer actually has in their account, since accounts were not updated with transactions from the previous day.
The problem now appears to be fixed (read: update backed out and control datasets restored), but they still have to run through three days of unprocessed transactions, so people are not getting money paid in during these three days into their accounts as expected, resulting in misery.
This is something which should have been detected and fixed in a competent mainframe site very quickly indeed, so I imagine that the wisdom of outsourcing any "back-office" function of this nature is shortly going to be a matter of very close scrutiny.
Hope this helps.
Actually, it is the fault of management for creating a lousy work environment.
Sometimes there's a limit to what you can test. Simulating a multilevel network with very broad trees, as a big bank probably uses, probably makes do with a significantly simplified environment. All it takes is some yutz somewhere to try to freeze an account that's already frozen (or do something else obtuse that's very hard to predict) to trigger an untested path bug and BOOM.
Being able to roll back a patch becomes a very useful feature at that point. But rolling back large amounts of financial transactions is a whole 'nother big can of worms.
Sometimes they just have to turn things on they shouldn't, to get the water flowing again, and try to clean up as best they can next weekend while they pour over the logs trying to catch things that got rolled back wrong etc. My bank had this happen once in their recent upgrades that I've caught, and it turned out to be a "bank error in your favor, collect $130". (they auto-paid a bill of mine and never debited my account) But that's the cheaper price they pay for screwing up. The alternative would be to let everyone know that they screwed up the books and are going to be fixing things for the next week, and that drops peoples' confidence in their bank which is incredibly bad for business at a bank, people really flip out when their bank tells them they don't quite know where all their money is right at the moment. So they just take a deep breath and take some little hits here and there to avoid much worse PR damage.
I work for the Department of Redundancy Department.
Well, I don't know if it's still the case, but when I worked in banking IT in the late 80s here in the US there was a standing rule: if you don't process checks for more than 24 hours, you can be taken over by the Federal Reserve--where that takeover implies the possibility of being shut down and your assets distributed to other banks.
That really kept the fear of god in management with regard to keeping core IT running, backups, disaster recovery, etc. Daily offsite backups, periodically loading the backups at a backup facility and running test loads...
There should still be such a rule, and it should apply to electronic transactions as well as checks (not much difference anymore anyway), and the UK ought to adopt it. If a bank takes down its main system with a fucked-up patch, and can't get its disaster recovery plan working in 24 hours, shut it down.
That's not a bug for the bank, that's a feature!
These are absolutely core business systems; making decisions which turn out to compromise their smooth running is unforgivable. If I banked with any bank in the RBS group, I'd be out of there ASAP if this indicative of how they run their operations.
lest we forget, a huge number of the subprime mortgage securites (CDOs) had the "RBS" name on them. RBS only exists because the taxpayers kept it afloat. it was 'too big to fail'.
the fact that it would fail again in some catastrophic way is not surprising. it is like watching a hippopotamous shit all over the floor and then scratching your head and asking "why did the hippo shit all over the floor?". because thats what hippos do. they shit on the floor. if you want to stop having to wipe hippo shit out of the floor by the truckload, you might want to just get the hippo out of the room instead of trying to figure out how to get it to stop shitting.