Slashdot Mirror
Faulty Patch Freezes Millions of UK Bank Accounts
frisket writes with news from The Register about ongoing problems for some UK banks: "'RBS and Natwest have failed to register inbound payments for up to three days, customers have reported, leaving people unable to pay for bills, travel and even food. The banks — both owned by RBS Group — have confirmed that technical glitches have left bank accounts displaying the wrong balances and certain services unavailable. There is no fix date available.' Customers of NatWest subsidiary Ulster Bank in Ireland have also been left without banking services. RTE reports that 'the problem had arisen within the systems of parent bank RBOS when an incorrect patch was applied.'"
Where are those cleanmypc.com ads when you need them?
... getting rid of all the expensive people with experience in the mainframe backend system...
I always mess up some mundane detail.
I hate it when I can't food. We all need to food sometime.
The simple truth is that interstellar distances will not fit into the human imagination
- Douglas Adams
To prevent another run on UK banks.
A feeling of having made the same mistake before: Deja Foobar
As a customer, I'm annoyed that a) A major high street bank doesn't have enough failsafes/testing to prevent this and b) That there is so little communication as to the cause and expected time to fix the problem.
Thankfully I don't live week to week off my wage like some people do, but if I did I'd be having major problems as evidenced by some of the BBC stories.
When you remove a 1000 members of IT staff [many of which were probably your best] and replace them with 500 offshore workers combined with the need to support *legacy* systems, you are asking for trouble.
http://www.computerweekly.com/news/1280093677/Royal-Bank-of-Scotland-cuts-1000-IT-jobs
Regardless of the technical problems, the root cause of this seems to be management......
For our critical stuff we hav 4 layers to get to production:
1) I do Foo and test it on my dev machine on an unsigned system
2) I submit my Foo to the build system, it builds it for unsigned systems and it is tested by our QRE/Validation department
3) Once things look good, it is signed, then deployed to our Validation dept to run on signed systems.
4) If it is still looking good, then it is deployed.
And we're not even a banking related operation...
-nB
whois gawk date unzip strip find touch finger mount join nice man top fsck grep eject more yes exit umount sleep dump
The summary says there's no fix date available, and I know that's what it says in the Register article, but the second article, linking to the bank's site, has this to say:
The bank says the issue has now been fixed but it will take the weekend to clear the backlog which amounts to millions of euro in transactions.
A few weeks ago (5th June, to be precise) a similar thing happened in Belgium. Caused chaos on the railways.
What, SNCB suddenly had trains leave on time?
It would appear that an update to CA-7 resulted in the actual schedule for these being corrupted or deleted. Therefore they do not know how much any customer actually has in their account, since accounts were not updated with transactions from the previous day.
The problem now appears to be fixed (read: update backed out and control datasets restored), but they still have to run through three days of unprocessed transactions, so people are not getting money paid in during these three days into their accounts as expected, resulting in misery.
This is something which should have been detected and fixed in a competent mainframe site very quickly indeed, so I imagine that the wisdom of outsourcing any "back-office" function of this nature is shortly going to be a matter of very close scrutiny.
Hope this helps.
I've got a nice bridge for sale.
ahh, so you're the guy who bought it! Big of you to own up.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
Well, I don't know if it's still the case, but when I worked in banking IT in the late 80s here in the US there was a standing rule: if you don't process checks for more than 24 hours, you can be taken over by the Federal Reserve--where that takeover implies the possibility of being shut down and your assets distributed to other banks.
That really kept the fear of god in management with regard to keeping core IT running, backups, disaster recovery, etc. Daily offsite backups, periodically loading the backups at a backup facility and running test loads...
There should still be such a rule, and it should apply to electronic transactions as well as checks (not much difference anymore anyway), and the UK ought to adopt it. If a bank takes down its main system with a fucked-up patch, and can't get its disaster recovery plan working in 24 hours, shut it down.
If anyone thinks it isn't a rehearsal for when Greece drops out of the Euro then I've got a nice bridge for sale.
I'd take you up on that offer except my bank account seems to be locked. However, I have a good friend in a Central African state that has access to many large bank's internal systems. He has generously offered to transfer the funds through his contacts. All he needs is your banking account number and routing data and he will gladly oblige us both.
Simply reply to this communicationl with the appropriate information.
Thanking you in advance.
Faster! Faster! Faster would be better!
Ah, but they've outsource their IT staff to save money, so they probably got rid of the test lab for the same reason. Never underestimate the stupidity of a group of executives looking for a short-term cost saving.
"Think about how stupid the average person is. Now, realise that half of them are dumber than that." - George Carlin
I honestly don't think you're right. I've worked in enough "brack & mortar" companies that have slowly over time transferred all of their operations and value to IT so that nobody in the company outside of IT clearly knows how the system is supposed to be working.
This would all be well and good if only the management - the very people that ordered the transfer - were aware of it. But no, they still see the value in all the little people doing nothing in their offices. After all, IT is a cost and doesn't bring money in ! Need to cut costs? Lay off those geeks. They serve no purpose.
And all of a sudden, you've lost the only people that keep the entire company afloat.
Because what these people need to realise is that the ONLY people needed in the company are the IT. You can layoff ALL OF THE REST OF THEM and the company will keep on running. Badly, not as efficient, maybe at 10% of its capacity. But the IT people can take over as a cashier, delivery boy, salesmen, etc. Those jobs are complex, if you want to be efficient at them. But anybody can do them badly.
Lay off ALL OF IT. You company dies right there. End of the story. Because nobody else in the company can understand the first thing about IT. They wouldn't even know how to log in the production servers.
They outsourcedtheir IT? Equivalent of selling off the company. They outsourced to some random dudes in some place they cannot even reach? They killed themselves.
Write boring code, not shiny code!