Faulty Patch Freezes Millions of UK Bank Accounts

frisket writes with news from The Register about ongoing problems for some UK banks: "'RBS and Natwest have failed to register inbound payments for up to three days, customers have reported, leaving people unable to pay for bills, travel and even food. The banks — both owned by RBS Group — have confirmed that technical glitches have left bank accounts displaying the wrong balances and certain services unavailable. There is no fix date available.' Customers of NatWest subsidiary Ulster Bank in Ireland have also been left without banking services. RTE reports that 'the problem had arisen within the systems of parent bank RBOS when an incorrect patch was applied.'"

Comments

Annoyed customer

[ralphius]

As a customer, I'm annoyed that a) A major high street bank doesn't have enough failsafes/testing to prevent this and b) That there is so little communication as to the cause and expected time to fix the problem.
Thankfully I don't live week to week off my wage like some people do, but if I did I'd be having major problems as evidenced by some of the BBC stories.

Re:test labs

[networkBoy]

For our critical stuff we hav 4 layers to get to production:
1) I do Foo and test it on my dev machine on an unsigned system
2) I submit my Foo to the build system, it builds it for unsigned systems and it is tested by our QRE/Validation department
3) Once things look good, it is signed, then deployed to our Validation dept to run on signed systems.
4) If it is still looking good, then it is deployed.

And we're not even a banking related operation...
-nB

consequences

[sribe]

Well, I don't know if it's still the case, but when I worked in banking IT in the late 80s here in the US there was a standing rule: if you don't process checks for more than 24 hours, you can be taken over by the Federal Reserve--where that takeover implies the possibility of being shut down and your assets distributed to other banks.

That really kept the fear of god in management with regard to keeping core IT running, backups, disaster recovery, etc. Daily offsite backups, periodically loading the backups at a backup facility and running test loads...

There should still be such a rule, and it should apply to electronic transactions as well as checks (not much difference anymore anyway), and the UK ought to adopt it. If a bank takes down its main system with a fucked-up patch, and can't get its disaster recovery plan working in 24 hours, shut it down.

Re:consequences

[Rich0]

I'd go a step further and limit the size of banks to something the government can reasonably deal with should things go south.

If the biggest banks had tens or hundreds of millions in deposits it wouldn't be such a big deal when one of them has an issue like this. Just have the government step in and take care of the people affected, and then fine the living daylights out of whoever messed up once it is all sorted out.

The problem is that these banks are getting so large that you can fit those responsible for 80% of the world's wealth into a single room. That means that they talk to each other, adopt similar policies, and so on. We have far more systemic risk, and even when you get a random failure like this the impact is enormous.