Royal Bank of Canada Software Upgrade Goes Awry

← Back to Stories (view on slashdot.org)

Royal Bank of Canada Software Upgrade Goes Awry

Posted by michael on Friday June 4, 2004 @04:40AM from the cosmic-justice-for-SCO-investment dept.

Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.

20 of 602 comments (clear)

Sticky karma.. by the_rajah · 2004-06-04 04:41 · Score: 5, Insightful

I guess that bad karma is pretty sticky. Even selling their preferred A-1 shares to Baystar didn't save them.

My Canadian friends are screaming bloody murder. I don't blame them.

"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain

--

"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
That's nothing. by Anonymous Coward · 2004-06-04 04:42 · Score: 5, Funny

As an employee of Interplay, I can say that a 1 or 2 day delay in pay is nothing at all to worry about.
Affects not just RB customers by forgetmenot · 2004-06-04 04:44 · Score: 5, Informative

This affects a lot of people - even if you don't bank with Royal Bank but your employer does then you will be affected. The HR manager where I work sent out a bulletin today that should apply to anyone affected by this situation:

----
All financial institution are on line with this issue, mortgage or automatic debit payments, will be honored, should anyone be charged interest , advise your bank,the Royal Bank will refund the charges.

All financial institution will advance cash based on an employee presenting a pay stub, they will not advance the full amount of the pay stub , they will however provide cash for the weekend.
----
Ah, Nostalgia... by ackthpt · 2004-06-04 04:48 · Score: 5, Insightful

Remember back when companies had Q/A departments and procedures? Wrote test plans and tried various scenarios to make sure the software was idiot and bullet proof? When routine software updates didn't suddenly pull your pants down and slap a creme pie into your face? When companies didn't just write any old thing and throw it out there for their customers to actually perform the test?
Geez, I'm showing my age again...

--

A feeling of having made the same mistake before: Deja Foobar
Re:Same as British Air Traffic Control? by MoonBuggy · 2004-06-04 04:49 · Score: 5, Insightful

"Never attribute to malice what can be adequately explained by stupidity"
Comment removed by account_deleted · 2004-06-04 04:49 · Score: 5, Funny

Comment removed based on user account deletion
This is why by Prince+Vegeta+SSJ4 · 2004-06-04 04:50 · Score: 5, Funny

I don't keep all of my money in a bank, I've never had the hole in my back yard, under the oak, next to the stream, 5 steps from the bush refuse to give me a withdrawal. Well, except for that one time the stream flooded.
Sure, I don't earn interest on it, but at least I have some in an emergency
1. Re:This is why by MooseByte · 2004-06-04 05:04 · Score: 5, Funny
  
  "I don't keep all of my money in a bank, I've never had the hole in my back yard, under the oak, next to the stream, 5 steps from the bush refuse to give me a withdrawal."
  Is that the blackberry bush or the raspberry bush? Never mind, I think I found it.
apt-get install bank-upgrade by Anonymous Coward · 2004-06-04 04:51 · Score: 5, Funny

No, Lou, you're in the unstable branch! Gaah!
I can see it now... by j0eshm0e · 2004-06-04 04:54 · Score: 5, Funny

junioradmin@rbchost:/> rm -rf core *

waiting

waiting

thinking...this is taking longer than it should

phone rings.

ctrl-| ctrl-| ctrl-| ctrl-|
Credit damage by elrick_the_brave · 2004-06-04 04:57 · Score: 5, Interesting

I love that.. any reasonable costs. How about the cost of damage to someone's credit when a payment can't go through... are they going to write a million credit apology notices? Are they going to write paper letters so you can keep a copy when someone calls into question your credit? The credit system is very damaging in these cases and has no easy fix.... I recommend all people go their RBC branch and get a letter explaining why payments were missed. Have them give you as many registered copies as you need for all your creditors affected.

--
(1st sig) If this were a snappy sig, you'd be reading it right now. (2nd sig) I'm a karma whore. >Insert FUD here
Re:Somebody should get fired by GoofyBoy · 2004-06-04 05:02 · Score: 5, Insightful

>Heads should roll for this one.

But are the "correct" heads going to roll?

>In cases like this, you should be lucky if you aren't held 100% liable.

If you were suppose to be held liable, do you think anything would change? Were any Professional Engineers held liable for the big blackout last year?

--
The surprise isn't how often we make bad choices; the surprise is how seldom they defeat us.
this just in..... by presmike · 2004-06-04 05:08 · Score: 5, Funny

a beer drinking beaver was found at the mainframe keyboard typing I AM CANADIAN over and over. More details to follow...

--
presmike
Those poor members of the IT department by pbailey · 2004-06-04 05:11 · Score: 5, Insightful

Can you imagine working in that IT department right now. My first reaction when I saw this story was that I felt incredibly sorry for those IT guys and gals. Must be hell over there right now!

I know I always sweat when releasing new software, at least I don't have to worry about effecting the bank accounts of millions of people. That would truly be scary!
Darn... by presarioD · 2004-06-04 05:28 · Score: 5, Funny

We finally reached an agreement with this Nigerean wife of an ex-general that she found $10,000,000 in a hidden vault behind the ex'es private toilet bin and agreed to give me 10% if I provide them with my checking account number.

Darn Canadian Bank, now the whole deal might not go through...

--
Yam, yam, uga booga, yam, yam, yade, yade, uga booga, yam, yam, yade, yade
Re:Eh by delcielo · 2004-06-04 05:31 · Score: 5, Funny

Programmer 1: It happened like 3 days ago, eh.

Programmer 2: And welcome to day 3.

Programmer 1: "make install" hosed it.

Programmer 2: Hosed it down, eh, like backbacon at a Bah Mitzvah.

Programmer 1: But it's okay, eh. I got my thinking touque on and the beer and pizza are on the way.

Programmer 2: Yeah, we should have it back up by tomorrow, eh. Only, we're gonna need some more vacation after this.

Programmer 1: And beer, eh.

Programmer 2: Yeah, more beer.

--
Hot Damn! It's the Soggy Bottom Boys!
Re:I wonder... by tool462 · 2004-06-04 05:33 · Score: 5, Insightful

Yeah, it was probably written in the US :)
Re:What sytems, what upgrade? by skateboard · 2004-06-04 05:39 · Score: 5, Informative

Yes, it was actually an upgrade. (Buddy works in the RBFG office with the fellow that screwed up.) I'm told that it was a single-character typo in the command line. Everything seemed OK until the system went on-line with live data. Then all hell broke loose. And my payroll deposit hasn't shown up yet either.
Re:Big Questions by Anonymous Coward · 2004-06-04 06:01 · Score: 5, Interesting

I work for a financial firm. Not a bank, but we talk with a lot of banks.

RBC is probably an IBM mainframe environment. OS/390 or similar OS. They probably have AS/400 and AIX hosts as well for other functions (printing, data transfer, etc.) IBM sells you a whole shop and then you're stuck with it.

The OS is irrelevant, this is most likely an application (transactional database) error. Read the Globe And Mail article.

You call in the vendor (IBM) to do system upgrades and the like. They are VERY careful. If it went wrong, RBC would be laying the blame on them right now, and publicly.

Switching between (physical) systems for a mainframe environment is commonplace enough and would be fully supported by IBM. Parallel Sysplex and all that. One of my vendors did it over a weekend without incident - except they didn't wait until month-end to do it! Doh! That was dumb.

I don't do anything around month-end or quarter-end if I can help it. It's asking for trouble.

I'm pretty sure RBC does their IT in-house. These tend to be large, customized or internally developed systems, so outsourcing would mean almost certain death to the company. Even moving/consolidating operations between different groups in the same company is very painful.

Unless they were trying to
Re:Wait a minute... by GordoTheGeek · 2004-06-04 06:59 · Score: 5, Insightful

Stop! My Sides! You're killing me! Both you and the other moron are letting your ignorance show. You have quite succinctly demonstrated exactly what it is that I and a very large portion of the world do not like about what, unfortunately, passes for representative Americans. Bravo
It's quite sad, really: any disagreement with you automatically means that I don't like Americans and any and all arguments can and will result in violence, in this case coupled with a strange assortment of insults, what I can only guess is a swipe at my sexuality, and a comment that shows your profound misunderstanding of geography. I must, however, point out that your message, such as it is, is getting garbled by your obvious mastery of grammar and spelling.