Slashdot Mirror
Royal Bank of Canada Software Upgrade Goes Awry
Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.
I guess that bad karma is pretty sticky. Even selling their preferred A-1 shares to Baystar didn't save them.
My Canadian friends are screaming bloody murder. I don't blame them.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
As an employee of Interplay, I can say that a 1 or 2 day delay in pay is nothing at all to worry about.
Achille Talon
Hop!
This affects a lot of people - even if you don't bank with Royal Bank but your employer does then you will be affected. The HR manager where I work sent out a bulletin today that should apply to anyone affected by this situation:
----
All financial institution are on line with this issue, mortgage or automatic debit payments, will be honored, should anyone be charged interest , advise your bank,the Royal Bank will refund the charges.
All financial institution will advance cash based on an employee presenting a pay stub, they will not advance the full amount of the pay stub , they will however provide cash for the weekend.
----
Now you know where the old wisdom "Never touch a running system" comes from.
I deposited a money order at an RBC branch on the 2nd of june, they told me it would take 12 hours. it still has not been put in my account, same with my pay from today.
At least I have real reason why my rent is late this month..
had their direct deposits delayed by a day or more."
Wait till your bank holds onto your payroll checks for 2 weeks.
Once a bank of mine made an addition mistake, i wrote a pile of checks that all bounced. The bank acknowledged their mistake, and restored funds in my account, but refused to help out with all the check-bouncing fees.
$25 X 17 Hurray.
do() || do_not();
Geez, I'm showing my age again...
A feeling of having made the same mistake before: Deja Foobar
why they wanted their money back from SCO. ;)
"Never attribute to malice what can be adequately explained by stupidity"
Comment removed based on user account deletion
Sure, I don't earn interest on it, but at least I have some in an emergency
No, Lou, you're in the unstable branch! Gaah!
I recently got an overdraft notice on my bank account, four days after depositing my paycheck, in the branch with a teller. After several days of wrangling with their phone customer service and various managers at the bank I finally found out what had gone wrong: the teller had entered the wrong account number into the computer and someone recieved my money in their account. After several more forms and a couple hours of waiting around in the bank they finally got me my money back, but this was after being without cash or check-card for a week. All this because someone, whos job is to be exact, typoed.
I'm sure if this had been their money they would've gotten it back in less than 7 days, and levied some hefty time and inconvenience charges.
-"Nice jacket, who shot the couch?."
junioradmin@rbchost:/> rm -rf core *
waiting
waiting
thinking...this is taking longer than it should
phone rings.
ctrl-| ctrl-| ctrl-| ctrl-|
Here's a quote from Judi Levita, a Royal Bank media-relations officer, explaining what went wrong:
"I honestly don't know. As I say, I mean, it's one of those tech things."
Source
Life is like a web application. Sometime you need cookies just to get by.
I love that.. any reasonable costs. How about the cost of damage to someone's credit when a payment can't go through... are they going to write a million credit apology notices? Are they going to write paper letters so you can keep a copy when someone calls into question your credit? The credit system is very damaging in these cases and has no easy fix.... I recommend all people go their RBC branch and get a letter explaining why payments were missed. Have them give you as many registered copies as you need for all your creditors affected.
(1st sig) If this were a snappy sig, you'd be reading it right now. (2nd sig) I'm a karma whore. >Insert FUD here
You know, if she's been contacted THREE TIMES this week about payment, then she's at LEAST 2 months overdue. Going to blame the other 2+ months on the computer glitch that started on Monday? Besides, their Visa bills aren't actually DUE until about 7 working days into the month, so if she was up to date, then her payment wouldn't even be due yet.
No, that's just somebody who thinks the world owes them everything taking the opportunity to complain because it might get them something they don't really deserve.
Dark Nexus
"Sanity is calming, but madness is more interesting."
No, this is management's fault. The only time a programmer fails is if something wasn't delivered on time, or they just don't produce, or their stuff doesn't make it past QC. Then fire them.
If bad code makes it into the wild, then somebody signed off on it. Somebody cut corners on testing. Somebody decided deadline is more important than quality. Somebody insisted it had to run the newest Microsoft code.
That somebody is the programmer's boss.
>Heads should roll for this one.
But are the "correct" heads going to roll?
>In cases like this, you should be lucky if you aren't held 100% liable.
If you were suppose to be held liable, do you think anything would change? Were any Professional Engineers held liable for the big blackout last year?
The surprise isn't how often we make bad choices; the surprise is how seldom they defeat us.
So that'd be a no, then ^_~
For the longest time I was living from paycheck to paycheck. Compared to my friends I made some pretty good money for being 20 (30K/Yr) but it didnt seem like much when you were living on your own with rent, insurance, car payments, electric, cable, phone, water, and a girlfriend. Its amazing how things add up. It just so happens that I receive the first paycheck of the month on the same day that rent is due. After paying rent I'm left with about $100. It is also convenient that the second and last paycheck of the month is received on the same day that all of the bills are due. After paying them I'm left with $300. Note I havent mentioned the G/F tax yet but that one is expensive.
Anyway, my tip is, next time you get a bonus, tax returns, some lump sum of money, spend it on next months rent before you can do anything else with it. Trust me on this. If you put it in your savings you can too easily transfer it to checking when you see Wizz-Bang4000 on pricewatch for only $499! I do this every chance I get and it really helps out a lot.
Now if I could only figure out what to do with the SO.
Im dreaming ofa big bndwdth, That can resist the
a beer drinking beaver was found at the mainframe keyboard typing I AM CANADIAN over and over. More details to follow...
presmike
A few short years ago, the Toronto Dominion's entire network went down for a whole day. I happened to be visiting a friend a train ride away, and could not get money out to get home out of any machine. It was frustrating, and it made me wonder why my service charges hadn't gone to redundant connections and machines, but I was able to borrow some cash and get home.
But when it happened a second time, in less than a year, I got a little frustrated and switched to president's choice bank.
My wife uses Royal Bank, and her pay has not gone through yet and it is now more than several days late. I certainly hope they work later than 4pm, monday to friday, to fix this. Some people who were supposed to be paid on the 31st have bounced their rent cheque!
It's makes you wonder what "money" really is, when a software error can make it disappear.
Can you imagine working in that IT department right now. My first reaction when I saw this story was that I felt incredibly sorry for those IT guys and gals. Must be hell over there right now!
I know I always sweat when releasing new software, at least I don't have to worry about effecting the bank accounts of millions of people. That would truly be scary!
We finally reached an agreement with this Nigerean wife of an ex-general that she found $10,000,000 in a hidden vault behind the ex'es private toilet bin and agreed to give me 10% if I provide them with my checking account number.
Darn Canadian Bank, now the whole deal might not go through...
Yam, yam, uga booga, yam, yam, yade, yade, uga booga, yam, yam, yade, yade
Programmer 1: It happened like 3 days ago, eh.
Programmer 2: And welcome to day 3.
Programmer 1: "make install" hosed it.
Programmer 2: Hosed it down, eh, like backbacon at a Bah Mitzvah.
Programmer 1: But it's okay, eh. I got my thinking touque on and the beer and pizza are on the way.
Programmer 2: Yeah, we should have it back up by tomorrow, eh. Only, we're gonna need some more vacation after this.
Programmer 1: And beer, eh.
Programmer 2: Yeah, more beer.
Hot Damn! It's the Soggy Bottom Boys!
Yeah, it was probably written in the US :)
Yes, it was actually an upgrade. (Buddy works in the RBFG office with the fellow that screwed up.) I'm told that it was a single-character typo in the command line. Everything seemed OK until the system went on-line with live data. Then all hell broke loose. And my payroll deposit hasn't shown up yet either.
That acutally fluctuates from a period to another... right this moment, a looney is actually only 75% of a buck, but it has not always been this way, nor will it be forever.
Here's a little history for y'all. In 1862, the loonie and the buck were worth about the same thing. Two years later, in 1864, the loonie was worth 265% of a buck. That's right, people could buy $2.65 of american money for a single canadian dollar. After the civil war, the buck slowly went back up to match the loonie.
The loonie began to lose value compared to the buck at the beginning of WWI, then slowly restabilized after, then lost value again during WWII. In 1945, right after the war, the loonie and the buck regained their equality.
In 1961, both were worth about the same, and in 1972 the loonie was worth about 5% more than the buck.
At the beginning of 2003, the loonie was worth slightly more than 60% of the buck, since then, it got a big boost and almost hit 80% to be at 75% today...
Notice how all those major fluctuations coincides with wars... Civil war, WWI, WWII, Viet-Nam, Irak... Re-elect Bush and see what happens to your buck...
After 3 days without programming, life becomes meaningless
- The Tao of Programming
In my case one CU is an account I've had since I was in my teens but their offices are about 15 miles away, and the other is from my former employer and is two states away. I drop in at the local one sometimes since it's about a mile from my girlfriend's office, but if I'm not headed up that direction I can go to the local credit union about a mile and a half from my house.
fencepost
just a little off
I work for a financial firm. Not a bank, but we talk with a lot of banks.
RBC is probably an IBM mainframe environment. OS/390 or similar OS. They probably have AS/400 and AIX hosts as well for other functions (printing, data transfer, etc.) IBM sells you a whole shop and then you're stuck with it.
The OS is irrelevant, this is most likely an application (transactional database) error. Read the Globe And Mail article.
You call in the vendor (IBM) to do system upgrades and the like. They are VERY careful. If it went wrong, RBC would be laying the blame on them right now, and publicly.
Switching between (physical) systems for a mainframe environment is commonplace enough and would be fully supported by IBM. Parallel Sysplex and all that. One of my vendors did it over a weekend without incident - except they didn't wait until month-end to do it! Doh! That was dumb.
I don't do anything around month-end or quarter-end if I can help it. It's asking for trouble.
I'm pretty sure RBC does their IT in-house. These tend to be large, customized or internally developed systems, so outsourcing would mean almost certain death to the company. Even moving/consolidating operations between different groups in the same company is very painful.
Unless they were trying to
That acutally fluctuates from a period to another... right this moment, a looney is actually only 75% of a buck, but it has not always been this way, nor will it be forever.
Dream on.
The canadian looney is artificially kept "low" by toying with the interest rates index to favor US exports, wich is the buyer of 90% of our exports.
Only once has the canadian dollar was worth more then the US dollar, and it proved disastrous for the economy.
It's quite sad, really: any disagreement with you automatically means that I don't like Americans and any and all arguments can and will result in violence, in this case coupled with a strange assortment of insults, what I can only guess is a swipe at my sexuality, and a comment that shows your profound misunderstanding of geography. I must, however, point out that your message, such as it is, is getting garbled by your obvious mastery of grammar and spelling.
On behalf of the United States, I would like to appologize to anyone who was offended by the moronic poster who decided to insult our Northern neighbor, as well as assume that Canada is smaller then Texas. To the insulting poster - buy a globe, buy a hammer. Study the globe, and smash your keyboard so you can't type anymore.
I mod down so you can mod up. Your welcome.
You're close to the money re: OS390, and it being a bank problem (not vendor). I work for another canadian bank, but have former associates at RBC. Here's what I've heard is happening:
- May 31st = month end. new month end code gets run first time in prod (JCL), there's problems - not sure of name of system, but the general bank ledger for customers is fubared.
- RBC rolls back all changes, so RBC is 1 day behind on Tuesday.
- re-run batch with old code on Wednesday am... unfortunately recovery procedures are flawed/human error, batch is screwed up again.
- now bank is 2-3 days behind...can't process transactions effectively, can't catch up with sequential batches in evenings because there's too much to run.
- RBC departments start running independently based on May 31st data...can't afford to be down more than 2 days. now the roadmap is a mess for recovery, general ledger is still at May 31st state (might be June 1st after a successful batch run last night???)
Apparently about 80-100 IT staff are living at the Skydome hotel in Toronto working 16 hour shifts (16 on, 8 off) to try and get caught up. Everything i've heard suggests that they know they can't get done during the business week...they're relying on 2 days of 24hr downtime on the weekend to reload the batches and get systems back in sync.
Based on my experience in Canadian banking (7 years) plus stories of old timers, this is the worst outage in close to 10 years, maybe the worst in 20 if my rumours prove to be true. I have no direct mainframe experience, though, so take my descripton of the problems with a grain of salt...
John Maynard Keynes: "When the facts change, I change my mind. What do you do?"
This has been widely mis-reported. RBC doesn't make such investments themselves, they act as the broker on behalf of clients, exactly the same as any other brokerage. In the case in question, the client chose to remain anonymous, and the shares in question were purchased by RBC, and then held 'in street name' for assignment to an internal client account. The actual details of who the account holder is, are protected by confidentiality laws, and would only become public information if the client requested certificates of shares issued in thier own name, rather than held by the brokerage in street name on thier behalf. the courts can also order such disclosures, but will only do so if there is a real requirement for said disclosure. In this case, there is no requirement for disclosure.
RBC has recieved a lot of negative exposure in the linux community simply because they have respected privacy laws. They acted as the broker in the transaction, and held the shares on behalf of a client. I'd commend the bank, in the face of a lot of pressure, never once have they released or leaked to the public the name of the client they are acting on behalf of. This is as it should be.
The real question in my mind, what individual/corporation outsourced this transaction to Canada, to take advantage of privacy laws that allowed them to do the entire deal anonymously, with the bank acting as the publicly visible broker of record?