Royal Bank of Canada Software Upgrade Goes Awry
Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.
They actually pay Canadians?... With money?
I guess that bad karma is pretty sticky. Even selling their preferred A-1 shares to Baystar didn't save them.
My Canadian friends are screaming bloody murder. I don't blame them.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
As an employee of Interplay, I can say that a 1 or 2 day delay in pay is nothing at all to worry about.
This affects a lot of people - even if you don't bank with Royal Bank but your employer does then you will be affected. The HR manager where I work sent out a bulletin today that should apply to anyone affected by this situation:
----
All financial institution are on line with this issue, mortgage or automatic debit payments, will be honored, should anyone be charged interest , advise your bank,the Royal Bank will refund the charges.
All financial institution will advance cash based on an employee presenting a pay stub, they will not advance the full amount of the pay stub , they will however provide cash for the weekend.
----
Now you know where the old wisdom "Never touch a running system" comes from.
I deposited a money order at an RBC branch on the 2nd of june, they told me it would take 12 hours. it still has not been put in my account, same with my pay from today.
At least I have real reason why my rent is late this month..
had their direct deposits delayed by a day or more."
Wait till your bank holds onto your payroll checks for 2 weeks.
Once a bank of mine made an addition mistake, i wrote a pile of checks that all bounced. The bank acknowledged their mistake, and restored funds in my account, but refused to help out with all the check-bouncing fees.
$25 X 17 Hurray.
do() || do_not();
Geez, I'm showing my age again...
A feeling of having made the same mistake before: Deja Foobar
why they wanted their money back from SCO. ;)
I've heard a lot about this here in Canada over the past few days. Does anybody know what systems RBC was using, and what upgrade they were doing? It definitely seems suspicious that they were doing an upgrade at the *end* of the month (May), which is the busiest time for a bank (I know, from having worked at one). Was this really an upgrade gone wrong? Are there any more technical details?
I visited RBC earlier this year to make changes to my retirement plan and I was shocked to see that the account manager used a single PC to manage the accounts and access the internet. When I pointed that out, he said "don't worry, we run the best anti-virus software there is" (McAfee by the look of the icon in the tray). Because, as we all know, it's those viruses that eventually steal passwords and break into the databases. *rollseyes*
"Never attribute to malice what can be adequately explained by stupidity"
Comment removed based on user account deletion
I am a Royal Bank customer too... fortunatly my company uses CIBC, so I went down to the bank on my way to work this morning with my paystub and left with my pay, and all my funds from my account. I closed it and gave my financial buddy at work a new account with CIBC.
Honestly though. Being a software developer and knowing the development cycle like the back of my computer leads me to wonder how in the world they didnt test it fully. I mean... comeon guys. And that kind of institution using SCO's brand of UNIX? face + palm
Oh well... i dont care anymore... i close the accounts and visa card and when they asked me why, I just said: "I can't trust a bank that can't deal with this kind of glitch."
while(1) { fork(); };
Sure, I don't earn interest on it, but at least I have some in an emergency
No, Lou, you're in the unstable branch! Gaah!
I just hope their programmers aren't unionized.
I hope their programmers ARE unionized. If not, you can bet who the scapegoats will be, regardless of whether they are actually to blame.
It's simple: I demand prosecution for torture.
I recently got an overdraft notice on my bank account, four days after depositing my paycheck, in the branch with a teller. After several days of wrangling with their phone customer service and various managers at the bank I finally found out what had gone wrong: the teller had entered the wrong account number into the computer and someone recieved my money in their account. After several more forms and a couple hours of waiting around in the bank they finally got me my money back, but this was after being without cash or check-card for a week. All this because someone, whos job is to be exact, typoed.
I'm sure if this had been their money they would've gotten it back in less than 7 days, and levied some hefty time and inconvenience charges.
-"Nice jacket, who shot the couch?."
junioradmin@rbchost:/> rm -rf core *
waiting
waiting
thinking...this is taking longer than it should
phone rings.
ctrl-| ctrl-| ctrl-| ctrl-|
I love that.. any reasonable costs. How about the cost of damage to someone's credit when a payment can't go through... are they going to write a million credit apology notices? Are they going to write paper letters so you can keep a copy when someone calls into question your credit? The credit system is very damaging in these cases and has no easy fix.... I recommend all people go their RBC branch and get a letter explaining why payments were missed. Have them give you as many registered copies as you need for all your creditors affected.
(1st sig) If this were a snappy sig, you'd be reading it right now. (2nd sig) I'm a karma whore. >Insert FUD here
You know, if she's been contacted THREE TIMES this week about payment, then she's at LEAST 2 months overdue. Going to blame the other 2+ months on the computer glitch that started on Monday? Besides, their Visa bills aren't actually DUE until about 7 working days into the month, so if she was up to date, then her payment wouldn't even be due yet.
No, that's just somebody who thinks the world owes them everything taking the opportunity to complain because it might get them something they don't really deserve.
Dark Nexus
"Sanity is calming, but madness is more interesting."
i bank at RBC, and i have no money in my account right now. and it's my girlfriend's birthday. and she hates the present i got her. and my dog got run over by the bank manager... if i had a dog he would have anyways. it turns out that _some_ of the transactions from the weekend actually did go through, like the ones from my account, so when they reapplied everything yesterday i got double-debited for everything and it emptied my account. whee. fun side note: if you walk into a Royal Bank branch, you'll notice that the terminals behind the counter are running 16bit windows apps. check it out frank, we got this great new version, it's For Workgroups!
Nice planning: an end-of-month upgrade at a financial institution when, by their own admission, transactions are at their peak.
Maybe they thought they would broaden their QA testing base to, say 20,000,000.
Have you Meta Moderated t
Well, whoever signed off on the code and said "it's ready to go" are the ones who fucked up. I mean I suppose you could have a situation where the actual production environment was vastly different from the development/testing one, but I find that doubtful.
Really, there's all kinds of blame to go around, and programmers deserve some of it, the system never should have been so brittle as to cause these kinds of problems in the first place.
autopr0n is like, down and stuff.
No, this is management's fault. The only time a programmer fails is if something wasn't delivered on time, or they just don't produce, or their stuff doesn't make it past QC. Then fire them.
If bad code makes it into the wild, then somebody signed off on it. Somebody cut corners on testing. Somebody decided deadline is more important than quality. Somebody insisted it had to run the newest Microsoft code.
That somebody is the programmer's boss.
>Heads should roll for this one.
But are the "correct" heads going to roll?
>In cases like this, you should be lucky if you aren't held 100% liable.
If you were suppose to be held liable, do you think anything would change? Were any Professional Engineers held liable for the big blackout last year?
The surprise isn't how often we make bad choices; the surprise is how seldom they defeat us.
Why post a bug story on /. if we can't blame it on M$?
I totally agree. Most experts recommand that you have three to six months worth of funds saved up, which means that delay of payroll check deposit of a day, or even couple of weeks should be non-issue.
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
For the longest time I was living from paycheck to paycheck. Compared to my friends I made some pretty good money for being 20 (30K/Yr) but it didnt seem like much when you were living on your own with rent, insurance, car payments, electric, cable, phone, water, and a girlfriend. Its amazing how things add up. It just so happens that I receive the first paycheck of the month on the same day that rent is due. After paying rent I'm left with about $100. It is also convenient that the second and last paycheck of the month is received on the same day that all of the bills are due. After paying them I'm left with $300. Note I havent mentioned the G/F tax yet but that one is expensive.
Anyway, my tip is, next time you get a bonus, tax returns, some lump sum of money, spend it on next months rent before you can do anything else with it. Trust me on this. If you put it in your savings you can too easily transfer it to checking when you see Wizz-Bang4000 on pricewatch for only $499! I do this every chance I get and it really helps out a lot.
Now if I could only figure out what to do with the SO.
Im dreaming ofa big bndwdth, That can resist the
a beer drinking beaver was found at the mainframe keyboard typing I AM CANADIAN over and over. More details to follow...
presmike
A few short years ago, the Toronto Dominion's entire network went down for a whole day. I happened to be visiting a friend a train ride away, and could not get money out to get home out of any machine. It was frustrating, and it made me wonder why my service charges hadn't gone to redundant connections and machines, but I was able to borrow some cash and get home.
But when it happened a second time, in less than a year, I got a little frustrated and switched to president's choice bank.
My wife uses Royal Bank, and her pay has not gone through yet and it is now more than several days late. I certainly hope they work later than 4pm, monday to friday, to fix this. Some people who were supposed to be paid on the 31st have bounced their rent cheque!
It's makes you wonder what "money" really is, when a software error can make it disappear.
Can you imagine working in that IT department right now. My first reaction when I saw this story was that I felt incredibly sorry for those IT guys and gals. Must be hell over there right now!
I know I always sweat when releasing new software, at least I don't have to worry about effecting the bank accounts of millions of people. That would truly be scary!
I just have a few big questions, not that the bank is likely to answer any of them. I suppose the inevitable lawsuits may flush out some of them though.
1. What OS(s) were they running before this happened?
2. Were they really doing an upgrade or a crossgrade, that is, switching to a new system altogether?
3. Was this being handled by in house IT or was it being outsourced half way around the world?
To the making of books there is no end, so let's get started
We finally reached an agreement with this Nigerean wife of an ex-general that she found $10,000,000 in a hidden vault behind the ex'es private toilet bin and agreed to give me 10% if I provide them with my checking account number.
Darn Canadian Bank, now the whole deal might not go through...
Yam, yam, uga booga, yam, yam, yade, yade, uga booga, yam, yam, yade, yade
Programmer 1: It happened like 3 days ago, eh.
Programmer 2: And welcome to day 3.
Programmer 1: "make install" hosed it.
Programmer 2: Hosed it down, eh, like backbacon at a Bah Mitzvah.
Programmer 1: But it's okay, eh. I got my thinking touque on and the beer and pizza are on the way.
Programmer 2: Yeah, we should have it back up by tomorrow, eh. Only, we're gonna need some more vacation after this.
Programmer 1: And beer, eh.
Programmer 2: Yeah, more beer.
Hot Damn! It's the Soggy Bottom Boys!
Yeah, it was probably written in the US :)
Who are these clowns that the media talk to for their "expert" opinion on computers?
"George Geczy, a software developer and computer consultant based in Ancaster, Ont., guessed that the problem involves identification numbers assigned to transactions"
Thousands of different reasons why their system cratered and some guy running a consulting firm from his basement nailed it for us! Guess his experience in installing MySQL a couple times helped him diagnose their massively huge database issue.
Just because you have a IT job and a bank card, doesn't make you an expert.
This is insightful? I hope aardwolf was actually trying to be funny. Otherwise his "don't save money because it is too easy to spend savings" plan will have him working until the day he dies. Personally, my SO and I live off of one paycheck and put the other into some form of savings or another. Using this plan, we're aiming at having a million smackers in the bank before we're fifty.
Better still, this plan doesn't have us up Shit Creek when a paycheck (or twenty) is missed.
If you have such a serious problem with raiding your savings account direct deposit can be a great tool for you. Have a small chunk of each check sent into this savings account and never touch it. Never ever. Hell, you'd probably be better off if you didn't even open your bank statements for that account but once a year. Whatever you do, living month to month is not the answer.
Peter
Downsize DC Today!
Try being fresh out of college. How the hell are you supposed to have 6 months of funds saved up when you've only been working full-time for 2 or 3? I'm sure that will teach those pesky kids trying to pay their rent and student loans for not saving anything up. Bottom line not everyone has that luxury.
In my case one CU is an account I've had since I was in my teens but their offices are about 15 miles away, and the other is from my former employer and is two states away. I drop in at the local one sometimes since it's about a mile from my girlfriend's office, but if I'm not headed up that direction I can go to the local credit union about a mile and a half from my house.
fencepost
just a little off
This has been widely mis-reported. RBC doesn't make such investments themselves, they act as the broker on behalf of clients, exactly the same as any other brokerage. In the case in question, the client chose to remain anonymous, and the shares in question were purchased by RBC, and then held 'in street name' for assignment to an internal client account. The actual details of who the account holder is, are protected by confidentiality laws, and would only become public information if the client requested certificates of shares issued in thier own name, rather than held by the brokerage in street name on thier behalf. the courts can also order such disclosures, but will only do so if there is a real requirement for said disclosure. In this case, there is no requirement for disclosure.
RBC has recieved a lot of negative exposure in the linux community simply because they have respected privacy laws. They acted as the broker in the transaction, and held the shares on behalf of a client. I'd commend the bank, in the face of a lot of pressure, never once have they released or leaked to the public the name of the client they are acting on behalf of. This is as it should be.
The real question in my mind, what individual/corporation outsourced this transaction to Canada, to take advantage of privacy laws that allowed them to do the entire deal anonymously, with the bank acting as the publicly visible broker of record?