Royal Bank of Canada Software Upgrade Goes Awry
Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.
They actually pay Canadians?... With money?
I guess that bad karma is pretty sticky. Even selling their preferred A-1 shares to Baystar didn't save them.
My Canadian friends are screaming bloody murder. I don't blame them.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
As an employee of Interplay, I can say that a 1 or 2 day delay in pay is nothing at all to worry about.
Coincidence, maybe, that England's air traffic control goes down during a software upgrade, and then the same happens to the Royal Bank of Canada?
Paranoia keeps you healthy!
This affects a lot of people - even if you don't bank with Royal Bank but your employer does then you will be affected. The HR manager where I work sent out a bulletin today that should apply to anyone affected by this situation:
----
All financial institution are on line with this issue, mortgage or automatic debit payments, will be honored, should anyone be charged interest , advise your bank,the Royal Bank will refund the charges.
All financial institution will advance cash based on an employee presenting a pay stub, they will not advance the full amount of the pay stub , they will however provide cash for the weekend.
----
I deposited a money order at an RBC branch on the 2nd of june, they told me it would take 12 hours. it still has not been put in my account, same with my pay from today.
At least I have real reason why my rent is late this month..
had their direct deposits delayed by a day or more."
Wait till your bank holds onto your payroll checks for 2 weeks.
Once a bank of mine made an addition mistake, i wrote a pile of checks that all bounced. The bank acknowledged their mistake, and restored funds in my account, but refused to help out with all the check-bouncing fees.
$25 X 17 Hurray.
do() || do_not();
I just hope their programmers aren't unionized. Heads should roll for this one. In cases like this, you should be lucky if you aren't held 100% liable.
Dark Nexus
"Sanity is calming, but madness is more interesting."
Geez, I'm showing my age again...
A feeling of having made the same mistake before: Deja Foobar
why they wanted their money back from SCO. ;)
I've heard a lot about this here in Canada over the past few days. Does anybody know what systems RBC was using, and what upgrade they were doing? It definitely seems suspicious that they were doing an upgrade at the *end* of the month (May), which is the busiest time for a bank (I know, from having worked at one). Was this really an upgrade gone wrong? Are there any more technical details?
I visited RBC earlier this year to make changes to my retirement plan and I was shocked to see that the account manager used a single PC to manage the accounts and access the internet. When I pointed that out, he said "don't worry, we run the best anti-virus software there is" (McAfee by the look of the icon in the tray). Because, as we all know, it's those viruses that eventually steal passwords and break into the databases. *rollseyes*
"Never attribute to malice what can be adequately explained by stupidity"
Comment removed based on user account deletion
I am a Royal Bank customer too... fortunatly my company uses CIBC, so I went down to the bank on my way to work this morning with my paystub and left with my pay, and all my funds from my account. I closed it and gave my financial buddy at work a new account with CIBC.
Honestly though. Being a software developer and knowing the development cycle like the back of my computer leads me to wonder how in the world they didnt test it fully. I mean... comeon guys. And that kind of institution using SCO's brand of UNIX? face + palm
Oh well... i dont care anymore... i close the accounts and visa card and when they asked me why, I just said: "I can't trust a bank that can't deal with this kind of glitch."
while(1) { fork(); };
And people wonder why I've become so incredibly disgusted with standard capitalism: "I owe them a Visa payment, but this glitch hasn't affected their ability to collect the money that's owed to them," Maria Janchenko, 26, said in Toronto, adding that she had been contacted three times this week by a collection agency.
It's their fault that these people aren't recieving paychecks and they're still hassling them aobut paying bills?
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
Sure, I don't earn interest on it, but at least I have some in an emergency
No, Lou, you're in the unstable branch! Gaah!
I made my credit card payment via online banking on June 1st, and the transaction went through. However, on June 2nd, the system decided to pay my credit card again a second time. Now I'm down a few hundred bucks.... should be fun getting this sorted out :|
"There is no spoon." - The Matrix
I don't typically work on these all-or-nothing type of systems. I usually work on embedded controllers, so we have the ability to put, say 10 units into the field for trials. Given the English air troubles and now this, isn't there a way to deploy a system such as this where it can be tested with real-world loads but not be the only system in use? Our controllers are not placed directly on the assembly line on the first day out, just for this purpose.
I for one welcome our new [insert main topic] overlords.
I recently got an overdraft notice on my bank account, four days after depositing my paycheck, in the branch with a teller. After several days of wrangling with their phone customer service and various managers at the bank I finally found out what had gone wrong: the teller had entered the wrong account number into the computer and someone recieved my money in their account. After several more forms and a couple hours of waiting around in the bank they finally got me my money back, but this was after being without cash or check-card for a week. All this because someone, whos job is to be exact, typoed.
I'm sure if this had been their money they would've gotten it back in less than 7 days, and levied some hefty time and inconvenience charges.
-"Nice jacket, who shot the couch?."
junioradmin@rbchost:/> rm -rf core *
waiting
waiting
thinking...this is taking longer than it should
phone rings.
ctrl-| ctrl-| ctrl-| ctrl-|
I love that.. any reasonable costs. How about the cost of damage to someone's credit when a payment can't go through... are they going to write a million credit apology notices? Are they going to write paper letters so you can keep a copy when someone calls into question your credit? The credit system is very damaging in these cases and has no easy fix.... I recommend all people go their RBC branch and get a letter explaining why payments were missed. Have them give you as many registered copies as you need for all your creditors affected.
(1st sig) If this were a snappy sig, you'd be reading it right now. (2nd sig) I'm a karma whore. >Insert FUD here
They tried to upgrade to Linux but they didn't pay the $699 license fee!
Tsunami -- You can't bring a good wave down!
i bank at RBC, and i have no money in my account right now. and it's my girlfriend's birthday. and she hates the present i got her. and my dog got run over by the bank manager... if i had a dog he would have anyways. it turns out that _some_ of the transactions from the weekend actually did go through, like the ones from my account, so when they reapplied everything yesterday i got double-debited for everything and it emptied my account. whee. fun side note: if you walk into a Royal Bank branch, you'll notice that the terminals behind the counter are running 16bit windows apps. check it out frank, we got this great new version, it's For Workgroups!
Nice planning: an end-of-month upgrade at a financial institution when, by their own admission, transactions are at their peak.
Maybe they thought they would broaden their QA testing base to, say 20,000,000.
Have you Meta Moderated t
Well, whoever signed off on the code and said "it's ready to go" are the ones who fucked up. I mean I suppose you could have a situation where the actual production environment was vastly different from the development/testing one, but I find that doubtful.
Really, there's all kinds of blame to go around, and programmers deserve some of it, the system never should have been so brittle as to cause these kinds of problems in the first place.
autopr0n is like, down and stuff.
I wonder if the software was written off shore?
Fantasy remains a human right; we make in our measure and in our derivative mode... -- JRR Tolkien
Why post a bug story on /. if we can't blame it on M$?
I totally agree. Most experts recommand that you have three to six months worth of funds saved up, which means that delay of payroll check deposit of a day, or even couple of weeks should be non-issue.
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
For the longest time I was living from paycheck to paycheck. Compared to my friends I made some pretty good money for being 20 (30K/Yr) but it didnt seem like much when you were living on your own with rent, insurance, car payments, electric, cable, phone, water, and a girlfriend. Its amazing how things add up. It just so happens that I receive the first paycheck of the month on the same day that rent is due. After paying rent I'm left with about $100. It is also convenient that the second and last paycheck of the month is received on the same day that all of the bills are due. After paying them I'm left with $300. Note I havent mentioned the G/F tax yet but that one is expensive.
Anyway, my tip is, next time you get a bonus, tax returns, some lump sum of money, spend it on next months rent before you can do anything else with it. Trust me on this. If you put it in your savings you can too easily transfer it to checking when you see Wizz-Bang4000 on pricewatch for only $499! I do this every chance I get and it really helps out a lot.
Now if I could only figure out what to do with the SO.
Im dreaming ofa big bndwdth, That can resist the
HPUX and AIX, at least on their web servers, and no doubt also on their critical systems.
c om
See nextcraft: http://uptime.netcraft.com/up/graph?site=www.rbc.
MW
---
BDOS ERR ON A:>
a beer drinking beaver was found at the mainframe keyboard typing I AM CANADIAN over and over. More details to follow...
presmike
A few short years ago, the Toronto Dominion's entire network went down for a whole day. I happened to be visiting a friend a train ride away, and could not get money out to get home out of any machine. It was frustrating, and it made me wonder why my service charges hadn't gone to redundant connections and machines, but I was able to borrow some cash and get home.
But when it happened a second time, in less than a year, I got a little frustrated and switched to president's choice bank.
My wife uses Royal Bank, and her pay has not gone through yet and it is now more than several days late. I certainly hope they work later than 4pm, monday to friday, to fix this. Some people who were supposed to be paid on the 31st have bounced their rent cheque!
It's makes you wonder what "money" really is, when a software error can make it disappear.
Can you imagine working in that IT department right now. My first reaction when I saw this story was that I felt incredibly sorry for those IT guys and gals. Must be hell over there right now!
I know I always sweat when releasing new software, at least I don't have to worry about effecting the bank accounts of millions of people. That would truly be scary!
I hope my Visa payment goes through ok...they had to do everything on paper.
I immigrated from the Netherlands to Canada and here are the differences:
1. Cheques? HAHAHHAHAHAH we didn't use them anymore for 15 years. NOT NECESSARY in Europe: you can just 'direct deposit' to everyone. And: NOT necessary to create 'bill payee' lists first - what's the use - just fill in the receivers' details if you want to pay (and yes, of course you can create lists of regular receivers). Also, people can pay ANYONE AT ANY BANK - not just payees who happen to be in YOUR bank's list.
2. Costs: Canadian banks are utterly expensive. Reason: no competition and he - why innovate or be more efficient when we can RAPE customers by charging outrageous fees for CRAP service. In Holland: crap service as well but you DON'T get charged! Just a very modest yearly FIXED fee.
3. Savings interest: in Europe it still pays to have savings account (beats inflation). Here in Canada I get 0.0000000001% (MINUS of course all kinds of 'service' fees because (a) I'm blond (b) I go biking 3 times a week (c) I just happen to be a customer who they can rape.
To give ONE good example that explains the difference. I still have Dutch bank accounts. From here (Canada) - I can use my computer to transfer money DIRECTLY from my Dutch bank account to - let's say - ANYONE in - let's say Greece; and WITHOUT costs. Now try doing that with Any north-American bank; they'll charge you with bizarre fees and the money is NOT in the account of the receiver the same day but instead you will have to wait a week at least. Yea you got a looong way to go (keep on dreaming that USA is the best in everything - take a vacation outside USA some day).
I just have a few big questions, not that the bank is likely to answer any of them. I suppose the inevitable lawsuits may flush out some of them though.
1. What OS(s) were they running before this happened?
2. Were they really doing an upgrade or a crossgrade, that is, switching to a new system altogether?
3. Was this being handled by in house IT or was it being outsourced half way around the world?
To the making of books there is no end, so let's get started
We finally reached an agreement with this Nigerean wife of an ex-general that she found $10,000,000 in a hidden vault behind the ex'es private toilet bin and agreed to give me 10% if I provide them with my checking account number.
Darn Canadian Bank, now the whole deal might not go through...
Yam, yam, uga booga, yam, yam, yade, yade, uga booga, yam, yam, yade, yade
Programmer 1: It happened like 3 days ago, eh.
Programmer 2: And welcome to day 3.
Programmer 1: "make install" hosed it.
Programmer 2: Hosed it down, eh, like backbacon at a Bah Mitzvah.
Programmer 1: But it's okay, eh. I got my thinking touque on and the beer and pizza are on the way.
Programmer 2: Yeah, we should have it back up by tomorrow, eh. Only, we're gonna need some more vacation after this.
Programmer 1: And beer, eh.
Programmer 2: Yeah, more beer.
Hot Damn! It's the Soggy Bottom Boys!
Who are these clowns that the media talk to for their "expert" opinion on computers?
"George Geczy, a software developer and computer consultant based in Ancaster, Ont., guessed that the problem involves identification numbers assigned to transactions"
Thousands of different reasons why their system cratered and some guy running a consulting firm from his basement nailed it for us! Guess his experience in installing MySQL a couple times helped him diagnose their massively huge database issue.
Just because you have a IT job and a bank card, doesn't make you an expert.
This is insightful? I hope aardwolf was actually trying to be funny. Otherwise his "don't save money because it is too easy to spend savings" plan will have him working until the day he dies. Personally, my SO and I live off of one paycheck and put the other into some form of savings or another. Using this plan, we're aiming at having a million smackers in the bank before we're fifty.
Better still, this plan doesn't have us up Shit Creek when a paycheck (or twenty) is missed.
If you have such a serious problem with raiding your savings account direct deposit can be a great tool for you. Have a small chunk of each check sent into this savings account and never touch it. Never ever. Hell, you'd probably be better off if you didn't even open your bank statements for that account but once a year. Whatever you do, living month to month is not the answer.
Peter
Downsize DC Today!
As someone effected by the mess, I have good news, My pay is in the bank. No news whether they "fixed" the problem or used a work around, which is more likely IMHO.
It makes me wonder; if you piss off enough of the clueful folks in this industry, would they simply not apply at your organization, insuring that the only people your HR department sees are the dregs of the vocational schools? Since HR people can't tell the difference between good IT people and bad, no one would get wind of the situation until such time as there were a major failure in your systems brought on by your shitty IT department. Makes you stop and think, doesn't it?
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Try being fresh out of college. How the hell are you supposed to have 6 months of funds saved up when you've only been working full-time for 2 or 3? I'm sure that will teach those pesky kids trying to pay their rent and student loans for not saving anything up. Bottom line not everyone has that luxury.
Royal, CIBC, Scotiabank, and Bank of Montreal
As someone pointed out, you missed TD Canada Trust (until recently two banks). There's also National Bank, and Laurentian (which again was recently acquired by another bank, but still has some locations open under that name).
And we even get stuff like Ethical Funds. Who aren't even the one I was looking for that offers a similar service.
I've been saying that to IT people for years. While security is important, having a sound disaster recover plan is even more important. Your system cannot be 100% secure; it's practically impossible. It can be 99.99999999% secure, but that one time can still happen, and you had better be prepared for it.
I've seen too many IT people focus entirely on security, and forget about having a sound backup strategy. Those IT people need to get fired. Sure, playing with Arcserve or Backup Exec or whatever sux, compared to finding some new hash algorithm to further encrypt your ssh session, and let's face it, we all like learning new things, but IT isn't about that kinda stuff.
I've had yelling matches at my last employer with IT people; they insisted on not having ANY FTP server, were farting around with NDS, yet when we asked them for a restore from tape, every time there was an issue. They had a 100% failure rate on their backup tapes.
There's too many retarded IT people out there. and too many retarded IT managers out there too.
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
I was at a remote site installing some new Unix workstations when I got an urgent call from the site administrator for some servers I had just installed a few days ago. One of the workstations was down, and wouldn't come up. I asked him what had changed, had he done anything, etc. He said that he hadn't done anything, could I come quick. I finished up what I was doing, and drove over there (45 miles, unfortunately).
/development directory off of the root disk, loaded a database application into it but filled up the hard disk, and then to clean up after himself did a "rm -r /dev*". The /development directory was gone, along with /dev!
(Background info: I had told the managers at the site that the site admin needed three classes of training, hands on work with me while I installed and implemented the systems, and some other experience before he went solo. The managers agreed to this but they never came through: He got zero training, and "was too busy" to work hands-on with me).
True enough, the system was down, and I had an appointment that night (Friday night, of course), but I would come in over the weekend to see what I could do. Of course, the guy hadn't backed up this system ever according to the backup procedures I had handed him.
I spend three hours on a Saturday trying to get this station up (it had design part data, and that data couldn't be permanently lost), and finally told the managers at the site that they needed to get the vendor out as it appeared to be a hardware problem based on what I was seeing (bios type messages, but once it hit the hard drive it died hard).
Vendor came out, checked out the hardware, and reported that nothing was wrong with the CPU, memory, SCSI cards, busses, disk drive, etc. The site administrator then remembers that the day before everything hit the fan that he created a
Immediately after he told me this little blurb (and I was red hot, Why didn't you tell me when I asked!) he informed me that it was time for him to leave and he did! Luckily for him he did leave, otherwise I was going to strangle him.
Fortunately, I was able to move the disk drive to piggyback off of a similar system, copy the device files from that system to the munged drive, and then recreate the couple of device files that were specific to this system. End result was that the system was back 100% again, and fully backed up (since I had zero confidence in the jerk). I told the managers what had happened, and what the actions of their site admin had cost me both personally and in terms of my work hours. I got blamed by the site admin for "not training him enough", for not being responsive enough, and for accepting his initial story and not digging into it to find out the root causes.
In my case one CU is an account I've had since I was in my teens but their offices are about 15 miles away, and the other is from my former employer and is two states away. I drop in at the local one sometimes since it's about a mile from my girlfriend's office, but if I'm not headed up that direction I can go to the local credit union about a mile and a half from my house.
fencepost
just a little off
BS, I used to think the exact same way. I've told myself "I'll save whatever's left over from my paycheck for an emergency fund". Sure enough, I always ended up not saving because there never were any money leftover.
I finally got sick and tired of living paycheck to paycheck and started checking out personal finance websites, such as The Motley Fool. I got myself a free 30-day membership and visited forums to learn about living below your means, cutting credit card interest rates and playing their "balance transfer games", and the "paying yourself first" method. With the lowered expenses, I put away at least $100 a month toward savings and investments. I treat that $100 or more as a bill that I must pay (thus "paying your self first") before any other bills. Otherwise I'll found a way to spend that money.
I have saving and non-retirement investments of $1200 right now. That's not that much but that's way better than being broke. It took me about 6 months to save up that much, and I estimate that I'll need $6000 for the 3 months worth, so at this rate, I'll need about 2 more years (yes, I wish that I started sooner).
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
So what do these "experts" recommend you spend during those 3 to 6 months?
Honestly, if everyone had the ability to just save up 6 months worth of funds then no one would be bouncing checks or investing in short term disability insurance. I had bills before I had a job, no one gave me a grace period to get together some emergency savings.
I save a pretty good amount, 20% of each check goes straight to savings, but it'll still take me most of a year to get 3 months of backup funds. Someone living on a more hand-to-mouth salary may never get that much saved.
-- I'm not a pessimist, I'm a realist. It's not my fault that life sucks so much. --
SO finds envelope :-)
Interested in a Flash-based MAME front end? Visit mame.danzbb.com
Since when does having a girlfriend mean having a money pit? It sounds like the girlfriend is more interested in material goods than you or you are out to make a big, but expensive impression on her.
RBC (ie Royal Bank) invested $50M in SCO simply on the odd chance that they did actually did pull off a court win. Basically they were covering all the bases for their investment funds, which you would expect from any decent financial institution. I don't know for a fact that RBC uses SCO-Unix, but that is not what the money they paid to SCO was for.
Re-elect Bush and see what happens to your buck...
Actually, it has more to do with the relative rate at which the US and Canada are inflating their currencies. Inflation, regardless of what the media is babbling about the Consumer Price Index, is about the number of units of currency and the available goods. Period. If the government prints more money, every dollar is worth less. However, that devaluation is not instant. It does take a while for prices to rise. The people who hold the money immediately after it is printed get nearly full value for it.
That's not to say that Pres. Bush and the wars in Afghanistan and Iraq are unrelated to the inflation of the US currency. However, the US Federal Reserve is also engaged in some pro-cyclic activities in an attempt first to get the US out of the most recent recession and then to keep the recovery going. That would be happening even if all of the US military was guarding airports in the US.
That's a highly desirable breathing space but still leaves you shackled to your paycheck. If you can learn discipline -- where "savings" means "that stack of money that keeps growing and that I will never touch unless my child is dying", you will be FREE.
FREE, to take a six-month leave of absence to do something that's important to you.
FREE, to quit the job that is making you ill with stress, even though you have no prospects at this time.
FREE, where your boss and your company's CFO and any of the financial institutions you keep your money in -- all these have NO SAY in your life, except as far as you wish them to.
You can't protect against everything, but 30K ought to be enough to get ahead. I started out at 15K a year in 1987, and gave some of that away to charities. I know what humble beginnings are like. ESCAPE THEM!
Couldn't happen to a nicer, more deserving bunch of jerkwads.
(/former field-support rep for a vendor, who got *burned* by the incompetence and mendacity of RBC IT personnel who lied to their manager, and my manager, when THEY screwed up their evaluation of our product - AFTER they had dragged the evaluation out past the 12-month mark. . . how the hell do you justify evaluating a product for 12 months?).
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
One thing I've never understood about bank computers, why is it they don't do transactions outside banking hours when it involves putting money INTO your account?
Sure I can make a deposit at an ATM and have it instantly accessable, but what I am talking about is between bank transfers and such, for some reason these seem to take place only on weekdays.
Is there really some peon sitting in front of a terminal approving every transaction?
What peeves me is when I have something being transferred from say my merchant account, it can take 5 business days to get to my account, I mean these are computers we're talking about here and that type of delay really makes no sense when we live in a world where instant transactions are available.
I've seen stuff start transfer on a Wednesday and take until the following Tuesday to show in the account, that is just sad.
>If you can learn discipline -- where "savings"
>means "that stack of money that keeps growing and
>that I will never touch unless my child is dying"
That's a good strategy and everything, but I learned that it would be better to go ahead and let that savings account go to about zero, if I can use it to get completely out of debt.
If I had a "child is dying" incident, I'm sure it will mean going into debt anyway, and it can't hurt to have zero debt in a situation like that.
-fb Everything not expressly forbidden is now mandatory.
A combination of Win2k, Solaris, Linux (various distributions) and VMS.
This has been widely mis-reported. RBC doesn't make such investments themselves, they act as the broker on behalf of clients, exactly the same as any other brokerage. In the case in question, the client chose to remain anonymous, and the shares in question were purchased by RBC, and then held 'in street name' for assignment to an internal client account. The actual details of who the account holder is, are protected by confidentiality laws, and would only become public information if the client requested certificates of shares issued in thier own name, rather than held by the brokerage in street name on thier behalf. the courts can also order such disclosures, but will only do so if there is a real requirement for said disclosure. In this case, there is no requirement for disclosure.
RBC has recieved a lot of negative exposure in the linux community simply because they have respected privacy laws. They acted as the broker in the transaction, and held the shares on behalf of a client. I'd commend the bank, in the face of a lot of pressure, never once have they released or leaked to the public the name of the client they are acting on behalf of. This is as it should be.
The real question in my mind, what individual/corporation outsourced this transaction to Canada, to take advantage of privacy laws that allowed them to do the entire deal anonymously, with the bank acting as the publicly visible broker of record?