Slashdot Mirror
Royal Bank of Canada Software Upgrade Goes Awry
Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.
Coincidence, maybe, that England's air traffic control goes down during a software upgrade, and then the same happens to the Royal Bank of Canada?
Paranoia keeps you healthy!
had their direct deposits delayed by a day or more."
Wait till your bank holds onto your payroll checks for 2 weeks.
Once a bank of mine made an addition mistake, i wrote a pile of checks that all bounced. The bank acknowledged their mistake, and restored funds in my account, but refused to help out with all the check-bouncing fees.
$25 X 17 Hurray.
do() || do_not();
I just hope their programmers aren't unionized. Heads should roll for this one. In cases like this, you should be lucky if you aren't held 100% liable.
Dark Nexus
"Sanity is calming, but madness is more interesting."
I visited RBC earlier this year to make changes to my retirement plan and I was shocked to see that the account manager used a single PC to manage the accounts and access the internet. When I pointed that out, he said "don't worry, we run the best anti-virus software there is" (McAfee by the look of the icon in the tray). Because, as we all know, it's those viruses that eventually steal passwords and break into the databases. *rollseyes*
I am a Royal Bank customer too... fortunatly my company uses CIBC, so I went down to the bank on my way to work this morning with my paystub and left with my pay, and all my funds from my account. I closed it and gave my financial buddy at work a new account with CIBC.
Honestly though. Being a software developer and knowing the development cycle like the back of my computer leads me to wonder how in the world they didnt test it fully. I mean... comeon guys. And that kind of institution using SCO's brand of UNIX? face + palm
Oh well... i dont care anymore... i close the accounts and visa card and when they asked me why, I just said: "I can't trust a bank that can't deal with this kind of glitch."
while(1) { fork(); };
I made my credit card payment via online banking on June 1st, and the transaction went through. However, on June 2nd, the system decided to pay my credit card again a second time. Now I'm down a few hundred bucks.... should be fun getting this sorted out :|
"There is no spoon." - The Matrix
I recently got an overdraft notice on my bank account, four days after depositing my paycheck, in the branch with a teller. After several days of wrangling with their phone customer service and various managers at the bank I finally found out what had gone wrong: the teller had entered the wrong account number into the computer and someone recieved my money in their account. After several more forms and a couple hours of waiting around in the bank they finally got me my money back, but this was after being without cash or check-card for a week. All this because someone, whos job is to be exact, typoed.
I'm sure if this had been their money they would've gotten it back in less than 7 days, and levied some hefty time and inconvenience charges.
-"Nice jacket, who shot the couch?."
I love that.. any reasonable costs. How about the cost of damage to someone's credit when a payment can't go through... are they going to write a million credit apology notices? Are they going to write paper letters so you can keep a copy when someone calls into question your credit? The credit system is very damaging in these cases and has no easy fix.... I recommend all people go their RBC branch and get a letter explaining why payments were missed. Have them give you as many registered copies as you need for all your creditors affected.
(1st sig) If this were a snappy sig, you'd be reading it right now. (2nd sig) I'm a karma whore. >Insert FUD here
I can guaranty that they will not spend 2 seconds finding out if all those overdraft charges and this charge that they are putting on us customers are to be refunded. They will instead wait till people come in one at a time to have them removed... how many people wont bother 1000, 2000 so lets say that's 5$ a head that's 5000$ to 10000$ of extra earnings for the bank.
A psychopath can't tell the difference between right and wrong. A sociopath knows the difference - he just doesn't care.
I'm pissed as hell! Goddammned incompetent RBC does it again!
I deposit my cheque May 31st. I make two transfers from my chequing account to my Visa and my Savings. Come today, there's no money in my Chequing account! I look at the transaction history, and there's no trace of the deposit I made, or the transfers. Strangely enough, the money I transfered out of the savings account is still in the chequing account. They were supposed to fix this when? two days ago? I went into the main branch here in Vancouver and they told me they wouldn't be finished with their transaction backlog until next week. I hope they get sued. Canada's largest bank, and crappiest. I'm taking my money elsewhere!!
Well, whoever signed off on the code and said "it's ready to go" are the ones who fucked up. I mean I suppose you could have a situation where the actual production environment was vastly different from the development/testing one, but I find that doubtful.
Really, there's all kinds of blame to go around, and programmers deserve some of it, the system never should have been so brittle as to cause these kinds of problems in the first place.
autopr0n is like, down and stuff.
I wonder if the software was written off shore?
Fantasy remains a human right; we make in our measure and in our derivative mode... -- JRR Tolkien
It's makes you wonder what "money" really is, when a software error can make it disappear.
I immigrated from the Netherlands to Canada and here are the differences:
1. Cheques? HAHAHHAHAHAH we didn't use them anymore for 15 years. NOT NECESSARY in Europe: you can just 'direct deposit' to everyone. And: NOT necessary to create 'bill payee' lists first - what's the use - just fill in the receivers' details if you want to pay (and yes, of course you can create lists of regular receivers). Also, people can pay ANYONE AT ANY BANK - not just payees who happen to be in YOUR bank's list.
2. Costs: Canadian banks are utterly expensive. Reason: no competition and he - why innovate or be more efficient when we can RAPE customers by charging outrageous fees for CRAP service. In Holland: crap service as well but you DON'T get charged! Just a very modest yearly FIXED fee.
3. Savings interest: in Europe it still pays to have savings account (beats inflation). Here in Canada I get 0.0000000001% (MINUS of course all kinds of 'service' fees because (a) I'm blond (b) I go biking 3 times a week (c) I just happen to be a customer who they can rape.
To give ONE good example that explains the difference. I still have Dutch bank accounts. From here (Canada) - I can use my computer to transfer money DIRECTLY from my Dutch bank account to - let's say - ANYONE in - let's say Greece; and WITHOUT costs. Now try doing that with Any north-American bank; they'll charge you with bizarre fees and the money is NOT in the account of the receiver the same day but instead you will have to wait a week at least. Yea you got a looong way to go (keep on dreaming that USA is the best in everything - take a vacation outside USA some day).
I just have a few big questions, not that the bank is likely to answer any of them. I suppose the inevitable lawsuits may flush out some of them though.
1. What OS(s) were they running before this happened?
2. Were they really doing an upgrade or a crossgrade, that is, switching to a new system altogether?
3. Was this being handled by in house IT or was it being outsourced half way around the world?
To the making of books there is no end, so let's get started
Yep.
It's called software release management (SRM) and infrastructure configuration management (ICM). Both are parts of general configuation management.
According to a Gartner study (I don't have it at my fingertips), over 90% of all failures in mission critical systems are due to poor problem or configuration management. When this study was done, Gartner estimated that only 6% of the major corporations would have industry best practices in place by 2004.
Hmm - sounds like both the Bank of Canada and the English FAA got bit by this big time.
This discipline sounds like a great business opportunity. Unfortunately my previous employer disagreed.
My previous employer also shut down a (formerly) successful consulting organization and laid off 1100 talented individuals.
I wonder how much the Bank of Canada would have paid for this type of expertise. Of course, the check might be a day or two late . . .
I work at RBC in the IT department and can tell you that there are a number of people who literally have not left the building since the issue started a week ago.
The question is though. Was it Royal Bank's IT dept or is their IT outsourced?
Who are these clowns that the media talk to for their "expert" opinion on computers?
"George Geczy, a software developer and computer consultant based in Ancaster, Ont., guessed that the problem involves identification numbers assigned to transactions"
Thousands of different reasons why their system cratered and some guy running a consulting firm from his basement nailed it for us! Guess his experience in installing MySQL a couple times helped him diagnose their massively huge database issue.
Just because you have a IT job and a bank card, doesn't make you an expert.
"Fun" is generally shared by all parties involved and shouldn't come at the expense of others. But, then, my innate Canuckness is showing.
As someone effected by the mess, I have good news, My pay is in the bank. No news whether they "fixed" the problem or used a work around, which is more likely IMHO.
I was at a remote site installing some new Unix workstations when I got an urgent call from the site administrator for some servers I had just installed a few days ago. One of the workstations was down, and wouldn't come up. I asked him what had changed, had he done anything, etc. He said that he hadn't done anything, could I come quick. I finished up what I was doing, and drove over there (45 miles, unfortunately).
/development directory off of the root disk, loaded a database application into it but filled up the hard disk, and then to clean up after himself did a "rm -r /dev*". The /development directory was gone, along with /dev!
(Background info: I had told the managers at the site that the site admin needed three classes of training, hands on work with me while I installed and implemented the systems, and some other experience before he went solo. The managers agreed to this but they never came through: He got zero training, and "was too busy" to work hands-on with me).
True enough, the system was down, and I had an appointment that night (Friday night, of course), but I would come in over the weekend to see what I could do. Of course, the guy hadn't backed up this system ever according to the backup procedures I had handed him.
I spend three hours on a Saturday trying to get this station up (it had design part data, and that data couldn't be permanently lost), and finally told the managers at the site that they needed to get the vendor out as it appeared to be a hardware problem based on what I was seeing (bios type messages, but once it hit the hard drive it died hard).
Vendor came out, checked out the hardware, and reported that nothing was wrong with the CPU, memory, SCSI cards, busses, disk drive, etc. The site administrator then remembers that the day before everything hit the fan that he created a
Immediately after he told me this little blurb (and I was red hot, Why didn't you tell me when I asked!) he informed me that it was time for him to leave and he did! Luckily for him he did leave, otherwise I was going to strangle him.
Fortunately, I was able to move the disk drive to piggyback off of a similar system, copy the device files from that system to the munged drive, and then recreate the couple of device files that were specific to this system. End result was that the system was back 100% again, and fully backed up (since I had zero confidence in the jerk). I told the managers what had happened, and what the actions of their site admin had cost me both personally and in terms of my work hours. I got blamed by the site admin for "not training him enough", for not being responsive enough, and for accepting his initial story and not digging into it to find out the root causes.
What is one day's interest on all of that money?
They should make the software company pay for that and then they will think harder next time
One thing I've never understood about bank computers, why is it they don't do transactions outside banking hours when it involves putting money INTO your account?
Sure I can make a deposit at an ATM and have it instantly accessable, but what I am talking about is between bank transfers and such, for some reason these seem to take place only on weekdays.
Is there really some peon sitting in front of a terminal approving every transaction?
What peeves me is when I have something being transferred from say my merchant account, it can take 5 business days to get to my account, I mean these are computers we're talking about here and that type of delay really makes no sense when we live in a world where instant transactions are available.
I've seen stuff start transfer on a Wednesday and take until the following Tuesday to show in the account, that is just sad.