Slashdot Mirror
Royal Bank of Canada Software Upgrade Goes Awry
Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.
This affects a lot of people - even if you don't bank with Royal Bank but your employer does then you will be affected. The HR manager where I work sent out a bulletin today that should apply to anyone affected by this situation:
----
All financial institution are on line with this issue, mortgage or automatic debit payments, will be honored, should anyone be charged interest , advise your bank,the Royal Bank will refund the charges.
All financial institution will advance cash based on an employee presenting a pay stub, they will not advance the full amount of the pay stub , they will however provide cash for the weekend.
----
I deposited a money order at an RBC branch on the 2nd of june, they told me it would take 12 hours. it still has not been put in my account, same with my pay from today.
At least I have real reason why my rent is late this month..
Holy crap, $25 an incident, who are you banking with??? You need to escalate your complaint. If your branch manager refuses to budge, contact the district manager, and so on. Eventually, somebody will will be willing to do something and you'll get your jack back.
And people wonder why I've become so incredibly disgusted with standard capitalism: "I owe them a Visa payment, but this glitch hasn't affected their ability to collect the money that's owed to them," Maria Janchenko, 26, said in Toronto, adding that she had been contacted three times this week by a collection agency.
It's their fault that these people aren't recieving paychecks and they're still hassling them aobut paying bills?
SJW: a person who perceives an injustice, and while correcting it, commits a greater injustice.
Yes, because increasing the load on their webserver will CERTAINLY increase stress on their banking mainframes!
From the sounds of it, it's not even REMOTELY a load issue in the first place.
Someone over at Groklaw this morning was claiming that RBoC is the only major Canadian bank not still running OS/2. RBoC uses an OS from a company that's fairly close to Vancouver that is unbeloved of /.ers.
--
E_NOSIG
Holy crap, $25 an incident, who are you banking with??? You need to escalate your complaint. If your branch manager refuses to budge, contact the district manager, and so on. Eventually, somebody will will be willing to do something and you'll get your jack back.
I once had a bank (Crestar, now acquired by SunTrust I think) that intentionally arranged the checks from highest-to-lowest so as to maximize the bounce fees. That way, the $1.53 check I wrote before my $200 bouncing check generated a bounce fee too. Once I had $125 in fees but if they had processed them in the order I wrote them it would have only been $25.
I agree about the escalation. I did, and eventually they refunded all but the one $25 bounce fee.
HPUX and AIX, at least on their web servers, and no doubt also on their critical systems.
c om
See nextcraft: http://uptime.netcraft.com/up/graph?site=www.rbc.
MW
---
BDOS ERR ON A:>
A few short years ago, the Toronto Dominion's entire network went down for a whole day. I happened to be visiting a friend a train ride away, and could not get money out to get home out of any machine. It was frustrating, and it made me wonder why my service charges hadn't gone to redundant connections and machines, but I was able to borrow some cash and get home.
But when it happened a second time, in less than a year, I got a little frustrated and switched to president's choice bank.
My wife uses Royal Bank, and her pay has not gone through yet and it is now more than several days late. I certainly hope they work later than 4pm, monday to friday, to fix this. Some people who were supposed to be paid on the 31st have bounced their rent cheque!
Ok, if your credit is significantly damaged by a few payments being a couple of days late, you had shitty credit to begin with. It usually takes several missed payments before a creditor will report you.
"The problem with internet quotations is that many are not genuine" -Abraham Lincoln
I've got to defend him in this case. He said an addition mistake occurred and was made by the bank. Now, if he does his part, reconciles his account monthly, etc, then he has no reason to believe that things failed to occur as he expected them to. For instance, consider this:
June 1, he has a balance of $1,200 ready to pay his bills. In fact, he even confirms with the bank that he has this much money, and no outstanding checks/debits to his account. He does so ($700 rent, $150 phone, $50 cable, $75 electric, and $200 groceries to get the whole month's staples). That means he's spent $1,175. Now, the bank does an addition error. Let's say the first check to get to the bank is the cable bill, and their error is to add a 0 at the end of the check, meaning he just paid for $500 worth of cable. Second, in comes the rent, at $700. There's $1,200 out of his account. He has made zero errors here, but he will now be hit with insufficient funds charges for the groceries, and for the electric. And the elctric company and grocery store will also add their own charges on top of it. Some companies will run checks through repeatedly until they clear, incurring further insufficient funds charges from the bank and from the company. Furthermore, the grocery store is likely to stop taking his checks because of this.
No errors on his part, the bank screwed up, but he gets hit with all the penalties. He did what he was supposed to do, what more could he do?
GPL made simple: What was my stuff is now our stuff. If you improve our stuff, please keep it our stuff.
RBC, back when the company name was still "Royal Bank" and not three meaningless letters, was very much an IBM shop until around 1995/6. I'm talking PS/2 workstations with OS/2 2.1, Lotus CC:Mail on OS/2 server, 3270 terminals, Token Ring LANs in office buildings still using DB-9 connectors (no twisted pair!), the whole nine yards.
When they heard from IBM that OS/2 wasn't going to continue past Warp, they brought Microsoft in an redesigned the whole network infrastructure. It was a 3" binder. The mainframes would still be around, but all office and branch systems (except for tellers) would move to Windows, Ethernet, etc. Rolling this out to the branches of course takes longer, but eventually it got done.
Fast forward to now: last week I had to go to RBC Main Branch to do some banking. I noticed that the old teller systems (which had still ran DOS and used your usual function-key-and-tab-through-the-form interface) had all been updated to... web-based point and click! AKA Windows 2000 workstations. Not that this should have had any impact on back-end batch transaction processing, which should still be done by those Big Blues in the basement. Still... there's still major technology that being replaced, so it feels to me like they're hiding behind this "routine upgrade" crap.
Did I mention that I didn't get paid today because of this?
Anyways, about OS/2... many many ATMs (at least in Canada) still run OS/2. Most NCR models still out there still run a year 2000 patched version of OS/2 2.1EE. They used to be powered by NCR workstations with MCA buses - just like those IBM PS/2s. No idea what hardware is in there now. Those bankbook update slots were originally IBM Proprinters with a special feed mechanism. Those OS/2 based ATMs use IBM's Communications Manager/2 software for talking to the backend mainframe. None of this is super-sekrit: just watch an ATM boot up when it eats your card and the branch tech has to reset it. You could see the OS/2 desktop before the ATM interface program took over.
Consider switching to President's Choice financial since you're with CIBC already. It's run by Amicus (who owns CIBC) but the difference is they don't charge any fees. You can use any Amicus or CIBC machine and interac for free plus you get points towards groceries at Great Canadian Superstore or Lawblaws (out east). I've banked with them for a year now and it's awesome. I pay for $1 coffees with my bank card without a care in the world.
Unfortunately Canadians don't have that option. They get to choose from Royal, CIBC, Scotiabank, and Bank of Montreal. There are a couple others I can't remember right now.
TD Canada Trust is another big one. The Dutch bank ING has been doing a lot of business in Canada in the last few years too.
Yay socialism.
What does this have to do with socialism? Hell, if Canada were more capitalistic, the government would have allowed the major banks to merge, as they've been asking to for years, and there would be less consumer choice.
Does the U.S. have an equivalent for the CDIC?
Like all the major Canadian banks, the transactional system is on an IBM Mainframe. Running either OS/370 or OS/390. The transactional integrity is handled either with CICS or with custom home-grown software.
As for OS/2, that was only on ATMs. And ATMs are running anything from VxWorks to System 7 unix to OS/2 to embedded windows.
Finally, if you're going to cite a comment on a web site, as reliable as that is, please give a reference to it that is more then the URL to the whole site...
Yes, it was actually an upgrade. (Buddy works in the RBFG office with the fellow that screwed up.) I'm told that it was a single-character typo in the command line. Everything seemed OK until the system went on-line with live data. Then all hell broke loose. And my payroll deposit hasn't shown up yet either.
That acutally fluctuates from a period to another... right this moment, a looney is actually only 75% of a buck, but it has not always been this way, nor will it be forever.
Here's a little history for y'all. In 1862, the loonie and the buck were worth about the same thing. Two years later, in 1864, the loonie was worth 265% of a buck. That's right, people could buy $2.65 of american money for a single canadian dollar. After the civil war, the buck slowly went back up to match the loonie.
The loonie began to lose value compared to the buck at the beginning of WWI, then slowly restabilized after, then lost value again during WWII. In 1945, right after the war, the loonie and the buck regained their equality.
In 1961, both were worth about the same, and in 1972 the loonie was worth about 5% more than the buck.
At the beginning of 2003, the loonie was worth slightly more than 60% of the buck, since then, it got a big boost and almost hit 80% to be at 75% today...
Notice how all those major fluctuations coincides with wars... Civil war, WWI, WWII, Viet-Nam, Irak... Re-elect Bush and see what happens to your buck...
After 3 days without programming, life becomes meaningless
- The Tao of Programming
NCR uses very standard x86 hardware for their ATMs now. A family member works their and sometimes brings home old pieces - pentium 3 procs now IIRC.
I got to tour one of the Canadian plants when I lived up north, pretty cool to watch them built. I forget what OS they run now though.
I'm pretty sure TD Canada Trust still runs OS2 on the computers in the branches. Too bad TD sucks.
Royal, CIBC, Scotiabank, and Bank of Montreal
As someone pointed out, you missed TD Canada Trust (until recently two banks). There's also National Bank, and Laurentian (which again was recently acquired by another bank, but still has some locations open under that name).
And we even get stuff like Ethical Funds. Who aren't even the one I was looking for that offers a similar service.
LOL
Never mind your confused ideas about capitalism ("unions are the source of all evil") - it's your ideas about capital institutions (banks) that have me laughing.
Ever wonder why a bank needs about a million years and 2 million wankers to do anything? Why everyone spends more time in meetings talking about doing things than actually doing anything? It's so that if something goes wrong, everyone will be at fault. That's useful, because when it's everyone's fault, it's really nobody's.
If any heads roll, it will be because of political scores to be settled, not actual misdeeds. Oh, and you can forget about your silly daydream of "100% liable" - as a bank customer, you can't afford what it would cost to hold individual employees liable. What would you charge for a contract that had the following terms: be perfect, gain 100,000; make a mistake, lose 100,000,000? Only the institution can afford to cover liability, and that's because the risk is spread over many employees. It's the same principle as insurance.
"The good reader is a rarer swan than the good writer."
AIX on RS/6000's? ;-)
Maybe it's on an IBM mainframe and they upgraded from MVS/OS390/ZOS?? Maybe the upgrade was to ZLinux? Wouldn't that make Bill happy!
Anyhow, whenever I go in to redo my mortgage and loans, I check out their desktop system. They used to run OS/2 (years ago), now it's either NT or 2000 on the IBM PCs on their desks.
I know they also used to run Netware (buddy was one of their NDS gurus 5 years ago).
Anyhow on their desk, they usually have a terminal session open, I suspect this goes back to their main datacentre. Hard to say what the back end is, I suspect it's IBM hardware/software though.
In my case one CU is an account I've had since I was in my teens but their offices are about 15 miles away, and the other is from my former employer and is two states away. I drop in at the local one sometimes since it's about a mile from my girlfriend's office, but if I'm not headed up that direction I can go to the local credit union about a mile and a half from my house.
fencepost
just a little off
BS, I used to think the exact same way. I've told myself "I'll save whatever's left over from my paycheck for an emergency fund". Sure enough, I always ended up not saving because there never were any money leftover.
I finally got sick and tired of living paycheck to paycheck and started checking out personal finance websites, such as The Motley Fool. I got myself a free 30-day membership and visited forums to learn about living below your means, cutting credit card interest rates and playing their "balance transfer games", and the "paying yourself first" method. With the lowered expenses, I put away at least $100 a month toward savings and investments. I treat that $100 or more as a bill that I must pay (thus "paying your self first") before any other bills. Otherwise I'll found a way to spend that money.
I have saving and non-retirement investments of $1200 right now. That's not that much but that's way better than being broke. It took me about 6 months to save up that much, and I estimate that I'll need $6000 for the 3 months worth, so at this rate, I'll need about 2 more years (yes, I wish that I started sooner).
1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
RBC (ie Royal Bank) invested $50M in SCO simply on the odd chance that they did actually did pull off a court win. Basically they were covering all the bases for their investment funds, which you would expect from any decent financial institution. I don't know for a fact that RBC uses SCO-Unix, but that is not what the money they paid to SCO was for.
If a day delay could send you under, you have big problems.
That acutally fluctuates from a period to another... right this moment, a looney is actually only 75% of a buck, but it has not always been this way, nor will it be forever.
Dream on.
The canadian looney is artificially kept "low" by toying with the interest rates index to favor US exports, wich is the buyer of 90% of our exports.
Only once has the canadian dollar was worth more then the US dollar, and it proved disastrous for the economy.
Re-elect Bush and see what happens to your buck...
Actually, it has more to do with the relative rate at which the US and Canada are inflating their currencies. Inflation, regardless of what the media is babbling about the Consumer Price Index, is about the number of units of currency and the available goods. Period. If the government prints more money, every dollar is worth less. However, that devaluation is not instant. It does take a while for prices to rise. The people who hold the money immediately after it is printed get nearly full value for it.
That's not to say that Pres. Bush and the wars in Afghanistan and Iraq are unrelated to the inflation of the US currency. However, the US Federal Reserve is also engaged in some pro-cyclic activities in an attempt first to get the US out of the most recent recession and then to keep the recovery going. That would be happening even if all of the US military was guarding airports in the US.
1. What OS(s) were they running before this happened?
This question has "newbie" written all over. Bank applications run on true and tested OSes written decades ago. They run on large mainframes with uptime that often goes back to the day the computer was turned on sometime in the 70s or 80s.
This problem is, in all likelihood, an application problem.
Way to ignore the whole point of the example. Bank made an addition error. GGP's bank made an addition error, and refunded diddly squat when they screwed up. Possible addition errors resulting in similar outcomes: Applying the same check twice, turning $700 rent check into $750 rent check, applying an NSF fee to his account which was meant for somebody else's account, etc. A more basic one might be holding a deposit for two days for some reason (but not informing the customer), and still applying checks which were written based on that deposit (I've had that happen to me, too). Maybe you are getting the point now?
GPL made simple: What was my stuff is now our stuff. If you improve our stuff, please keep it our stuff.
I can't tell you what upgrade, but the backend is on HP Nonstop S Series servers, running a Unix variant.
Bank of Canada
That would be the Royal Bank of Canada. The Bank of Canada is a totally separate entity and is in fact run by the government to manage the money supply and interest rates.
- - - - - - - - - - -
I am a programmer. I am paid to produce syntax not grammar. Deal with it.
You're close to the money re: OS390, and it being a bank problem (not vendor). I work for another canadian bank, but have former associates at RBC. Here's what I've heard is happening:
- May 31st = month end. new month end code gets run first time in prod (JCL), there's problems - not sure of name of system, but the general bank ledger for customers is fubared.
- RBC rolls back all changes, so RBC is 1 day behind on Tuesday.
- re-run batch with old code on Wednesday am... unfortunately recovery procedures are flawed/human error, batch is screwed up again.
- now bank is 2-3 days behind...can't process transactions effectively, can't catch up with sequential batches in evenings because there's too much to run.
- RBC departments start running independently based on May 31st data...can't afford to be down more than 2 days. now the roadmap is a mess for recovery, general ledger is still at May 31st state (might be June 1st after a successful batch run last night???)
Apparently about 80-100 IT staff are living at the Skydome hotel in Toronto working 16 hour shifts (16 on, 8 off) to try and get caught up. Everything i've heard suggests that they know they can't get done during the business week...they're relying on 2 days of 24hr downtime on the weekend to reload the batches and get systems back in sync.
Based on my experience in Canadian banking (7 years) plus stories of old timers, this is the worst outage in close to 10 years, maybe the worst in 20 if my rumours prove to be true. I have no direct mainframe experience, though, so take my descripton of the problems with a grain of salt...
John Maynard Keynes: "When the facts change, I change my mind. What do you do?"
A combination of Win2k, Solaris, Linux (various distributions) and VMS.
This has been widely mis-reported. RBC doesn't make such investments themselves, they act as the broker on behalf of clients, exactly the same as any other brokerage. In the case in question, the client chose to remain anonymous, and the shares in question were purchased by RBC, and then held 'in street name' for assignment to an internal client account. The actual details of who the account holder is, are protected by confidentiality laws, and would only become public information if the client requested certificates of shares issued in thier own name, rather than held by the brokerage in street name on thier behalf. the courts can also order such disclosures, but will only do so if there is a real requirement for said disclosure. In this case, there is no requirement for disclosure.
RBC has recieved a lot of negative exposure in the linux community simply because they have respected privacy laws. They acted as the broker in the transaction, and held the shares on behalf of a client. I'd commend the bank, in the face of a lot of pressure, never once have they released or leaked to the public the name of the client they are acting on behalf of. This is as it should be.
The real question in my mind, what individual/corporation outsourced this transaction to Canada, to take advantage of privacy laws that allowed them to do the entire deal anonymously, with the bank acting as the publicly visible broker of record?