Slashdot Mirror


Royal Bank of Canada Software Upgrade Goes Awry

Reader mks113 writes "Many Canadians living payday to payday have been in for a shock this week. Canada.com along with many other sources is reporting how thousands of customers have been inconvenienced following an unsuccessful software upgrade at the Royal Bank of Canada on Monday. All government employees (including me) in several provinces had their direct deposits delayed by a day or more." RBC has a comment on the mess.

36 of 602 comments (clear)

  1. Sticky karma.. by the_rajah · · Score: 5, Insightful

    I guess that bad karma is pretty sticky. Even selling their preferred A-1 shares to Baystar didn't save them.

    My Canadian friends are screaming bloody murder. I don't blame them.

    "Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain

    --


    "Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
  2. Re:Coincidence? by Sique · · Score: 4, Insightful

    Now you know where the old wisdom "Never touch a running system" comes from.

    --
    .sig: Sique *sigh*
  3. Ah, Nostalgia... by ackthpt · · Score: 5, Insightful
    Remember back when companies had Q/A departments and procedures? Wrote test plans and tried various scenarios to make sure the software was idiot and bullet proof? When routine software updates didn't suddenly pull your pants down and slap a creme pie into your face? When companies didn't just write any old thing and throw it out there for their customers to actually perform the test?

    Geez, I'm showing my age again...

    --

    A feeling of having made the same mistake before: Deja Foobar
    1. Re:Ah, Nostalgia... by ackthpt · · Score: 4, Insightful
      I dont see why this isn't the case nowadays really..

      It's called "Risk Management"

      Sure, some big company gets burnt trying to cut some corners; I'd be willing to bet that the vast majority of companies out there that demand high availability (also the gov't) still maintain policies and procedures for their upgrades.

      The deal is you have companies which now asses the costs of proper testing verses the cost of defending themsevles against their product blowing up and opt for whichever is cheapest.

      There are companies which must maintain a higher standard, by law or existing contract. Unfortunately the trend I've been watching over the years is an acceptable level of incompetence or defects. Manufacturers of PC parts, f'rinstance, are fine with a 15% failure rate off the line. I couldn't imagine such being acceptable with pacemakers.

      Ironically, most of the PC's in the world run on some version of Windows and even XP still loses its marbles on a regular basis. Thanks to the complexity of some products, some companies simply weigh the rist and make a financial decision and some CYA plan for Image Damage Control -- Gee, sounds just like the war on Iraq, come to think of it, it's a pervasive attitude.

      --

      A feeling of having made the same mistake before: Deja Foobar
  4. What sytems, what upgrade? by iso · · Score: 3, Insightful

    I've heard a lot about this here in Canada over the past few days. Does anybody know what systems RBC was using, and what upgrade they were doing? It definitely seems suspicious that they were doing an upgrade at the *end* of the month (May), which is the busiest time for a bank (I know, from having worked at one). Was this really an upgrade gone wrong? Are there any more technical details?

  5. Re:Same as British Air Traffic Control? by MoonBuggy · · Score: 5, Insightful

    "Never attribute to malice what can be adequately explained by stupidity"

  6. Partial Deployment Possible? by gmletzkojr · · Score: 2, Insightful

    I don't typically work on these all-or-nothing type of systems. I usually work on embedded controllers, so we have the ability to put, say 10 units into the field for trials. Given the English air troubles and now this, isn't there a way to deploy a system such as this where it can be tested with real-world loads but not be the only system in use? Our controllers are not placed directly on the assembly line on the first day out, just for this purpose.

    --
    I for one welcome our new [insert main topic] overlords.
  7. Re:Just like the suits by aardvarkjoe · · Score: 2, Insightful

    You don't get contacted three times by a collection agency for being one day late with your Visa payment.

    --

    How can we continue to believe in a just universe and freedom to eat crackers if we have no ale?
  8. Re:Just like the suits by Dark+Nexus · · Score: 4, Insightful

    You know, if she's been contacted THREE TIMES this week about payment, then she's at LEAST 2 months overdue. Going to blame the other 2+ months on the computer glitch that started on Monday? Besides, their Visa bills aren't actually DUE until about 7 working days into the month, so if she was up to date, then her payment wouldn't even be due yet.

    No, that's just somebody who thinks the world owes them everything taking the opportunity to complain because it might get them something they don't really deserve.

    --
    Dark Nexus
    "Sanity is calming, but madness is more interesting."
  9. Who Scheduled this upgrade? by eltoyoboyo · · Score: 3, Insightful

    Nice planning: an end-of-month upgrade at a financial institution when, by their own admission, transactions are at their peak.

    Maybe they thought they would broaden their QA testing base to, say 20,000,000.

    --
    Have you Meta Moderated t
  10. Re:Oh no! by shawn(at)fsu · · Score: 2, Insightful

    Something like this happened at my company with the direct deposit so I understand you situation but shouldn't you have made sure that you had the money in the account before you wrote the check?

    I know it's a common practice to float a check, you take in to account that the check will take X days via us mail, it will then take x days to post, my paycheck will go in to my account in X - y days so I'll be fine.

    Just because it's common practice doesn't make it right. At some point you have to take responsibility, you singed your name to that check and said that at from the date on this check forward I have the funds in my account to cover it.

    I'm not trying to be an troll or anything, I just think that you share a part of the blame for your checks bouncing.

    --
    500 dollar reward for tip(s) leading to the arrest of the person(s) who stole my sig.
  11. Re:Somebody should get fired by Pxtl · · Score: 4, Insightful

    No, this is management's fault. The only time a programmer fails is if something wasn't delivered on time, or they just don't produce, or their stuff doesn't make it past QC. Then fire them.

    If bad code makes it into the wild, then somebody signed off on it. Somebody cut corners on testing. Somebody decided deadline is more important than quality. Somebody insisted it had to run the newest Microsoft code.

    That somebody is the programmer's boss.

  12. Re:Somebody should get fired by Rude-Boy · · Score: 2, Insightful

    I'd bet on managers pushing through the update before it was ready.

  13. Re:Somebody should get fired by GoofyBoy · · Score: 5, Insightful

    >Heads should roll for this one.

    But are the "correct" heads going to roll?

    >In cases like this, you should be lucky if you aren't held 100% liable.

    If you were suppose to be held liable, do you think anything would change? Were any Professional Engineers held liable for the big blackout last year?

    --
    The surprise isn't how often we make bad choices; the surprise is how seldom they defeat us.
  14. Re:May be that will teach you by Bull999999 · · Score: 3, Insightful

    I totally agree. Most experts recommand that you have three to six months worth of funds saved up, which means that delay of payroll check deposit of a day, or even couple of weeks should be non-issue.

    --
    1f u c4n r34d th1s u r34lly n33d t0 g37 l41d
  15. As one who is just making it by I offer this advic by aardwolf204 · · Score: 4, Insightful

    For the longest time I was living from paycheck to paycheck. Compared to my friends I made some pretty good money for being 20 (30K/Yr) but it didnt seem like much when you were living on your own with rent, insurance, car payments, electric, cable, phone, water, and a girlfriend. Its amazing how things add up. It just so happens that I receive the first paycheck of the month on the same day that rent is due. After paying rent I'm left with about $100. It is also convenient that the second and last paycheck of the month is received on the same day that all of the bills are due. After paying them I'm left with $300. Note I havent mentioned the G/F tax yet but that one is expensive.

    Anyway, my tip is, next time you get a bonus, tax returns, some lump sum of money, spend it on next months rent before you can do anything else with it. Trust me on this. If you put it in your savings you can too easily transfer it to checking when you see Wizz-Bang4000 on pricewatch for only $499! I do this every chance I get and it really helps out a lot.

    Now if I could only figure out what to do with the SO.

    --
    Im dreaming ofa big bndwdth, That can resist the /.crowd.May ur days b merry & bright & may al
  16. Those poor members of the IT department by pbailey · · Score: 5, Insightful

    Can you imagine working in that IT department right now. My first reaction when I saw this story was that I felt incredibly sorry for those IT guys and gals. Must be hell over there right now!

    I know I always sweat when releasing new software, at least I don't have to worry about effecting the bank accounts of millions of people. That would truly be scary!

  17. Re:Banking Hazards by Corporate+Gadfly · · Score: 4, Insightful
    I finally found out what had gone wrong: the teller had entered the wrong account number into the computer and someone recieved my money in their account.
    Hate to be a troll, but if you go to the bank teller, always ask them for a receipt and double-check your account numbers on that receipt. Yes, the teller shoulders some of the responsibility, but so do you.
    --
    Corporate Gadfly
    Jonathan Archer: the most beaten up Enterprise captain in Star Trek history
  18. Re:I wonder... by tool462 · · Score: 5, Insightful

    Yeah, it was probably written in the US :)

  19. Re:Banking Hazards by CarrionBird · · Score: 3, Insightful

    I would disagree. Yes you should double check, but doing thier job is 100% thier responsibility.

    --
    Free Mac Mini Yeah, it's
  20. Re:As one who is just making it by I offer this ad by pyro_peter_911 · · Score: 3, Insightful

    This is insightful? I hope aardwolf was actually trying to be funny. Otherwise his "don't save money because it is too easy to spend savings" plan will have him working until the day he dies. Personally, my SO and I live off of one paycheck and put the other into some form of savings or another. Using this plan, we're aiming at having a million smackers in the bank before we're fifty.

    Better still, this plan doesn't have us up Shit Creek when a paycheck (or twenty) is missed.

    If you have such a serious problem with raiding your savings account direct deposit can be a great tool for you. Have a small chunk of each check sent into this savings account and never touch it. Never ever. Hell, you'd probably be better off if you didn't even open your bank statements for that account but once a year. Whatever you do, living month to month is not the answer.

    Peter

  21. Yup by Greyfox · · Score: 2, Insightful
    They have already angered the IS/IT Gods and will not be able to successfully use computers for at least 3 lifetimes. No amount of bailing out of SCO now will save them.

    It makes me wonder; if you piss off enough of the clueful folks in this industry, would they simply not apply at your organization, insuring that the only people your HR department sees are the dregs of the vocational schools? Since HR people can't tell the difference between good IT people and bad, no one would get wind of the situation until such time as there were a major failure in your systems brought on by your shitty IT department. Makes you stop and think, doesn't it?

    --

    I'm trying to teach myself to set people on fire with my mind... Is it hot in here?

  22. Re:May be that will teach you by clart · · Score: 3, Insightful

    Try being fresh out of college. How the hell are you supposed to have 6 months of funds saved up when you've only been working full-time for 2 or 3? I'm sure that will teach those pesky kids trying to pay their rent and student loans for not saving anything up. Bottom line not everyone has that luxury.

  23. Re:Instability? by rainman_bc · · Score: 2, Insightful

    I've been saying that to IT people for years. While security is important, having a sound disaster recover plan is even more important. Your system cannot be 100% secure; it's practically impossible. It can be 99.99999999% secure, but that one time can still happen, and you had better be prepared for it.

    I've seen too many IT people focus entirely on security, and forget about having a sound backup strategy. Those IT people need to get fired. Sure, playing with Arcserve or Backup Exec or whatever sux, compared to finding some new hash algorithm to further encrypt your ssh session, and let's face it, we all like learning new things, but IT isn't about that kinda stuff.

    I've had yelling matches at my last employer with IT people; they insisted on not having ANY FTP server, were farting around with NDS, yet when we asked them for a restore from tape, every time there was an issue. They had a 100% failure rate on their backup tapes.

    There's too many retarded IT people out there. and too many retarded IT managers out there too.

    --
    09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
  24. Re:Wait a minute... by Anonymous Coward · · Score: 2, Insightful

    Oh, excellent economic analysis.

    Really, the only factor is whether the US is at war or not? Really!? Really, really!??

    Let's see ... seems to me Canada was in WWI and WWII as well, rendering your analysis on even the most facile level moot and not a little idiotic.

  25. Re:May be that will teach you by Jtheletter · · Score: 2, Insightful
    Most experts recommand that you have three to six months worth of funds saved up

    So what do these "experts" recommend you spend during those 3 to 6 months?

    Honestly, if everyone had the ability to just save up 6 months worth of funds then no one would be bouncing checks or investing in short term disability insurance. I had bills before I had a job, no one gave me a grace period to get together some emergency savings.

    I save a pretty good amount, 20% of each check goes straight to savings, but it'll still take me most of a year to get 3 months of backup funds. Someone living on a more hand-to-mouth salary may never get that much saved.

    --
    -- I'm not a pessimist, I'm a realist. It's not my fault that life sucks so much. --
  26. Why is your girlfriend expensive? by Webapprentice · · Score: 2, Insightful

    Since when does having a girlfriend mean having a money pit? It sounds like the girlfriend is more interested in material goods than you or you are out to make a big, but expensive impression on her.

  27. Re:is this not your greatest fear? by thelexx · · Score: 2, Insightful

    See my sig...

    --
    "Gold still represents the ultimate form of payment in the world." - Alan Greenspan, 1999
  28. Re:Wait a minute... by GordoTheGeek · · Score: 5, Insightful
    Stop! My Sides! You're killing me! Both you and the other moron are letting your ignorance show. You have quite succinctly demonstrated exactly what it is that I and a very large portion of the world do not like about what, unfortunately, passes for representative Americans. Bravo

    It's quite sad, really: any disagreement with you automatically means that I don't like Americans and any and all arguments can and will result in violence, in this case coupled with a strange assortment of insults, what I can only guess is a swipe at my sexuality, and a comment that shows your profound misunderstanding of geography. I must, however, point out that your message, such as it is, is getting garbled by your obvious mastery of grammar and spelling.

  29. Re:As one who is just making it by I offer this ad by Clod9 · · Score: 2, Insightful
    This advice isn't bad as far as it goes, but it only gets you a one-month buffer.
    That's a highly desirable breathing space but still leaves you shackled to your paycheck. If you can learn discipline -- where "savings" means "that stack of money that keeps growing and that I will never touch unless my child is dying", you will be FREE.

    FREE, to take a six-month leave of absence to do something that's important to you.
    FREE, to quit the job that is making you ill with stress, even though you have no prospects at this time.
    FREE, where your boss and your company's CFO and any of the financial institutions you keep your money in -- all these have NO SAY in your life, except as far as you wish them to.

    You can't protect against everything, but 30K ought to be enough to get ahead. I started out at 15K a year in 1987, and gave some of that away to charities. I know what humble beginnings are like. ESCAPE THEM!

  30. Re:Wait a minute... by pmsr · · Score: 4, Insightful
    Mate, according to the CIA World Factbook, Canada is "somewhat larger than the US". So, you have a neighbour country, you don't know it's size, say it is "probably" smaller than some random state in your country, and have the nerve to insult others in such a rash tone? I have to tell you this: american or not, you are a sorry excuse for a human being, and the world doesn't really need your kind. Well, maybe your mother does, but then again, what would be of some people without their moms?

    /Pedro

  31. my take: by jafac · · Score: 2, Insightful

    Couldn't happen to a nicer, more deserving bunch of jerkwads.

    (/former field-support rep for a vendor, who got *burned* by the incompetence and mendacity of RBC IT personnel who lied to their manager, and my manager, when THEY screwed up their evaluation of our product - AFTER they had dragged the evaluation out past the 12-month mark. . . how the hell do you justify evaluating a product for 12 months?).

    --

    These are my friends, See how they glisten. See this one shine, how he smiles in the light.
  32. Re:is this not your greatest fear? by RetroGeek · · Score: 2, Insightful

    It's makes you wonder what "money" really is,

    Money is a concept built on mutual trust. I trust that the money you give me will be honored at its face value in another place.

    Be it dollars, euros, gold, or matchsticks.

    Money, as such, is meaningless without that trust.

    --

    - - - - - - - - - - -
    I am a programmer. I am paid to produce syntax not grammar. Deal with it.
  33. Re:Wait a minute... by AviLazar · · Score: 4, Insightful

    On behalf of the United States, I would like to appologize to anyone who was offended by the moronic poster who decided to insult our Northern neighbor, as well as assume that Canada is smaller then Texas. To the insulting poster - buy a globe, buy a hammer. Study the globe, and smash your keyboard so you can't type anymore.

    --

    I mod down so you can mod up. Your welcome.
  34. Re:As one who is just making it by I offer this ad by fishbowl · · Score: 2, Insightful

    >If you can learn discipline -- where "savings"
    >means "that stack of money that keeps growing and
    >that I will never touch unless my child is dying"

    That's a good strategy and everything, but I learned that it would be better to go ahead and let that savings account go to about zero, if I can use it to get completely out of debt.

    If I had a "child is dying" incident, I'm sure it will mean going into debt anyway, and it can't hurt to have zero debt in a situation like that.

    --
    -fb Everything not expressly forbidden is now mandatory.
  35. Re:Big Questions by rborek · · Score: 2, Insightful
    I guess this begs the question - why was the new code not tested, using live data, on the test/devel machine? In an environment such as this, you can't do half-assed testing - you have to run real, live data into it and test it out (ie one month before, duplicate the data - if it isn't already duplicating it real-time into the test machine, which I would expect - and run the code. Any problems, wait one month and try again).

    This comes down to poor testing. The manager(s) involved should (and probably will) be fired - they would have had to have signed off on the code change, and odds are that they rushed the programmers/QA people through testing.