Slashdot Mirror
Software Glitch Leads To $23,148,855,308,184,500 Visa Charges
Hmmm2000 writes "Recently several Visa card holders were, um, overcharged for certain purchases, to the tune of $23,148,855,308,184,500.00 on a single charge. The company says it was due to a programming error, and that the problem has been corrected. What is interesting is that the amount charged actually reveals the type of programming error that caused the problem. 23,148,855,308,184,500.00 * 100 (I'm guessing this is how the number is actually stored) is 2314885530818450000. Convert 2314885530818450000 to hexadecimal, and you end up with 20 20 20 20 20 20 12 50. Most C/C++ programmers see the error now ... hex 20 is a space. So spaces were stuffed into a field where binary zero should have been."
Interesting? You're assuming we're all computer geeks. Wait a minute...
Meh. What's 23 quadrillion dollars really worth these days?
This guy's the limit!
In EBCDIC, hex 40 is a space. Making this error if EBCDIC was used would make the charge a whopping $4,629,771,061,636,895,312 - 4 quintillion dollars!
While all this is plausible, of course, the 12 is octal for a UNIX newline and the 50 is the '@' symbol; let us not forget that there are a lot of assumptions being made here and a lot of speculation.
This is a boring sig
So what was the minimum payment on that?
Well, it has never been successfully tested.
This is how Obama is paying for health care.
Paying taxes to buy civilization is like paying a hooker to buy love.
Isn't that about the cost of a couple of packs of smokes and a bag of chips at one of those gas station stores? If he filled up the truck, too...well, that would just about account for it.
Dude should shut up and pay what he owes.
I've calculated my velocity with such exquisite precision that I have no idea where I am.
He also felt a stab of fear that he had saddled all his unborn grandchildren -- and their grandchildren -- with a lifetime of debt. "Down the generational line, nobody would have any money."
Give me a break.
-Xoltri
...is that this was not caught by validity checks. Was this perhaps an error that affected only the printing of the statement?
Warning: this article may contain humor, sarcasm, parody, and perhaps even irony. Read at your own risk.
It was probably partly overwritten by spaces
"There is more worth loving than we have strength to love." - Brian Jay Stanley
Is not so much the error(stupid; but, if corrected, not ultimately a giant deal); but the response of the cardholder to the error:
"The bank kept him on hold for two hours, during which time he contemplated the impossibly bleak financial future that might await him. He also felt a stab of fear that he had saddled all his unborn grandchildren -- and their grandchildren -- with a lifetime of debt. "Down the generational line, nobody would have any money."
For fuck's sake, people, the credit card guys haven't actually bought a law concerning hereditary debt slavery yet, and this guy thinks that it is already on the books?
Muszynski compared the giant debt reprieve to receiving "an amazing Monopoly card that says, 'Bank error in your favor.' "
Pathetic. This guy is grateful that Visa condescended to fix their obvious mistake(this isn't some he said/she said billing dispute, this is someone who allegedly spent more than the world GDP at a gas station)? What is this cringing bullshit? Either this guy is just a sad sack or, rather worse, the "customer service" we get, along with the kangaroo courts that are "mandatory binding arbitration" actually make thankfulness for not being screwed a reasonable response.
"Do you owe $23 quadrillion or more on your credit cards? Well I'm about to tell you a secret that the credit card companies don't want you to know. You can settle your debt for pennies on the dollar and get out of debt fast!"
Tired of FB/Google censorship? Visit UNCENSORED!
I must've put a decimal point in the wrong place or something. I always do that. I always mess up some mundane detail.
Seriously though the Visa transaction charge of 2% = 462,977,106,163,690
How could this transaction go through?
love is just extroverted narcissism
0x20 is "space" in ASCII. The ASCII table maps numerical values to (mostly) readable characters. Unicode is a buffed out table supporting international (and other) characters. The programmers forgot to strip the whitespace from their input.
If I were him, I would have applied for a bailout, then gave myself a nice hefty bonus before going bankrupt.
It's the American Dream!
It's good to know their system is able to handle $23 quadrillion charges, now I just need to get them to raise my limit a bit.
Probably more offensive is that a glitch happened at all, large or small. It could have just as easily been $2.31 in which case he may have not noticed the overcharge and paid it. Charge several thousand people $2.31 too much and you can make an alright profit.
"During My Service In The United States Congress, I Took The Initiative In Creating The Internet." -Al Gore
Our credit score cannot repel debt of that magnitude!
I work in this industry. The only novelty here is that the error got into production, and was not caught and corrected before it went that far.
Submitters send files to processors which are supposed to be formatted according to specifications.
Note I wrote 'supposed to be'.
Some submitters do, from time to time, change their code, and sometimes they get it wrong. For instance padding a field with spaces instead of zeros. Woopsie...!
Seems that's what happened here. Sounds like a hex or dec field got padded with hex 20, and boom.
This is annoying, especially when the processor gets to help correct the overwhelming number of errors, and then tries to explain that it wasn't their fault. Plenty of blame to go around with this one.
And then explains why they don't both validate/sanitize input, and test for at least some reasonable maximum value in the transaction amount. A max amount of $10,000,000 would have fixed this. That and an obvious lapse in testing. This is what keeps my bosses awake sometimes, fearing they will end up on the front page of the fishwrap looking stupid 'cause their overworked minions screwed something up, or didn't check, or didn't test very well. I love one of the guys we have testing. He's insufferable, and he catches genuine show-stoppers on a regular basis. They can't pay him what he's been worth, literally $millions, just in avoiding downtime and re-working code that went too far down the wrong path.
Believe me, this is in some ways preferable to getting files with one byte wrong that doesn't show up for a month, or sending the wrong data format (hex instead of packed binary or EBCDIC, for instance) and crashing the process completely. Please, I know data should never IPL a system. Tell it to the architects, please. As if they don't know now, after the one crash...
If you knew what I know, you'd chuckle and share this story with some of your buddies in development and certification.
And pray a little.
At least it didn't overbill the cardholders by $.08/transaction. That would suck. This is easy by comparison. Just fix the report data. Piece of cake. Evening's worth of coding and slam it out in off-peak time. Hahahahaha!
deleting the extra space after periods so i can stay relevant, yeah.
Does he still get the airline miles for that one? I mean, even at 1 mile per dollar spent.... He can now book a first class ticket to mars...
I hope it was on one of the cards that gives him 1% cash back.
Lemme debunk a myth real quick for you folks.
If you EVER see a bank error "in your favor" or if your payroll check is off and you are over paid.......DO NOT SPEND THE MONEY
You will be charged for what you owe, and in some circumstances you can be prosecuted for using money that "was reasonably evident that you did not earn"
"This is the value of a summer spent and a winter earned"
Oh please... if the person on the phone knew anything about programming, they wouldn't be working the phones, they would be coding their apps like the guys who got promoted from answering the phones last week.
I Am My Own Worst Enemy
Maybe not the wrong currency soon, the way we're printing money here...
There is very little future in being right when your boss is wrong.
Holly: Busy, Dave?
Lister: Well, yeah. I am, actually.
Holly: Oh, then you won't want to know about the two super-lightspeed
fighters that are tracking us.
Lister: What?!
Holly: I'll leave you to your bubble blowing, mate.
Lister: No, Hol, come on, come on.
Holly: They're from Earth.
Lister: Three million years away?
Holly: They're from the NorWEB federation.
Lister: What's that?
Holly: The North Western Electricity Board. They want you, Dave.
Lister: Me? Why? What for?
Holly: For your crimes against humanity.
Lister: You what!
Holly: It seems when you left Earth three million years ago, you
left two half-eaten German sausages on a plate in your
kitchen.
Lister: Did I?
Holly: You know what happens to sausages left unattended for
three million years?
Lister: Yeah. They go all mouldy.
Holly: Your sausages, Dave, now cover seven-eighths of the Earth's
surface. Also you left seventeen pounds, fifty pence in a
bank account. Thanks to compound interest you now own
ninety-eight percent of all the world's wealth, but since
you've hoarded it for three million years nobody's got any
money except for you and NorWEB.
Lister: Why NorWEB?
Holly: You left a light on in the bathroom. I've got a final demand
here for one hundred and eighty billion pounds.
Lister: A hundred and eighty billion pounds! You're kidding!
Holly: (wearing Groucho Marx disguise) April fool.
Lister: But it's not April.
Holly: Yeah, I know, but I could hardly wait six months with a red-hot
jape like that under my belt.
Actually, I think it is more like 95% of the time So it's not so bad after all.
"In a statement, Visa said the rogue charges affected "fewer than 13,000 prepaid transactions" and resulted from a "temporary programming error at Visa Debit Processing Services ... [which] caused some transactions to be inaccurately posted to a small number of Visa prepaid accounts.""
I call bullshit, Visa. Don't you people have some basic QA? If, say, a monthly statement (especially on a PREPAID CARD, for frack's sake...) exceeds the spending potential of a given client, flag the statement and alert a regional or local processing center manager.
FRACK! At the very LEAST your programmers should have been told (or, if they asked, been allowed) to put QA bounding-box fields on the statements. If a monthly charge font size to be printed is longer than the width of the statement imaginary box, eject the statement from the enveloping system, then punt it to a manager.
Having even FIFTY of these things get out is unprofessional, and plain stupid. Unfortunately, some dumbos pay without checking, then may have to wait several days or weeks, only to be told they won't get a reversal, but only a credit to offset future purchases...
Previously: "Linux... Toward the Sunrise..." Now: "Linux... Toward the-- No, now, part of Every Sunrise"
This reminds me of the time I transferred 100,000 miles from my Amex rewards account to one of my airline mileage accounts. I initiated the transfer on the Amex website. When I logged into the airline web site to confirm the transfer, I saw that it had been transferred in increments: ;)
32767,
32767,
32767,
and finally 1699
Oh, and it wasn't as simple as padding with spaces. Space is hex 20. Zero is hex 30. They should have been been billed 30 quadrillion-something. More likely it was a bad conversion. Still reason to waterboard the testers.
You should try converting packed binary to some flavor of EBCDIC, not knowing in advance which particular version EBCDIC they meant.
deleting the extra space after periods so i can stay relevant, yeah.
What's really strange is they're using 64 bits to express a charge amount. How many people are charging manned missions to Mars or the military invasion of a superpower to their Visa? A 64 bit credit limit must be quite the status symbol.
FRACK! At the very LEAST your programmers should have been told (or, if they asked, been allowed) to put QA bounding-box fields on the statements. If a monthly charge font size to be printed is longer than the width of the statement imaginary box, eject the statement from the enveloping system, then punt it to a manager.
That isn't even close to how the financial organizations function. There is simply zero drive to pre-empt problems as there is no major authority breathing down their necks and auditing every single iteration of their customer-facing software processes in great detail.
Moreover, the customers are individuals or small businesses, meaning there is practically nothing to fear in the form of loss of business due to dissatisfied customerbase or defamation. It's not like they have too many other choices.
Yet Socrates himself is particularly missed.
A lovely little thinker but a bugger when he's pissed.
I had a roommate who had a calling card that had rolled over to maxint minutes remaining. He checked the balance on a speakerphone to prove it to me.
Oh, say does that Star-Spangled Banner entwine / The myrtle of Venus with Bacchus's vine?
Shouldn't have checked his email from Mexico.
Since our national debit has just hit 1 trillion dollars, we in the United States of American will be second to him in debit.
I'm very surprised that credit card company, bank or anybody else didn't have any alarm bells (more air raid sirens in this case) when this went through. Also I thought there will be limit on anyone could charge, not only the credit card, bank or even this case, the nation could get.
This shows there is something wrong with the financial system and that is the understatement of the century.
But just think of the airline miles!
Advice: on VPS providers
I work in this industry. The only novelty here is that the error got into production, and was not caught and corrected before it went that far.
That explains why there are so many software testers currently looking for work. The CC industry doesn't use as many of them, anymore.
Some submitters do, from time to time, change their code, and sometimes they get it wrong. For instance padding a field with spaces instead of zeros. Woopsie...!
You're still using legacy zero padding? You should be doing things in XML.
Seems that's what happened here. Sounds like a hex or dec field got padded with hex 20, and boom.
Not quite. If it were padded with space characters, you get 0x20 in each byte (and that's just what this number had in the first 6 of 8 bytes). If it were padded with zero characters, you would get 34,723,282,962,276,803.04 or so.
The REAL problem here is that the code was interpreting 64 bits as internal binary integer, when the data that arrived was at least 6 ASCII space characters. In other words, the data was in an old legacy format with space padding (which is easily handled by decimal conversion code along with zero padding), but the code expected a raw 64 bit integer, perhaps in the big-endian byte order.
This is annoying, especially when the processor gets to help correct the overwhelming number of errors, and then tries to explain that it wasn't their fault. Plenty of blame to go around with this one.
It was clearly someone's fault. Possibly a programmer not applying the proper field conversion? Perhaps the code was intended field conversion in place (replace the memory that has ASCII digit characters with the binary representation) and the conversion bailed out for some reason and the calling code didn't properly detect that (hint: in today's dollar amounts, the highest order byte of 64 bit integers should be zero).
Even if a programmer is to blame, management is not blameless. And maybe it's not even a programmer to blame. Ultimately, the real blame does lie with management who should have seen to it that errors never happen. Of course, errors do happen and while the blame may be correct, it's really cheaper apply 99.999% perfection instead of 100% perfection. It's not that serious a blame ... at least as long as problems get corrected.
And then explains why they don't both validate/sanitize input, and test for at least some reasonable maximum value in the transaction amount. A max amount of $10,000,000 would have fixed this. That and an obvious lapse in testing. This is what keeps my bosses awake sometimes, fearing they will end up on the front page of the fishwrap looking stupid 'cause their overworked minions screwed something up, or didn't check, or didn't test very well. I love one of the guys we have testing. He's insufferable, and he catches genuine show-stoppers on a regular basis. They can't pay him what he's been worth, literally $millions, just in avoiding downtime and re-working code that went too far down the wrong path.
I don't know that $10,000,000 would be high enough. The kind of error this one is could be detected with a test against 2 to the 56th power. A lower test might catch other errors. And whatever tests are done, they should be in their own class, module, function, macro, or whatever, and separate from the mainline code. Two tests should be applied, one being conservatively high but fixed by the coders (e.g. the 2 to the 56th power test) and the other being configurable for code used by multiple processors (they can choose their own closer to the edge sanity check).
Believe me, this is in some ways preferable to getting files with one byte wrong that doesn't show up for a month, or sending the wrong data format (hex instead of packed binary or EBCDIC, for instance) and crashing the process completely. Plea
now we need to go OSS in diesel cars
Yikes.
Padding is used in fixed-length fields.
There. Was that so hard?
Oh yeah, it is. Let me finish this.
'Our' submission files use combinations of fixed-length and variable-length fields. Fixed-length is easy, but variable-length is usually designated by a check byte that tells you what sort of data follows. Our submission file can overall be submitted as either fixed-length or variable length. Yes, the terms are mixed in the specification.
Why?
Well, among other things, many submission files include data intended for disparate systems. Some of these systems are new and purpose-built, and the data format is fairly efficient, but some systems are ancient and were never intended to do what they do today. It happens.
Some systems talk EBCDIC, some can't tolerate decimal, others are sensitive to input data. One can't pass anything above dec 0x127. One has to accept a binary blob. Yep, we have to parse that stuff out and mske sure it goes to the right system. Input validation is a bear.
I'm amused at this problem, as allowing that much field width for transaction amounts is pretty bad design. It is supremely unlikely to see a $10,000,000 charge for your typical cardholder. Even for your commercial cardholder. Just an unfortunate example of bad execution. Here, fixed-width is your friend.
And yes, an XML file would be magnificent. How would we improve things when most of our systems cannot parse XML? No, doing that much parsing in a separate system system is not worth the cycles, and interpreting data is left to the actual receiving system. And while XML is extensible, you don't want that in a submission file. It is supposed to conform to the specification.
As an analogy, you don't answer your teen-age daughter's phone calls and then scream the conversation to her and scream her responses back to the caller. When you realize the call (data field) is for her (other system) you just hand the phone (data) to her. Let her deal with it. If she has a question for you, like how much money can she have to buy something, you deal with that.
XML is not the solution. Neither is putting everything on one system. Those of you in the card business know why fraud stays off to the side. Like integrating anti-virus into Windows for instance. Once some processes are integrated into others, it becomes *one* process. Checks and balances disappear. Chaos reigns.
deleting the extra space after periods so i can stay relevant, yeah.
I'm afraid you're wrong, sir or madam.
I am one of the victims of this programming error, and I can tell you that several thousand VISA debit transactions were miscoded with the same amount: $23,148,855,308,184,500.00.
I was not smart enough to look at my card number before I sent it off to Consumerist so that VISA could be made fun of. Happily, the string does not contain my (or apparently anybody's) credit (or debit) card number.
Never attribute to malice that which can be explained by mere idiocy.