Slashdot Mirror
Programming Bug Costs Citigroup $7M After Legit Transactions Mistaken For Test Data For 15 Years (theregister.co.uk)
An anonymous reader shares a report on The Register:A programming blunder in its reporting software has led to Citigroup being fined $7m. According to the US Securities and Exchange Commission (SEC), that error [PDF] resulted in the financial regulator being sent incomplete "blue sheet" information for a remarkable 15 years -- from May 1999 to April 2014. The mistake was discovered by Citigroup itself when it was asked to send a large but precise chunk of trading data to the SEC in April 2014 and asked its technical support team to help identify which internal ID numbers they should run a request on. That team quickly noticed that some branches' trades were not being included in the automated system and alerted those above them. Four days later a patch was in place, but it wasn't until eight months later that the company received a formal report noting that the error had affected SEC reports going back more than a decade. The next month, January 2015, Citigroup fessed up to the SEC.The glitch resided in new alphanumeric branch codes that the bank had introduced in the mid-1990s. The program code filtered out any transactions that were given three-digit branch codes from 089 to 100 and used those prefixes for testing purposes. The report adds, "But in 1998, the company started using alphanumeric branch codes as it expanded its business. Among them were the codes 10B, 10C and so on, which the system treated as being within the excluded range, and so their transactions were removed from any reports sent to the SEC."
EBCIDIC
As would having a separate system for testing, rather than trying to create a test island within your actual real live [that's enough - Ed] production system.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
They could just be comparing the strings dictionary order:
if ((strcmp(x, "089") > 0) && (strcmp(x, "100") < 0))
{
filter_test_data(x);
}
The above works for ASCII, but not EBCDIC
EBCDIC
I have a feeling that the whole "Nobody ever got fired for choosing IBM" thing is about to become very untrue at Citigroup....
Check out my sci-fi/humor trilogy at PatriotsBooks.
Anyone who has worked in the finance industry on the tech side of things has probably seen eye-searing levels of problems like this. It's clusterfucks all the way down. It always surprised me that something that seems like such a natural fit for software was always, without fail, so riddled with glaring bugs that it's almost unfathomable that you are the first person to notice them. At a lot of shops, the bugs are so ingrained in the process that they can't even be fixed. Working in the finance industry certainly doesn't inspire confidence in the finance industry.
Just curious, are you drunk, or are you stoned?
Citigroup's trading and reporting systems aren't open source, quite the opposite; they are closely guarded secrets.
Are you confused because the output is required to be in a specified format?
The banksters are simply too big to jail and too big to even question. Break up the banks.
"Honey I shrunk the government. And the banksters drowned it in the bathtub!"
sed -e 's/Chuck Norris/Rajnikant/g' joke > fact
"He was paid well to provide software that functioned to the defined spec, and he failed to do that."
Not correct at all. The software (apparently) worked well according to the original specification. Then they extended their business to open new branches, but did not adequately update the software. Not a problem of the programmer, but a problem of change management. You could just as well complain that your toy tricycle is not safe on the highway - possibly quite correct, but it's your fault if you are operating it outside its specification ("use only by children up to 30 kg on the sidewalk"), not the tricycle engineers problem.
Stephan
which eventually became AOL, we were routinely sent CDs with patches on them. Eventually we got the CDs that would patch our beta releases to become public release apps. As beta testers the service was charged at half price. Almost a year into the public release, I got a phone call from Steve, the boss at Quantum, letting me know that the one thing they forgot to patch in the upgrade CDs was the switch to full price. So would you please cut us a check for everything you paid us already for the past year.? Um, no... by the way how many users did this affect? We're not sure. Dozens? Well yeah. Hundreds? Yeah. Thousands? Look, that's not the important part. I believe I offered to pay double the monthly bill until I was caught up. Never heard back, next release placed us at full charges. I bailed once it was AOL, and it was back to Delphi and The WELL.
"Win treats sysadmins better than users. Mac treats users better than sysadmins. Linux treats everyone like sysadmins."
This is a perfect illustration of why "smart" IDs are a bad idea. Any time you encode attributes (like "this is a test transaction") into an ID (like a range of bank branch IDs) you are asking for trouble. Everybody does it, but it's usually just plain lazy and careless. DON'T! Add an attribute that marks the transaction as a test transaction! Then anybody who sees it will instantly know the difference.