Slashdot Mirror
How Linux Mastered Wall Street
itwbennett writes "Linux has become a dominant player in finance thanks to its ability to pass messages very quickly, said Linux kernel contributor Christoph Lameter. 'The trading shops saw that the lowest-latency solutions would only be possible with Linux,' Lameter said. 'The older Unixes couldn't move as fast as Linux did.' One key attribute was the TCP/IP stack, the configuration of which determines how fast a message can be passed between two systems. Linux also offers financial firms the ability to modify the source code to further speed performance. 'It depends on how daring the exchange is,' he said, noting that NASDAQ uses a modified version of the Gentoo Linux distribution. Lameter will discuss how Linux became widely adopted by financial exchanges at the LinuxCon conference in Vancouver this week."
Oh dear god our economy is a whole lot more fragile than I ever imagined. Brings a new meaning to "emerge world".
But I don't see too many thinking that's a big deal, either !!
I still wont trust WallStreet with my money, which would be about like trusting an alcoholic with the beer at a party...
Politics is Treachery, Religion is Brainwashing
lets just ignore the fact that the Great Recession was directly enabled by the PHDs who turned their eyes askance at what the Gaussian Copula Function code was being used for. Not your problem right? You just make a tool, not your responsibility how others use it.
you had five kids to feed.
Linux is extremely good at networking. Stick the LGPL zeromq library on top and you get low latency messaging middleware. Afaik this is also really popular in financial IT systems. People are doing 5M-8M messages per second with it.
Since when has Slashdot started to think that Wall Street has favored quality over price? Oh, that's right. This is Linux. I guess if it was about anything else a thousand fanbois would be screaming about kick backs and profit motives.
Quick, hackers! Start assimilating the Linux source code with invisible patches that take every 0.1% of every transaction running on Wall Street and donate that money to the poor countries!
Insert code to take down the whole corrupt ponzi scheme that those heartless bastards are running there! Now is our chance to strike back! Think about the possibilities here people!
GeoKone.NET
As mentioned in the summary, linux allows the firms to modify the OS kernel to serve their purpose. For example, the performance of trading algorithms would considerably degrade if context switching is allowed. So you can modify the kernel so as to dedicate certain cores for the main algorithms to which the OS can pass a very limited number of signals.
I know for a fact that at least one bank employs this in their high frequency trading group and probably all of them do.
This is the very last place I like seeing Linux.
The article is saying (obviously) that Linux is the chosen platform for high-frequency trading, i.e. algorithm-dominated trading that has everything to do with manipulating and responding to the market in nanosecond time frames for a quick buck and nothing to do with making stable, long-term investment decisions.
I'd rather see evidence of a Linux machine in Hitler's bunker than hear about Linux helping Wall Street punks get even further from real, useful activities than they used to be.
I don't think that IBM will be pleased to be told that HP produced AIX!
are the only reason that Goldman exists right now.
It's sad to see how much money Wall Street firms were able to generate using this open source platform. We have yet to see what the open source communities get from these welches!
Development and administration at Fortune 100 companies in Manhattan is different than any other place I encountered, including other large companies. There is a lot of message-oriented middleware to patch together different systems.
You'd see a lot of strange stuff - a batch job printing from an IBM mainframe would be routed to the Unix print server, and be sent off to a junky old printer in some foreign country. Not always easy to debug when there is a problem.
Where I was, there were a ton of these old programs written in FORTRAN, COBOL and whatnot which had had business logic put in them for decades sitting on these modern IBM mainframes. Some of the business logic within it was probably lost long ago, it all just "worked", with a lot of the output routed to more modern equipment and technology. I guess they figure if anything ever goes wrong, they have almost unlimited money to throw at the problem so they don't worry about it.
You also have things happen. A business group has their developers write some program, it goes production on a machine or two, and then for whatever reason it generates a lot of money. Suddenly you have millions, sometimes even billions of dollars going over one production machine in a day. Everything happens so fast that it was never planned out to be scalable, and the main developer is too busy tweaking the program to make it make more money than to be scalable etc. If you're lucky, its market is closed during the week and you get to work on adding in additional levels of redundancy to the machine which suddenly has billions flowing through it every day. Despite the lack of planning, you better bet people will be flipping out if the machine goes down during the day, and the traders hear that their trades aren't going through due to "computer problems".
At the Fortune 100 financial I was at, Windows was considered a joke. Even the local head of the Windows team admitted that the Unix side was where things were really happening. It was just more flexible, focused on high availability and so on. With Linux coming in so much on the Unix side, that flexibility has only increased. I'm sure whatever RHEL or SUSE edition being run on most servers is so heavily modified internally by the various companies internal engineering teams, that it doesn't look like a RHEL or SUSE anyone here has ever seen. And RHEL and SUSE bend over backwards to get the business - which can be on tens of thousands, even hundreds of thousands of machines around the world.
I work for a major telco and we use Linux for almost all of our production equipment. In the early days, I thought I would have trouble with the concept of using Linux, but when I began investigating and found the major stock exchanges and many other major stock brokers were already using Linux, it made the job of convincing management a whole lot easier. Since Oracle has swallowed up Sun and their support has gone down hill, having two production systems recently trashed by incompetent engineers (one who couldn't even change a failed mirror disk without wrecking the whole system), has made us all the more determined. I am looking forward to a 100% Linux environment. It simplifies training and support too.
Not Wall Street or finance.
http://saveie6.com/
They're still the master. Linux is their eager little pet, far more willing to learn new tricks than their old pet Solaris.
but not relatively. The general economic downturn affected them as well, but in relative terms their wealth ballooned. They got rich off of the whole thing even without a bailout.
This high frequency trading is stupid. Everyone knows that it is a scam that is just making the markets more unstable. Yet no one does anything about it. IMHO the markets should have a clock speed like a CPU. All of the trades enter a queue and the queue gets executed once a second. This would limit each trading day to X number of ticks per day. This would go a long way to removing high frequency crap from the system. Of course people will then try to improve short term predictions rather than long term like they should. But it would be a step in the right direction.
well that seems retarded if you are so interested in speed. And I would think any of the *bsd's would perform just as well as linux and might be more secure too.
were based on credit default swaps, which are basically, gambling, and have no relationship to any actual cash flow.
and a lot of the mortgage securities were based on fraud, and the ratings agencies purposely did not even look into them to see what kind of loans they had (let alone what kind of credit default swaps they had... )
and the bankers payed them to rate this stuff very highly.
there are dozens of books about this. its fraud, not a 'mathematical mistake' or 'not understanding math'. the CDO desk bankers who made millions of bonuses on 'futurue projected profits' while they destroyed their own companies understood the math extremely well.
about the crisis. including
On the Brink by Henry Paulson, who was CEO of Goldman until 2006, and sec. of treasury during the crash
Too Big to Fail, Andrew Ross Sorkin
etc etc etc.
if Merrill hadn't been bought by BoA, and Morgan Stanley hadn't been bailed out by the Japanese banks, then Goldman would have fallen soon after.
Goldman got its credit default swap deals with AIG payed off, 100 cents on the dollars, when they werent even worth a fraction of that... payed off when taxpayers bailed out AIG. if AIG had gone down, goldman would have gone down, crash, explosion. none of its 'big shorts' (or 'hedges' as lloyd blankfein likes to describe it, but alot of other people dont) would have worked.
they were for house flipping and cash out refinancing.
and Synthetic CDOs are not based on mortgages, they are based on credit default swaps.
-reference
All the Devils are Here, Nocera and McLean
there are people with lots of savings and lots of education and lots of experienece who became unemployed, because of these games being played by others.
if they had 'put their money in the savings account', it wouldnt have helped them one iota when the economic system went into recession.
Maybe I'm old school, but I remember investing in the early part of the dot-com bubble. Making a trade cost $200 - $400 in commission, not including the fact that the spread was 1/8 (12.5 cents) wide. So I don't mind these guys getting me $7.95 trades with a penny spread. One way, somebody on wall street is going to take a cut. Whether it's the specialist, market maker, or brokerage, retail investors always get screwed a little. At least now it's a lot less than I used to get screwed for.
Is Google big enough for you? If they're betting on Linux, then I'm comfortable with it!
What occurs to me is that 15-20 years ago, when Sun iron dominated Wall Street, is that all the Sparcstations came IIRC with 10baseT on the motherboard, and I don't recall there being faster NICs available. And even if there were, in a lot of cases all the machines slots were populated with graphics cards.
Then three things all sort of happened around the same time: Linux (kernel and user land) reached a level of maturity and stability, inexpensive 100baseT and later 1000baseT NICs became available, and Intel closed the CISC/RISC performance gap. I don't know when Sun started shipping faster networking, but if the only way to get it was to buy a whole new machine, as opposed to plugging in a $30 NIC, it isn't hard to imagine which way the purchasing decision would go, even for money-is-no-object Wall Street.
No surprise then that Linux whupped Solaris' butt. Sun didn't help things by dawdling on fixing known Solaris performance problems. By the time Sun fixed them it was too late; Linux had already gained a foothold.
I work at a major wall street bank. We used to be heavily Sparc/Solaris/C++. Over time the Intel platforms became much faster and much cheaper than the Sparc ones. There was some early concerns about reliability but it was not warranted. The boxes are so fast now we are almost exclusively using virtual linux boxes too.
We are doing a lot of Java these days. The JVM's are much improved. It is very easy to write large heavily multithreaded Java apps to replace the our large C++ distributed systems. The Java development, build, debug, and deployment tools are great.
One can spend time arguing the merits of C++ vs Java. The reality is in most cases the C++ development time is slower, and the coding patterns used do not produce code that is faster than Java. C++ development and deployment across different platforms is a pain.
Hey look, we made it possible for the already wealthy to have an unfair advantage over those without the money for supercomputers and millisecond trading!
That's exactly right. Linux and Wall St. are actually very similar in this regard. Someone is going to make money providing support regardless, because there is a market for it. Either you accept a monopoly (or an oligopoly), and prices eternally rise. Or you embrace the free market, and prices fall as innovation and competition improve quality. Complaining that someone makes money using algorithmic trading is like complaining that RedHat sells free software for $400 a seat.
"I assumed blithely that there were no elves out there in the darkness"
It really comes down to a man in the middle attack.
The message is coming through that you want to buy shares, so some bastard in the middle buys them before you can get them and sells them to you at an inflated price.
I'll leave whether it is a level morally above or below using keyloggers to find credit card details as an exercise for the reader.
I would like to know what a modified version of Gentoo is. Since it is installed from source and usually requires the kernel to be compiled during installation every Gentoo installation is modified. Would be nice to find out more details on what they have done to make it go faster.
Not surprising to see GNU/Linux popping up in finance world. Not just trading, but banking, insurance, government finance. IBM: You can have IFL processors on your mainframe (Integrated Facility for Linux) with or without z/VM, and there is Linux on zSeries. You can run Linux on your System x or System p.
Oracle/Sun: solaris or linux on your Oracle x86 box. Unbreakable Linux for Oracle.
HP: Redhat and SuSE were supported on Integrity servers (not to be supported on the new 9300, but that's another story), on x86 you can have Redhat, SuSE, Debian, Oracle Linux
Linux also offers financial firms the ability to modify the source code to further speed performance.
So does anyone have a link to any source code that was made open under the license? They may be taking our money, but at least they are giving us code, right?
Really? There's hardly a default version. I wonder if their customization fell within the bounds of Gentoo's normal options, or they had to create their own overlay. I wouldn't be surprised if they are in fact running vanilla Gentoo.
Does anyone know, if FreeBSD or any other BSDs have still faster TCP/IP processing than Linux?
I think is was last year or something. They said they first tried Windows, which crashed and then used some Linux as their OS for more reliability.
http://www.zdnet.com/blog/open-source/the-london-stock-exchange-moves-to-novell-linux/8285
Don't quite understand this. The implication is that the time spent processing the message in the TCP/IP stack (and not the network latency itself) is a significant percentage of the overall transaction time. Which seems unlikely. Also they have (say) 1000 front-ends connecting to 100 matching engines. So why not simply add extra cores to the front-ends and do the matching there? Just cut out the network latency altogether. I know they say the traders can't connect directly to the matching engines but that's just semantics - there's no difference between having the separation physically and within the same machine.
According to the article a Microsoft spokesman mentioned KAS Bank as one of their customers in the financial services industry. I have worked there in the past, and still hear stories about what's happening there from former collegues.
KAS Bank specializes in post-trade services (clearing and settlement), for which they offer a single interface to the European markets. They partner with Sungard for linking high speed traders to Europe, but that only involves the post-trade services, not the high speed trading itself. They are a mainframe site in the process of moving to dotNet and a number of Microsoft products such as BizTalk. The mainframe hardware has been outsourced, but the bulk of the business logic is stil in mainframe applications. While part of the message handling is now implemented in BizTalk, the settlement process is still running on the mainframe.
The transition to MS is a difficult one. I gather they are recovering from a period with very MS centric IT management (now gone) who apparently were blind to the fact what they looked down upon as an overexpensive, archaic and rather small server actually ran an amount of complex business logic beyond their comprehension. I would agree that a move to a more modern software platform was overdue, but it appears they hugely underestimated what was accomplished on the old platform. I get the impression that at least some of the people from Microsoft involved also have the problem that IT outside of their platform doesn't really exist for them.
I think it's disingenious for Microsoft to use KAS Bank as an example. The performance requirements aren't as extreme as the subject suggests and it isn't even their platform that does the heavy lifting. Is that the best example they can come up with?
i have a few Gs in a savings account but i could pull it out any time i wanted it, and there would more than likely be a few bucks in interest added to it...
I used to be like that when I was younger. A few Gs in a savings accounts. It is fucking sad, and reckless as a few Gs can evaporate quickly in an emergency. I say this in retrospect because I've been there. Luckily I grew up. Even in these times, working in software is one of the most profitable and safe careers there is. Anyone who works in software and only has a few Gs in savings is doing something very stupid with his finances.
Quick trading is nothing but speculation. I think there must be some cooldowns in place to slowdown things, so you can't buy at 3.88 then sell at 3.89 after 3 seconds ( or 0.0003 seconds).
-Woof woof woof!
Don't try and add a scientific polish to something that is nothing more than glorified horse trading. There is no "physics"; waves are predictable, the stock market as has been proved time and again - is not.
You mean sponsors of government policies that allowed anyone breathing to get a loan and Dodd and Frank amongst them?
I totally agree with you.
Structured Finance and Collateralized Debt Obligations - Janet Tavakoli
Lost Trust - Lang Gibson (CDO expert who worked for Merrill Lynch)
The Trillion Dollar Meltdown - Charles R Morris
A Colossal Failure of Common Sense - Lawrence McDonald (former Lehman bond trader) and Pat Robinson
Confidence Game - Christine S Richard
EConned - Yves Smith
Diary of a Very Bad Year - Anonymous Hedge Fund Manager + Keith Gessen
The Big Short - Michael Lewis
All the Devils are Here - Bethany McLean and Joe Nocera
The Zeroes - Randall Lane
On the Brink - Henry Paulson
Crash of the Titans - Greg Farrell
How I Caused the Credit Crunch (novel) - Tetsuya Ishikawa (ex-Goldman worker)
Mark Williams - Uncontrolled Risk
Vicky Ward - Devil's Casino
Chasing Goldman Sachs - Suzanne McGee
And Then The Roof Caved In - David Faber
Inside Job - Charles Ferguson (film)
Panic - Andrew Redleaf
Too Big to Fail - Andrew Ross Sorkin
The Sellout - Charles Gasparino
Street Fighters - Kate Kelley
"1. Synthetic products were not significant, and certainly not a significant cause of the crisis. Cash products are the ones that created excess demand for real mortgages."
--- The books that mention Synthetics, especially Tavakoli's, do not agree with this.
"2. "Gambling" is a subjective term with negative connotations. I claim that any definition of "gambling" you can post will either be far to broad, or not include asset-backed securities or their synthetic counterparts. "Investing" is usually the prefered term for risk-taking in financial products. "Gambling" usually refers to games of chance, and betting on games of skill or sports."
-- Gambling is not my phrase alone, several of these authors use the word, most notable Tavakoli and McDonald, both of whom have decades of experience in high finance. There is also a definition of gambling and speculation and how they compare to legitimate businesss activity or hedging, given in Edward Chancellors ' Devil Take the Hindmost ' -- a history of financial bubbles
"3. Synthetic CDOs are based entirely on real cashflows."
The 'flow of cash' in a CDS does not come from someone who bought a product, it comes from one counterparty who is gambling that a certain debt will never be payed off, to another counterparty, neither of which need to have any relationship, whatseover, with the debt they are betting on. This is described in Janet Tavakoli's book in great detail. Synthetic CDOs are made of CDS, thats why they are gambling.
"Second, I'm sure it's completely false that ratings agencies didn't look at the underlying loans."
Michael Lewis ver batim quotes an internal memo from ratings agencies in The Big Short, where a manager tells the workers that they do not have any underlying loan data, and that the products need to be rated anyways.
"False, but points to a truth. First, ratings are always paid by the issuer (note I didn't say "banker" since banks aren't the only issuer). Remove the words "very highly": the issuer doesn't get to pick their rating. They can haggle "
They 'haggled' by saying that if the rating agency didnt rate the bond highly enough, then the bank would go to another ratings company. This is well documented in books like The Big Short.
"I can assure you that the books were written to generate sales, rather than provide an accurate historical record"
Wow. I dont know what to tell you. If you beleive the entire industry of journalism, including hundreds of articles describing in detail the dozens of CDO frauds like ABACUS, Magnetar Capital, etc, and that its all a conspiracy theory then I guess I have nothing to argue against it.
"And so did the CDO desks that made tons of money for their companies. And I think you gave credit to the desks where it's due: "
Every thing ever written about the fall of Merill Lynch points directly to the CDO desk. The same could be said of Lehman and others. In fact, that is Goldman and Detuschebanks' defense of some of their acti
and these models were the fundamental enabler of the fraud.
these models were directly linked, in a fundamental manner, to the corrupt and conflicted relationship between the banks, the ratings agencies, and the insurance companies.
the models were created by a collaboration between the ratings agencies and the banks. the banks also employed people to 'game' the models. and the banks also payed the ratings agencies based on the results of those models calculations.
it was massive, industry wide fraud at a fundamental level. without that fraud, there could have been no massive amount of bad debt pile up.
one great example of this is the story of Magnetar Capital ----- if it hadn't been for the fraud of the 2006/2007 era, then the bubble would have collapsed much, much sooner. but it kept inflating and inflating - because synthetic CDOs could continue being created and rated highly and sold to investors, based on the fraudulent use of these models.
you've not heard of microsecond trading before?
Why this obsession with high frequency trading among nerds? It's no different from any other industrial equipment.
I cannot build a car in my garage shop because I do not have the high-power presses needed to stamp structural steel body parts. I cannot make a million dollars in my home office because I do not have the billion dollars it take to effectively use high frequency trading.
I could build a fiberglass kit car or invest my savings on the stock market, but it will not bring the same results.
It takes big investment to profit from economy of scale, it's as simple as that.
before it started. lets take hedge fund manager Bill Ackman for example.
he was telling anyone who would listen that the Monoline insurance companies were based on fraud. they were insuring stuff they were never supposed to have insured, risky stuff, and then lying to investors about it.
what happened to him? He got investigated by the SEC. he could have gone to jail for a long time. the monolines didnt investigated, the ratings agencies didnt get investigated, the New York State Insurance regulators didnt get investigated for allowing this.... Bill Ackman got investigated.
the monolines were like mini-AIGs - they enabled the fraud of the CDO market to continue even after AIG stopped insuring certain CDOs at the end of 2005. the monolines continued, and helped push the bubble up. then they all collapsed and died in 2009 - they were too small to get bailed out. their only mistake was not being bigger.
its described in Confidence Game by Christine S Richard
the 'bullshit morality' of Ackman , well, it actually made him a lot of money.
the same for the 3 main groups in the Big Short .... they tried to warn the government about what was happening, and journalists, and they were largely ignored. their 'bullshit morality' made them a lot of money too.
of course some people dont think these 'big shorts' behaved morally..... because they profited from the crash (Yves Smith thinks they helped it... because Synth CDOs were made up of their bets (CDS))
anyways.
there were others. Janet Tavakoli in particular wrote some textbooks on CDOs , describing a lot of the fraud that was inside the business. Her 'bullshit morality' might not get her the fame and TV spots, but then again, her type of 'bullshit morality' is the foundation of all modern civilization.
fraud is not the same thing as leverage. fraud enables leverage.
and these quants enabled the fraud. while looking the other way.
if you are smart enough to figure out all of these complicated algorithms and math to model CDOs then you were smart enough to ask some basic fundmanetal questions, like 'where are the fucking loan tapes'.
in fact, some people at JP Morgan did exactly that, and thats why for a good while, JP Morgan did not get involved in any Mortgage CDOs. they knew they were crappy, risky products, incredibly overvalued.
its all in 'Fool's Gold' by Gillian Tett. . . JP Morgan actually invented the precursor to the Synthetic CDO, they called it a BISTRO and it was just a way to market Credit Default Swaps.
this shit is not complicated. fraud is not complcated. only the justifications, rationalizations, and lies that are piled on top of it are complicated.
I do not want to directly oppose some of the misguided views here but I see a lot of hatred for HFT, finance etc but at a very basic level there should be no regulation or hindrance for 2 ppl to exchange value (trade) as long as they believe they are deriving some value out of the transaction they are being part of even if others believe that the transaction is of no benefit (after all who can decide what's of benefit or not).. The idea of trade (any form of trade or barter) from an individual perspective is not for the betterment of society but for their own good but the unintended consequence is advantageous to society as a whole.. People might hate globalisation, free flow of capital etc but there is no denying that the onslaught of globalisation has reduced global poverty but I agree this has been at the price of increased inequality... Increased inequality in my opinion is better than all being equally poor... And for ppl who would disagree that global poverty has reduced just check world bank data on all statistics such as nutrition, health care, life span, access to electricity, telecommunication etc and you will see all there has been a remarkable rise in these factor over the last 30 years all over the world...
And you said it damn well!
Does the interest you receive even keep up with inflation? I doubt it.
Most of the financial market ran on Unix. Linux is gradually replacing Unix everywhere. The growth in Linux has mostly been at the expense of Unix.
Have any of these companies contributed their kernel changes back to the community?
Your darling Republicans arent exactly innocent. Google bush and "ownership society". Nuff said.
Monstar L
TCP !!! Latency Sensitive!!! Are you kidding??? nanex
I suggest you read this and learn something:
http://economix.blogs.nytimes.com/2010/01/13/traders-vs-chimps/
like Lawrence McDonald (Lehman bond trader, wrote Colossal Failure of Common Sense) and Janet Tavakoli, (wrote textbooks on Structured Finance and CDOs)
the calling of 'gambling' as 'insurance' is a piece of propaganda, that goes back to the 1800s, when the "Policy Shops" sold "insurance policies" that certain lottery numbers would come up (and sold these 'policies' to people too poor to buy real lottery tickets)
the thing that makes CDS 'gambling' is that the two parties exchanging cash don't have to have any relationship, whatsoever, to the thing they are gambling on.
sometimes CDS are used as a hedge, but the vast majority of CDS in the bubble period (which were the guts of Synthetic CDOs) had nothing to do with hedging.
and most of this 'hedging' argument is used by people in positions like Llloyd Blankfein , being dragged in front of Congress to explain why his company was selling CDOs that his own people called 'crap' as though they were good investments.
Let me explain. Those people were talking about real problems and incidentally, had real solutions. Superficially, your point of view does look a lot like them. The difference is that you are witch-hunting, blaming people that had nothing to do with the fraud that typically accompanies large bubble collapses.
Ignorance of who to blame greatly weakens your case.