Wall St. Trading Servers To Power Off-Hour Clouds?
miller60 writes "As cloud computing gains traction, some Wall Street firms running armadas of servers to power high-frequency trading operations are contemplating leasing out their excess computing capacity after the trading day ends at 4 p.m. 'Once 4:30 rolls around, we don't need those machines,' said one CTO of a market data firm. 'There may be an opportunity there.' A similar revelation led to the creation of the cloud computing operation at Amazon.com, which built its infrastructure to handle peak Christmas-season loads that lasted just a few weeks each year."
I wonder if this poses any security concerns or problems? Is letting 'cloud users' access the servers that run out financial markets really a good idea?
This seems like a security nightmare waiting to happen. I understand everything would likely be virtualized, but it just makes me nervous that you would be able to rent time on servers that interact with the stock market, especially considering how panicky the market can be, and how badly everyone suffers when it does panic.
Compuserve was founded by a life insurance company to make a consumer service run on their business systems with the expectation that the consumers would use it during non-working hours.
When it comes down to it, nobody needs their clock cycles 24/7 at even load, even though that's what computers are designed to do. Shared services for the win!
The cost of renting the server farm has to be twice as much as the savings they'd have if they just powered them off.
Maybe even a little more, so that someone makes a profit.
I still don't understand why the Stock Market can't run 24/7.
Nothing like lending out some hardware to save a few $$ but potentially risking your account(s) to some haxors? As to saying virtual machines can't be haxored, that is incorrect. They are simply more difficult to get to.
This idea is similar to leasing out IRS or NSA free-computer time in the off-hours. After all, IRS peak is at tax time?
A better idea for them maybe just to turn the machines off when not in use???? I know, it's radical.
$20 says this idea was cooked up by someone who heard about "cloud computing" on the radio while in his cushy office, signing official looking papers and making a big fuss about "revenue".
Living With a Nerd
Oh yeah, that sounds... like a disaster waiting to happen.
I sure hope Wall Street is utterly confident in the security of their operating systems, VMs, low-level peripheral firmware, etc. Because if they're not absolutely confident, they should treat all of those machines as potentially untrusted from the moment they open them up to the world. This holds even if they constantly re-image.
When you're talking about the kind of money Wall Street stands to lose from a clever security breach, no amount of paranoia is too ridiculous.
Not to derail the conversation, but high frequency trading doesn't contribute much to the stock market's ability to set optimal prices.
What actually happens is that high frequency traders squeeze in while prices are moving and they siphon off money. Neither the original seller, nor the original buyer gets the best price, and the high frequency traders make a mint.
[Fuck Beta]
o0t!
Said the CTO who is now looking for a job.
NYSE closes at 4:30. But there are other markets. And the data flows 24/7.
There is no reason for these systems to have spare cycles.
by running an extension cord out to the sidewalk and running my batch jobs on a PC sitting outside of my house. That way, I stand a chance of actually seeing if anyone is copying my stuff off if I wake up in the middle of the night to get a glass of water.
Yes. If Amazon went down tomorrow and never came back, society would be fine. If the stock exchange were taken over by malicious but hidden computer software for months, and then finally was taken down, the damage to society would be MUCH more severe. It's not just a way of exchanging everything, it's a way of establishing who owns what. If suddenly nobody knows who really owns every stock that's traded in the last six months, we've got a major frikkin problem. We shouldn't, maybe, but we do because money is an illusion.
-- IANAL, this isn't legal advice, and definitely isn't legal advice for you. Also, Squee!
consult with his technical people.
What am I thinking, Of course not.
The Kruger Dunning explains most post on
I'm sure that there are lots of people who would like to run jobs on machines that normally process billions of dollars in transactions.
Of course, a certain percentage of them are going to be probing for security holes so they can steal financial data, but no worries -- the government will bail them out, right?
Proud member of the Weirdo-American community.
I dream that this entire high frequency scam is declared illegal and all involved are placed where they belong with all of their property confiscated.
Here is what happens when HFT is done: 2 parties agree on a price, the HFT meddles with in a way, that takes out money from the transaction, so the buyer sells lower and the seller buys higher. That little bit of difference is stolen by HFT.
These are thieves, we are discussing here, understand that. So they found a way to make some profit on their infrastructure? Well, great for them. 4,000,000 transactions per second they are talking about for one shop. That's 4,000,000 thefts per second.
You can't handle the truth.
I guess if an idea is 20+ years old the statute of limitations has run out and someone can use it again as NEW and EXCITING.
Sounds like a great way to do the 'ol "Transfer .01 cents to my account on every trade" trick.
Can someone post a source for the claim that Amazon did EC2 because of Christmas time servers that are no longer in use during the year?
in girum imus nocte et consumimur igni
Since most computing power is needed during the business day, during the same time that they need their processing power ... it would seem they have little value in what they have to offer.
What do you do during the peak time when they don't offer you processing time?
This pretty much limits the data that can be handled on these servers to batch processing of data that doesn't have constraints on how long it can wait.
Basically its useful for low budget scientific research. Thats not saying its not very useful in that sense. Just recouping wasted CPU cycles could be great for science in general. I just don't see there being enough of a market for a stock exchange to justify the expense of running the business.
Persistent Volume manager for Kubernetes - https://github.com/dwimsey/openshift-pvmanager
I've been in trading business for a while but never met him. Still "press" keep talking about him. Especially when it comes to getting someone to pay for something guvmint wants. Enyone met him yet?
America Online (AOL) was started in this way by Quantum Computer Services looking for a revenue stream during the night.
It could be secure, given the following constraints:
1. The computers contain NO mass storage at all.
2. The mass storage is external to the computer and is disconnected from the computer during cloud computing.
3. The computer is rebooted from CD (or DVD) before and after cloud computing.
Of course, the odds on those constraints being met are pretty low if non-technical types are involved.
linquendum tondere
These servers make a *ridiculous* amount of money for their owners/shareholders. Revenue from hosted services would be a drop in the bucket in comparison....no a single water molecule in a bucket. Leasing out their servers in the off hours would barely pay for the brokers cocaine-and-hookers slush fund. Hardly enough to warrant the effort and associated risk.
It seems like a novel idea, but in the end I bet it remains just that.
This might explain how Wikileaks did it.
It would be great for batch rendering. Sure, the hardware might not be optimized for it, but it would probably be cheaper than renting one that is. And running it during off-hours is actually a plus: set up a scene during the day, render overnight, then see the results in the morning.
...said one CTO of a market data firm...
Note that it wasn't the CTOs of the actual trading firms speaking. In reality, almost every cycle that's not being used during the trading day is being used either in trading on foreign markets when yours are closed or running stats to drive the next day's domestic trading. A "market data" firm? Yeah, I can believe that. Real trading companies? Not so much.
That is all.
Is the Microsoft Azure cloud software available to install and run on these Wall St server farms during the off hours? I could see running the two separate installs on two separate fiberchannel SANs, and physically switching between them (plugging cables) and rebooting/reflashing to ensure none of either was left available to the other during the alternate operating cycles. A lot of Wall St server farms are optimized to run Windows, because otherwise the farm is too slow. If a local install of an Azure cloud could be quickly deployed and removed, that could be another huge new Microsoft product. Or, if these farms let MS operate it, a huge MS cloud service for which it doesn't have to pay for the hardware.
--
make install -not war
But so far as I know Amazon doesn't shutdown the cloud at Christmas? Meaning they have to have enough capacity now to handle both the cloud AND their peak Christmas load. Well I guess they could sell the spare capacity for cloud computing the rest of the year... Recursion, anyone?
It's the original CompuServ business model all over again!
We really have come full circle.
--
BMO
...and he is named "That Guy", suffers from bone-itis and wears a yellow tie.
I visited this talk and the reporting is just wrong. They just said they have a lot of time CPU are idle.. that could be used.. Too many security issues involved to be reality today.
I could easily see them doing this via sanitized OS images with no local disk access for PXE boot. People have been doing similar things for private intranet cluster/cloud platforms. You simply set the BIOS to boot to PXE if a magic packet comes in, and fire up the bastards late at night. Close out and shutdown before the grunts come in to work. Simple and a good reuse of generic company resources, provided your network is up to snuff (multicast is your friend).
Actually this is nothing new... a lot of companies specially in the finance sectors have adopted this method to use the computing power of idle CPU's. Financial companies have a lot of data to be processed, and a lot of it can be done off hours. A lot of them have devised in house distributed software which uses the computing power of "idle" desktops of employees to run tasks. Though this might seem to be a "security" risk by running data on random set of machines, it can be taken care of by running the distributed task under special privalges, and making sure that the task cleans up the memory and if need be deletes itself after the computation has been done. These special privialges could vary from the task can only connect to network on a specified port, it cannot write to the hard disk etc.
When it comes down to it, nobody needs their clock cycles 24/7 at even load, even though that's what computers are designed to do.
Except for anyone with more work than resources batched up, like Pixar, oil & gas exploration, the NSA, etc.
You know, those organizations that set up compute grids before they were called "clouds."
that old granddaddy of online services - the pre-Internet era CompuServe and others ran basically on the same idea. Most of the compute capacity would be used for business clients in the daytime and then give way to the evening and late night use by the paying public to access the various forums and SIGs that were hosted there. Sure there would be some users in the daytime hours however it would not have been as heavy as the regular hours. I'm certain there were security concerns then as well as those early servers were hacked quite regularly in various ways contrary to what the companies running them might say.
These Wall Street chaps know what they're doing! Their immaculate track record speaks for itself!
Nobody would bet their farm (pun intended) running someone else's software on their critically important server farm. Nor would any admin worth this salt invite that amount of someone else's traffic onto the network they use for trading. Or limit the already-often-restrictive windows of service time. The biggest flaw in this supposition is that most firms who have this much infrastructure are big enough that their trading isn't limited to US markets, and so would be trading over many time zones, limiting the useful downtims.
They would have to devise not only a proper time sheet of when to shut them off in advance to also let most of their servers
become secluded, but also then to regain proper connectivity to their own network and be able to handle the massive load of prestocks in the morning.... also time tables should add a sort of maximum possible allowance no matter what in case the peak season (as christmas is for amazon) would sort of super load their connections and avoid having to shut it all down in order to regain order for their own machines.
You should mention that all of the questions in Frustration Trivia are so USA-centric that people from other countries need not bother.