Instead of dreaming up new ways to interpolate over spotty and incomplete data, why don't they invest in some thermometers and stick them where they need to fill in the data gaps going forward? Real measurements trump "we think this is what the measurements would have been" any day of the week.
And if the response is, it's hard to put up weather stations in all of these far off and exotic locales, tough beans. The fact that science is hard doesn't make incomplete measurements and convoluted interpolations any more solid.
According to the article posted elsewhere on slashdot about this:
Somebody placed massive orders for gold futures contracts betting on exactly that outcome within a millisecond or two of 2 p.m. that day -- before the seven milliseconds had passed that would allow the transmission of the information from the Fed's "lock-up" of media organizations who get an early look at the data and the arrival of that information at Chicago's futures markets
"within a millisecond or two" is still physically impossible given the 3.2 millisecond bound implied by the speed of light in a vacuum, but even the 2-3 millisecond figure quoted in your other article is much less than the actual fiber latency of about 6.5 milliseconds between Chicago and NYC (presumably similar between Chicago and DC), and is still somewhat less than the fastest microwave links available between NYC and Chicago, which come in at about 4.1 milliseconds. (I'm not sure there are any microwave links operational between Chicago and DC as of yet.) These figures are somewhat worse than physical speed of light bounds because they include actual geographical routes and hops, and an index of refraction in the fiber case.
Which is why there should be an investigation - it certainly looks like someone was sitting on the news beforehand and traded on it as soon as 2PM rolled around. But it could also be that a local news organization broke the news embargo ahead of Washington DC. If I understand the article, it seems like news organizations have access to such announcements beforehand so they can prepare ahead of time release their canned articles at the same time. That would be interesting to find out too - I can see local newsmen tipping off their trading buddies where/what to look at precisely 2PM, or just inaccurate network time synchronization.
Well yes you're right; more precisely, it is arbitrage that contributes to the price stability. But most arbitrage takes place today with HFT. Before HFT was possible, sure, there were other means to accomplish arbitrage. However, that doesn't change the fact that today, you have HFT to thank for the benefits of arbitrage.
One of the side benefits of the electronic trading is the traders are anonymous to each other (but not the regulators) and an audit trail gets automatically generated. Some people might call those the main benefits. Why is speed necessary? Because the matching algorithms are mostly FIFO. If you can think of a more fair way to match up buyers and sellers you're welcome to try. But FIFO is the least biased, and therefore being first is a premium.
The news you refer to was notable precisely because it is physically impossible for an electronic signal to get from DC to Chicago in the specified timeframe. I think your culprit is a leaker and accomplices guilty of insider trading, not HFT.
As usual when an "HFT" article appears on slashdot, there is little or no commentary on the actual technical issue at hand, in this case putting custom code into FPGAs on Arista switches. Nor is there any discussion or curiosity about what actually happens in such code as far as HFT is concerned. Instead, a band of emotionally challenged, bitter trolls descends to squelch any useful conversation. Most of the negative DDOSsing comes from people who don't know what HFT is or does; they just "know" it is another way "rich" people are taking money from "poor" people, and comment accordingly with all of the fervor and intelligence of a typical 18 year old member of the local college chapter of Democratic Socialists of America. They have very little or no idea who participates in HFT, they have given very little thought to the complex issues behind why HFT has arisen in the first place, and most of their proposed and regurgitated "solutions" to HFT would laughably do nothing more than to put more power in the hands of price fixers or further entrench too-big-to-fail Wall Street institutions. Reading these posts is pointless, and the all-around antipathy for innovation is surprising in a forum renowned for technology.
There is an inherent cost to conducting transactions in a market place. You can think of it as the cost of buying X and then turning around and selling X: you are left in your initial state minus an amount called the bid/ask spread. In general, reducing the bid/ask spread reduces the inherent cost to you of buying and selling things.
I'm not sure I follow why HFT is bad for mutual funds. There is a small premium that is being effectively paid to arbitrageurs, but the gain is offset by the fact that the mutual fund doesn't have to employ a team of analysts to scour N different venues for a best price and doesn't have to worry about getting out of a trade soon after getting into it (which also manifests as a cost: the fund has to factor in that as a risk).
Arbitrage increases liquidity - essentially you are supporting prices in one market with similar products from other markets. Maybe APPL against an ETF that contains APPL and a basket of the other stocks in that ETF, or a June futures contract against a July contract and a June-July spread. HFT is the tool of arbitrage, mainly because the matching at the exchanges is FIFO. I've not seen any convincing evidence that it increases volatility. There are some well documented cases of mini-crashes that happen when HFT operations are turned off suddenly and in concert, in response to human error or some bug, leading to momentary liquidity crises. People obsess about that and no one seems to remark on how quickly markets recover after the HFTs are turned back on.
Is it a benefit to you that you can pull into a gas station and be assured that you're getting close to the best price for gas in your geographical area?
Is it a benefit to you that you can buy bacon, milk and bread at the store, and be assured that any price differences actually do reflect differences in quality and/or taste, and are not just artifacts of where you bought those items?
When you buy a stock like APPL, is it a benefit to you that you're getting the best available price and you don't have to worry about checking 10 different trading venues before you make a purchase?
All of the above are commodities (or financial instruments in the case of APPL) that are subject to arbitrage and HFT. Before you vomit, consider why it is that prices have stability outside of day to day macro-economic effects.
Put everybody's orders in 15 second time buckets and match randomly? Here's a fictionalized rendition of how that would go:
Let's see, we have 1000 shares of APPL for sale at the bargain price of $485. Who gets these shares... sold to JP Morgan Chase!
And we have 1000 shares of GOOG for sale at the bargain price of $879. Who gets these shares... sold to Goldman Sachs!
And we have 1000 shares of IBM for sale at the bargain price of $190. Who gets these shares... sold to Berkeshire Hathaway!
Well, that's all the matches for this time bucket. John Q. Public, if you didn't get filled this time, try again in 15 seconds, and don't forget to raise your bid price!
The NSA could lease out some of their infrastructure to help move the cloud along. I hear they have some pretty fast networks and large storage capacity. If they leased that out to cloud services, then those companies wouldn't have to develop their own infrastructure.:-p
(1) First law: A quantitative easing either is zero or pumps at a constant rate, unless acted upon by an current or former partner of Goldman Sachs.
(2) Second law: The rate of accumulation of wealth is directly proportional to, and in the same direction as, the net wealth a person already has, and proportional to his/her social standing. Thus, F = ma, where F is the rate of accumulation of wealth, m is the social standing of the person and a is the net wealth already accumulated.
(3) Third law: When one body with high accumulated wealth exerts a force on many second bodies with little or no wealth, the second bodies simultaneously try to exert a force equal in magnitude and opposite in direction to that of the first body, but more often than not they are smothered by the first body in a morass of procedural hurdles, trojan horses and rigged games.
I had Ubuntu 12.04 installed for a short time and HATED it as well. The overall look and feel of the default Unity desktop manager is like it is trying to be a hybrid desktop/tablet OS and doing a half assed job at both. It managed to combine annoying, confusing and pandering on a level almost up there with Microsoft Bob. One wonders if this is a surreptitious reverse advertisement for Windows 8.
One criticism have about the content of the article is that the while the authors argue that the UEE's (the fast price spikes) aren't correlated with news events and can't be explained by random behavior, they don't really address another obvious source of UEEs: hedgers or speculators in the market who are liquidating a position as part of their **long term** needs or trading behavior. Ie- how do they know these aren't market orders (eg- Sell 100 APPL) placed by some guy day trading his brokerage account? If the order book is populated at a few lots per price level, a larger market order will momentarily cut through those levels and then recover. I think they try to sweep that issue under the rug by claiming that the trading volume during these spikes is not appreciably different from trading volume during quiescent times. OK, but it doesn't address who placed the order and why, and what are those volumes anyway?
A second criticism I have is that they explicitly imply that there may be "degree of causality between propagating cascades of UEEs and subsequent global instability". I see correlation here, but no causality. Indeed, if there is global instability, a lot of people in the market will be unwinding positions, and you will see a lot of UEEs as they do so. In other words, there is a strong common sense case to be made for causality **in the other direction** than what they are claiming. I would need to see better evidence than just an overlaid plot (figure 1C).
Finally, and this is more of an observation, does anyone pay attention to how fast the price recovers after a UEE, for whatever reason it originally happens? The recovery happens so quickly because of the dreaded high frequency algorithms. What did UEEs look like ten years ago - when someone came along and sold 100 shares of X, did the market take minutes to recover? That it comes back within a second seems like progress to me.
Every time I buy a stock or sell one, the IRS and other taxing authorities suck some money out of me.
When these computers buy and sell shares several times a second, they do not get taxed. That is not fair.
There should be a tax maybe.001 cents per transaction.
No they don't. The IRS only takes a bite if you make a profit buying or selling stock. It is called capital gains tax. What you may be thinking of is the fee that your brokerage charges for making a transaction.
Professional trading firms also get charged fees by whatever exchange they are trading on, charged per transaction, and they get taxed on any profits generated. Just like you!
I agree Google was put between a rock and a hard place by the NSA. It doesn't change the problem with the cloud itself: there is no practical technical way to make it reasonably secure, unless you're a bobbing cheery Alfred E. Newman "What me worry?" type. Trust is therefore key to commercial cloud computing, to a much greater extent than corresponding "locally resident" solutions. It's a problem for everyone, I'm only singling Google out because of the original post.
Since Snowden’s leaks about PRISM, Google has been leading the charge for legal rights to disclose information about government requests with users.
I don't see how a new encryption effort helps. Anytime you trust a third party to handle your data in the cloud, you are open to having that data compromised because somebody else codes it, somebody else builds it, somebody else deploys it, somebody else administers it, etc. Many who fell for the charming upstart company with the motto "Don't be evil" the first time around feel burned, and there is no technical solution to that problem.
Compare this to Facebook or LinkedIn or even Twitter, who are NOT upfront about what is collected and shared, and who not only share data with governments, but ALSO 3rd party companies at will as part of their business models. As well as your bank, your telco, etc again - all of whom routinely sell client lists including names, addresses, and phone numbers.
Who is the poster child again?
Oh I get it. The problem is everyone EXCEPT Google. Thanks for clearing that up.
I think the totalitarian sickness Schneier describes goes well beyond the NSA. Computers and especially mobile devices are becoming creepy, for lack of a better word, even without government intervention. They are the prying eyes in your house Harriton High School Used Laptop Webcams To SPY On Students At Home, they are following your every move Government Location Tracking: Cell Phones, GPS Devices, and License Plate Readers, they are keeping tabs on what you like and don't like Mapping, and Sharing, the Consumer Genome (featured on slashdot yesterday, itself a thinly veiled phishing scam IMHO). Although subject to government abuse, none of the "services" highlighted in those links were instigated by the government. Just yesterday I was innocuously checking for prices for various professional training seminars on Google, and on cue my Email inbox started overflowing with unsolicited offers. On some days, I want to throw my smartphone in the trash and unplug my computer from the internet and only plug it back in when I need to access the SVN repository.
So Kudos to Bruce Schneier for addressing his call to the engineering community, but now it begs a question: aren't engineers, including those outside the NSA/DEA/FBI, somewhat responsible for creating this creepy user experience? I don't think they're suddenly going to wake up one day and fix it; a significant subset has embraced the creepiness and fundamentally doesn't understand why it might be a problem for others.
Thought I'd look at my own data, but when they started asking for the last 4 digits of my SSN I decided I didn't care so much about what they knew about me...
My god, it's some of the same info I give to my brokerage when I call them on the phone. Why would I give it to some random jerk at a marketing firm who could then impersonate me on the phone, along with the keys needed to look up answers to common "security" questions like mother's maiden name, or name of best man at my wedding, etc, that may have been entered into ancestry.com or may have been posted on Facebook?
Slashdot Mirror
If I had any mod points, I'd mod this up +1 Insightful.
Instead of dreaming up new ways to interpolate over spotty and incomplete data, why don't they invest in some thermometers and stick them where they need to fill in the data gaps going forward? Real measurements trump "we think this is what the measurements would have been" any day of the week.
And if the response is, it's hard to put up weather stations in all of these far off and exotic locales, tough beans. The fact that science is hard doesn't make incomplete measurements and convoluted interpolations any more solid.
We're getting off on a tangent here.
Good point. Technically thought, the assignment doesn't return a boolean, it returns a number.
The "if" statement on the other hand treats integer arguments as implicit booleans: if (a != 0) ... that's where the type unsafety lies.
I am impressed that the kernel team caught that. Kudos!
According to the article posted elsewhere on slashdot about this:
Somebody placed massive orders for gold futures contracts betting on exactly that outcome within a millisecond or two of 2 p.m. that day -- before the seven milliseconds had passed that would allow the transmission of the information from the Fed's "lock-up" of media organizations who get an early look at the data and the arrival of that information at Chicago's futures markets
"within a millisecond or two" is still physically impossible given the 3.2 millisecond bound implied by the speed of light in a vacuum, but even the 2-3 millisecond figure quoted in your other article is much less than the actual fiber latency of about 6.5 milliseconds between Chicago and NYC (presumably similar between Chicago and DC), and is still somewhat less than the fastest microwave links available between NYC and Chicago, which come in at about 4.1 milliseconds. (I'm not sure there are any microwave links operational between Chicago and DC as of yet.) These figures are somewhat worse than physical speed of light bounds because they include actual geographical routes and hops, and an index of refraction in the fiber case.
Which is why there should be an investigation - it certainly looks like someone was sitting on the news beforehand and traded on it as soon as 2PM rolled around. But it could also be that a local news organization broke the news embargo ahead of Washington DC. If I understand the article, it seems like news organizations have access to such announcements beforehand so they can prepare ahead of time release their canned articles at the same time. That would be interesting to find out too - I can see local newsmen tipping off their trading buddies where/what to look at precisely 2PM, or just inaccurate network time synchronization.
Well yes you're right; more precisely, it is arbitrage that contributes to the price stability. But most arbitrage takes place today with HFT. Before HFT was possible, sure, there were other means to accomplish arbitrage. However, that doesn't change the fact that today, you have HFT to thank for the benefits of arbitrage.
One of the side benefits of the electronic trading is the traders are anonymous to each other (but not the regulators) and an audit trail gets automatically generated. Some people might call those the main benefits. Why is speed necessary? Because the matching algorithms are mostly FIFO. If you can think of a more fair way to match up buyers and sellers you're welcome to try. But FIFO is the least biased, and therefore being first is a premium.
The news you refer to was notable precisely because it is physically impossible for an electronic signal to get from DC to Chicago in the specified timeframe. I think your culprit is a leaker and accomplices guilty of insider trading, not HFT.
As usual when an "HFT" article appears on slashdot, there is little or no commentary on the actual technical issue at hand, in this case putting custom code into FPGAs on Arista switches. Nor is there any discussion or curiosity about what actually happens in such code as far as HFT is concerned. Instead, a band of emotionally challenged, bitter trolls descends to squelch any useful conversation. Most of the negative DDOSsing comes from people who don't know what HFT is or does; they just "know" it is another way "rich" people are taking money from "poor" people, and comment accordingly with all of the fervor and intelligence of a typical 18 year old member of the local college chapter of Democratic Socialists of America. They have very little or no idea who participates in HFT, they have given very little thought to the complex issues behind why HFT has arisen in the first place, and most of their proposed and regurgitated "solutions" to HFT would laughably do nothing more than to put more power in the hands of price fixers or further entrench too-big-to-fail Wall Street institutions. Reading these posts is pointless, and the all-around antipathy for innovation is surprising in a forum renowned for technology.
There is an inherent cost to conducting transactions in a market place. You can think of it as the cost of buying X and then turning around and selling X: you are left in your initial state minus an amount called the bid/ask spread. In general, reducing the bid/ask spread reduces the inherent cost to you of buying and selling things.
I'm not sure I follow why HFT is bad for mutual funds. There is a small premium that is being effectively paid to arbitrageurs, but the gain is offset by the fact that the mutual fund doesn't have to employ a team of analysts to scour N different venues for a best price and doesn't have to worry about getting out of a trade soon after getting into it (which also manifests as a cost: the fund has to factor in that as a risk).
Arbitrage increases liquidity - essentially you are supporting prices in one market with similar products from other markets. Maybe APPL against an ETF that contains APPL and a basket of the other stocks in that ETF, or a June futures contract against a July contract and a June-July spread. HFT is the tool of arbitrage, mainly because the matching at the exchanges is FIFO. I've not seen any convincing evidence that it increases volatility. There are some well documented cases of mini-crashes that happen when HFT operations are turned off suddenly and in concert, in response to human error or some bug, leading to momentary liquidity crises. People obsess about that and no one seems to remark on how quickly markets recover after the HFTs are turned back on.
And slowing down HFT would fix this ... how?
Is it a benefit to you that you can pull into a gas station and be assured that you're getting close to the best price for gas in your geographical area?
Is it a benefit to you that you can buy bacon, milk and bread at the store, and be assured that any price differences actually do reflect differences in quality and/or taste, and are not just artifacts of where you bought those items?
When you buy a stock like APPL, is it a benefit to you that you're getting the best available price and you don't have to worry about checking 10 different trading venues before you make a purchase?
All of the above are commodities (or financial instruments in the case of APPL) that are subject to arbitrage and HFT. Before you vomit, consider why it is that prices have stability outside of day to day macro-economic effects.
Put everybody's orders in 15 second time buckets and match randomly? Here's a fictionalized rendition of how that would go:
Let's see, we have 1000 shares of APPL for sale at the bargain price of $485. Who gets these shares ... sold to JP Morgan Chase!
And we have 1000 shares of GOOG for sale at the bargain price of $879. Who gets these shares ... sold to Goldman Sachs!
And we have 1000 shares of IBM for sale at the bargain price of $190. Who gets these shares ... sold to Berkeshire Hathaway!
Well, that's all the matches for this time bucket. John Q. Public, if you didn't get filled this time, try again in 15 seconds, and don't forget to raise your bid price!
Perhaps the most soul crushing phrase you will ever hear as a programmer is: "Don't re-invent the wheel."
Go ahead, re-invent that wheel every now and then. That's why you got into programming in the first place. You can do a better job.
The NSA could lease out some of their infrastructure to help move the cloud along. I hear they have some pretty fast networks and large storage capacity. If they leased that out to cloud services, then those companies wouldn't have to develop their own infrastructure. :-p
But the map of the world with all of the pretty dots on it makes a convincing case otherwise. How could it be wrong? [/sarcasm]
Bernanke's Laws of Econophysics:
(1) First law: A quantitative easing either is zero or pumps at a constant rate, unless acted upon by an current or former partner of Goldman Sachs.
(2) Second law: The rate of accumulation of wealth is directly proportional to, and in the same direction as, the net wealth a person already has, and proportional to his/her social standing. Thus, F = ma, where F is the rate of accumulation of wealth, m is the social standing of the person and a is the net wealth already accumulated.
(3) Third law: When one body with high accumulated wealth exerts a force on many second bodies with little or no wealth, the second bodies simultaneously try to exert a force equal in magnitude and opposite in direction to that of the first body, but more often than not they are smothered by the first body in a morass of procedural hurdles, trojan horses and rigged games.
This had Unity on by default, doesn't it? Epic fail.
We can almost hear one of one Redmond’s richest residents rubbing his hands together with glee.
Unity is almost as relevant to Ubuntu 12.04 as ASIMO is to Honda’s latest hatchback.
I had Ubuntu 12.04 installed for a short time and HATED it as well. The overall look and feel of the default Unity desktop manager is like it is trying to be a hybrid desktop/tablet OS and doing a half assed job at both. It managed to combine annoying, confusing and pandering on a level almost up there with Microsoft Bob. One wonders if this is a surreptitious reverse advertisement for Windows 8.
One criticism have about the content of the article is that the while the authors argue that the UEE's (the fast price spikes) aren't correlated with news events and can't be explained by random behavior, they don't really address another obvious source of UEEs: hedgers or speculators in the market who are liquidating a position as part of their **long term** needs or trading behavior. Ie- how do they know these aren't market orders (eg- Sell 100 APPL) placed by some guy day trading his brokerage account? If the order book is populated at a few lots per price level, a larger market order will momentarily cut through those levels and then recover. I think they try to sweep that issue under the rug by claiming that the trading volume during these spikes is not appreciably different from trading volume during quiescent times. OK, but it doesn't address who placed the order and why, and what are those volumes anyway?
A second criticism I have is that they explicitly imply that there may be "degree of causality between propagating cascades of UEEs and subsequent global instability". I see correlation here, but no causality. Indeed, if there is global instability, a lot of people in the market will be unwinding positions, and you will see a lot of UEEs as they do so. In other words, there is a strong common sense case to be made for causality **in the other direction** than what they are claiming. I would need to see better evidence than just an overlaid plot (figure 1C).
Finally, and this is more of an observation, does anyone pay attention to how fast the price recovers after a UEE, for whatever reason it originally happens? The recovery happens so quickly because of the dreaded high frequency algorithms. What did UEEs look like ten years ago - when someone came along and sold 100 shares of X, did the market take minutes to recover? That it comes back within a second seems like progress to me.
Every time I buy a stock or sell one, the IRS and other taxing authorities suck some money out of me. When these computers buy and sell shares several times a second, they do not get taxed. That is not fair. There should be a tax maybe .001 cents per transaction.
No they don't. The IRS only takes a bite if you make a profit buying or selling stock. It is called capital gains tax. What you may be thinking of is the fee that your brokerage charges for making a transaction.
Professional trading firms also get charged fees by whatever exchange they are trading on, charged per transaction, and they get taxed on any profits generated. Just like you!
I agree Google was put between a rock and a hard place by the NSA. It doesn't change the problem with the cloud itself: there is no practical technical way to make it reasonably secure, unless you're a bobbing cheery Alfred E. Newman "What me worry?" type. Trust is therefore key to commercial cloud computing, to a much greater extent than corresponding "locally resident" solutions. It's a problem for everyone, I'm only singling Google out because of the original post.
I don't see how a new encryption effort helps. Anytime you trust a third party to handle your data in the cloud, you are open to having that data compromised because somebody else codes it, somebody else builds it, somebody else deploys it, somebody else administers it, etc. Many who fell for the charming upstart company with the motto "Don't be evil" the first time around feel burned, and there is no technical solution to that problem.
Compare this to Facebook or LinkedIn or even Twitter, who are NOT upfront about what is collected and shared, and who not only share data with governments, but ALSO 3rd party companies at will as part of their business models. As well as your bank, your telco, etc again - all of whom routinely sell client lists including names, addresses, and phone numbers.
Who is the poster child again?
Oh I get it. The problem is everyone EXCEPT Google. Thanks for clearing that up.
I think the totalitarian sickness Schneier describes goes well beyond the NSA. Computers and especially mobile devices are becoming creepy, for lack of a better word, even without government intervention. They are the prying eyes in your house Harriton High School Used Laptop Webcams To SPY On Students At Home, they are following your every move Government Location Tracking: Cell Phones, GPS Devices, and License Plate Readers, they are keeping tabs on what you like and don't like Mapping, and Sharing, the Consumer Genome (featured on slashdot yesterday, itself a thinly veiled phishing scam IMHO). Although subject to government abuse, none of the "services" highlighted in those links were instigated by the government. Just yesterday I was innocuously checking for prices for various professional training seminars on Google, and on cue my Email inbox started overflowing with unsolicited offers. On some days, I want to throw my smartphone in the trash and unplug my computer from the internet and only plug it back in when I need to access the SVN repository.
So Kudos to Bruce Schneier for addressing his call to the engineering community, but now it begs a question: aren't engineers, including those outside the NSA/DEA/FBI, somewhat responsible for creating this creepy user experience? I don't think they're suddenly going to wake up one day and fix it; a significant subset has embraced the creepiness and fundamentally doesn't understand why it might be a problem for others.
Thought I'd look at my own data, but when they started asking for the last 4 digits of my SSN I decided I didn't care so much about what they knew about me...
My god, it's some of the same info I give to my brokerage when I call them on the phone. Why would I give it to some random jerk at a marketing firm who could then impersonate me on the phone, along with the keys needed to look up answers to common "security" questions like mother's maiden name, or name of best man at my wedding, etc, that may have been entered into ancestry.com or may have been posted on Facebook?