Slashdot Mirror
Robust Timing Over the Internet
ChelleChelle writes "The NTP (Network Time Protocol) system for synchronizing computer clocks has been around for decades and has worked well for most general-purpose timing uses. However, new developments, such as the increasingly precise timing demands of the finance industry, are driving the need for a more precise and reliable network timing system. Julien Ridoux and Darryl Veitch from the University of Melbourne are working on such a system as part of the Radclock Project. In this article they share some of their expertise on synchronizing network clocks. The authors tackle the key challenge — taming delay variability — and provide useful guidelines for designing robust network timing algorithms."
let's do away with the arbitrage, gambling, and bullshit from wall street. make them own a stock for ONE WHOLE DAY. No more of this low-latency trading bullshit. They're just skimming money off the top of the financial markets--away from the regular folks. EVERY PENNY THESE GUYS MAKE COMES OUT OF OUR POCKET.
How about GPS receiver + exact known location?
I would think just eliminating an unknown latency and replacing it with a known one (radio waves generally travel at a consistent rate) would get you a pretty accurate time. If you add one more known variable which is the exact location of the device, then you should be able to get an even more accurate time.
Microsoft refuses to consider the notion of clocks accurate to within 2 seconds.
I'm surprised the article didn't mention PTPd, which is an implementation of the IEEE 1588 precision time-synchronization standard. I was under the impression that was the standard way to solve this sort of problem when NTP wasn't enough.
I don't care if it's 90,000 hectares. That lake was not my doing.
It's not clear to me the financial system needs such high precision timing.
My cynicism tells me they need it now to implement fraudulent micro-second trading, where see what you're buying, correlate with other people's buys and then buy a fraction of a second ahead of you.
At any rate, it's not clear they need it, or that giving it to them won't lead to instability.
Maybe someone should actually design a financial system, and design one that can use imprecise, or precision timing.
NTP is accurate to within 10 milliseconds in any decent connection. Get a really good, stable, low latency connection and hook yourself up to a stratum 1 server and you can cut that number down tenfold ...
Can anyone explain what kind of financial application would require bigger accuracy?
I understand that certain scientific experiments require more accuracy than that, but those guys are probably already hooked up to stratum 0 servers directly.
Why on earth would any financial system (yes, even the stock market) require 10ms time accuracy? They were able to rip us off with fucking sand clocks. NTP should be enough for them.
WTF am I doing replying to an AC at 5 A.M on a Friday night?
EVERY PENNY THESE GUYS MAKE COMES OUT OF OUR POCKET.
I see you have not the foggiest idea how "investments" and "fluidity" make the economy work.
Hint: When the Great Depression hit, the money lost by the traders on Wall Street did not suddenly spew forth into the hands of "us".
>> Standing on head makes smile of frown, but rest of face also upside down.
let's do away with the arbitrage, gambling, and bullshit from wall street. make them own a stock for ONE WHOLE DAY. No more of this low-latency trading bullshit.
A Capital Gains Tax that started at 95% and decreased ~2.5% per month would go a long way towards fixing many problems (/avoiding them in the first place).
Network issues. If say, on average, once every 25 polls, ntpd doesn't get realistic data from higher stratum servers because of the network, don't let ntpd crank up the frequency to ridiculous values like it does when this occurs.
There are realistic values for the frequency on every machine with a good clock. It is ridiculous to set the frequency below or above these values.
Last time I checked, there is no way in ntpd to configure these values. The typical ntp guru reply will be: "Get a decent network connection". The author in TFA noted that such a "decent connection" is virtually impossible to achieve because of "variable delay". He also noted that it is sometime better to trust yourself, which is kind of what that script does.
Everything I write is lies, read between the lines.
Phew that's OK then, it's only everyone with a fucking superannuation.
Author of TFA also mention:
"More generally, the algorithm should be designed never to overreact to anything. Remember, its view of the world is always approximate and may be wrong, so why try to be too clever when inaction works so well? Unfortunately, feedback algorithms such as ntpd have more reactive strategies that drive the clock more strongly in the direction of their opinions. This is a major source of their nonrobustness to disruptive events."
I fully agree with this and I realized it several years ago while testing ntpd in various configurations.
Everything I write is lies, read between the lines.
Agreed. When are some of these people going to realize that greed is bad. Trying to game a system is inherently wrong. Not just morally, but wrong systemically. It will always result in harm to the system. These people soothe their conscience with excuses- "We're spreading the risk and that's a good thing", "It's only fractions of pennies off of many people, nobody will notice." , etc. Until the whole thing comes crashing down because in reality the financial system is core dependent on actual investment and trust, not on gambling. To quote Chinatown:
Jake Gittes: How much are you worth?
Noah Cross: I have no idea. How much do you want?
Jake Gittes: I just wanna know what you're worth. More than 10 million?
Noah Cross: Oh my, yes!
Jake Gittes: Why are you doing it? How much better can you eat? What could you buy that you can't already afford?
let's do away with the arbitrage, gambling, and bullshit from wall street. make them own a stock for ONE WHOLE DAY. No more of this low-latency trading bullshit. They're just skimming money off the top of the financial markets--away from the regular folks. EVERY PENNY THESE GUYS MAKE COMES OUT OF OUR POCKET.
You are absolutely, painfully right and it pains me deeply that I don't have any mod points.
These jokers want time more accurate than NTP can provide but I'll bet most of them still believe time is a universal invariant. They don't need time more accurate than NTP can provide, they need to be hit in the head with a hammer until they stop moving.
Hint: When the Great Depression hit, the money lost by the traders on Wall Street did not suddenly spew forth into the hands of "us".
Because the money you mention never really existed.
If I had a thousand dollars in paper money under my bed before the great depression it would have had a certain buying power. In the middle of the great depression it would have had a much greater buying power. The value lost on Wall Street spewed forth into paper money and real physical assets.
Hi!
Last few years, most cellular operators are rolling out IP to the base stations.
For HSPA/LTE speeds, E1/T1 PDH/ATM is out of the question,
and as IP is rolled out, GSM gets there as well.
Accurate timing is essential: even transmitter frequency is synchronized over IP,
and keeping 2 GHz accurate to 100 Hz is not simple.
Local oscillator can do it short term, but external correction is needed as well.
Beside GPS (used in IS-95), several systems are used.
IEEE 1588v2 is obvious one. Works.
Some vendors do it with modified NTP: generate a lot of requests, and do some statistic processing.
Thus, most of the jitter can be compensated, and accuracy can be kept.
Normally, it works over intranets, with NTP having high priority, but it is used over internet as well
(femto and pico base stations).
Real life tests indicate it works, just do not put it over ADSL ;-)
73
Iztok
I don't understand the need for wide-area highly accurate (less than 1ms accurate) timing. Once you get beyond a reasonable distance, the speed of light starts playing into the equation.
Like most smart people you are forgetting how dumb most people are.
Wall street traders don't understand much about time. They only understand how to obfuscate financial transactions like a high speed shell game. To them more speed means more chances to skim off a percentage and the laws of physics don't even get considered.
GPS is accurate to about 50 nanoseconds. All kinds of devices that need precision time get it from GPS. You don't need much more than a standard receiver, just one that is designed to place a high priority on time updates. The GPS system itself keeps very accurate time since each satellite has an atomic clock, and they all sync to the master clock.
To me that would seem the best way, if accuracy is really important and the systems are high end. Have each device have its own receiver and just have them sync to that. Don't sync them to each other, since you aren't going to get anything more accurate.
While I think that's a bit extreme, I think you and the GP are both on to the right track. I think stocks should have a fairly short minimum hold time. A day would probably be fine, just something such that you aren't trying to gamble with them more or less, because that's why high speed day trading really is. It is gambling in every way that Vegas is.
I also thing a sliding tax would make sense. If you execute quick trades, you are subjected to a fairly high tax. 95% is a bit excessive, but something still fairly high. Then, as you say, the tax drops on a monthly basis until it hits zero. That would act to encourage investing, rather than gambling. It rewards people who put their money away for the future, and do so in a manner that allows it to help the economy grow. It would also still allow for quicker trades, but the government gets to take a cut in that case.
I really do think we need to start looking at something like this for three reasons:
1) Because there are companies that try to game the system. That's never a good thing. You game the system on a large scale, you'll break it. When you are talking about something highly important to the economy, that's not acceptable.
2) Because there are too many gamblers. There are plenty of traders who basically gamble on the market. They make day trades not based on anything about the company, but just to try and grab money from trends. This is not useful, and isn't what the stock market is for. It leads to unnecessary instability and thus should be discouraged.
3) To help combat people's fear response. A non-trivial part of the big drop in the stock market last year was people being fearful. They panicked because of the economic troubles and wanted all their money out. That means even if the fund managers thought it was a bad idea, they had no choice. You have to do what your client wants. That of course lead to more drops and more panic and so on. Perhaps tax incentives could help stave that off with people.
Actually this kind of trading (especially arbitrage) narrows the spread making it cheaper for ordinary people to enter the market.
So if you ban it you go back to only the big boys being able to afford to execute trades.
However, using any system, high frequency or otherwise, to deliberately manipulate the market is a different thing.
I'm sorry I'm posting as AC but it could affect my job.
I vote that all traders - and lawyers and estate agents for that matter - are sent such a reasonable distance from the planet that the speed of light indeed starts playing into the equation.
ALl in the name of research, of course.
AT&ROFLMAO
Who the hell isn't greedy. If you were offered more money at work, would you turn it down. Most people wouldn't. And all your colleagues are going to get paid more.
This is all these people are doing, they are doing there jobs, and they get paid bonuses for doing it well.
It is ignorant and stupid to pretend that this situation is just that simple.
You don't need accurate timing for that, you just need to be first.
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
Easy Answer: (in most tax laws)
Any arangement who's primary purpose is to avoid taxation is illegal, and will be charged tax at the rate at which it would normally have been taxed.
The authors of the RADclock article don't seem to realize that most of the issues they "solve" with their algorithm has been considered, and also more or less solved, by the NTP hackers severeal years ago.
I.e. when the article shows that in a variable-latency environment you need to filter the measurements by the minimum rtt: This is of course default behaviour for ntp as well, but only over a limited number of polling intervals (8). If this isn't sufficient then there are a number of configuration options available, like the huff-puff filter which can be configured to maintain rtt history long enough to "coast" past any expected periods of large and asymmetrical network delays.
They also give out some _really_ bad advice, like claiming that "One server is enough": This is only true if you know that this particular server can never be wrong!
If you want to survive the situation where a single servers starts to lie, you shold configure at least 4 independent sources.
Personally I have seen 3 different gps-based stratum-1 black box ntp servers go crazy, suddenly returning timestamps far into the future.
OTOH, my freebsd-based servers with Motorola Oncore UT+ timing optimized gps firmware has never returned bad time, but I still have 6 such servers spread around our corporate network.
Terje
"almost all programming can be viewed as an exercise in caching"
Parent characterises the complete detachment from reality that characterises the modern "innovative" financial industry.
I operate a small business. I don't give a fsck about your claims that we need "liquidity" when I can't get a simple business loan for less than 8-9%, despite being well-capitalised, while my personal savings attract all of r pct, where 0.01 < r < 0.1. Quite the scam there. As a result, I use only retained earnings to pay for anything, which while a prudent thing to do given my situation, is hardly a recipe for large-scale economic growth. The financial industry does not fulfil its most basic imperative of funnelling capital to productive use, why should I care about whether some institutions can hold a stock for 1ms or 1 day?
I suppose things could be worse. Maybe in addition to being completely starved for capital, I may be subject to "protection" payments in the near future just to be able to stay in business at all! Financial innovation, 'Ndrangheta style! Wouldn't that be a boon for Wall Street and the City of London.
I think people outside of the industry don't really understand the need for day traders.
I write trading software for the futures and options industries (I was the lead architect for the CBOT's Order Routing System, and have written trading systems for other exchanges and banks). The type of day trading that you and the GP have issues with is an absolute requirement for liquid markets and the efficient working of those markets.
Let's say you are a firm that needs to buy a large number of shares or contracts (in the futures world, you buy contracts for the future delivery of product). In the type of buy and hold market that seems "fair", that large order is going to go into a market of other buy and hold traders that are only going to trade out of their positions if the trade makes sense in their long-term strategy. Huge effective tax rates are going to make that trade unprofitable unless you've held the position a very long time (increasing your risk of market volatility) or you charge a premium (increase the price) in order to offset the tax hit.
This type of "illiquid" market raises prices dramatically. You have to factor in the time value of money, the risk exposure of holding shares for a long time (increasing the likelihood of a bad market event) and tax overhead. All that means that buying shares/contracts is riskier and selling them is more expensive.
Day trading fixes this. Day traders don't really care about the long-term direction of the market - they make money on minor intra-day price fluctuations. Because of this, they are nearly always willing to take on and shed positions. And because there are a lot of them chasing the same tiny fluctuations, they "shrink the spread" - they're going to give you a very good price because they shrink the difference between the bid and offer (price to buy/price to sell).
Now, when that original firm wants to buy that large number of shares, they are doing that in a market where there is almost always a large number of shares being offered at competitive prices. The day traders will sell off shares and pick up shares all day, just trying to end the day flat (holding nothing) to reduce after market risk.
Day trading does extract a price (they are making money) on the market, but the efficiency gains that they bring outweighs that price in almost all markets. They distribute a lot of the risk involved in trading among a bunch of smaller firms, rather than concentrating all the risk in big funds. This is generally a good thing - the government and the market in general are better off when a small firm goes bust than when one of the "too big to fail" firms starts having problems.
Now, the rise of algorithmic trading is causing some stress in this pattern - I can see some real issues coming in the next few years as these algorithms become more and more precise. I'm less concerned about a rogue algorithm dragging a market down (or up) than of all the algos converging to a point where the market gets stuck because the algos are just too damn efficient. But that's a post for another day...
NTP is accurate to within 10 milliseconds in any decent connection. Get a really good, stable, low latency connection and hook yourself up to a stratum 1 server and you can cut that number down tenfold ...
Can anyone explain what kind of financial application would require bigger accuracy?
Where have all the nerds gone?
Why are we debating whether or not the people who think they need highly accurate time really need it? What's cool about this article is the nerdly reaching for every higher degrees of accuracy, of digging deeply into a seriously difficult technical challenge and achieving new levels. What's cool for those of us who don't work on super-accurate timescales is the opportunity to read about how those who do think about this stuff, to see what math they use and what the issues are.
I don't mind a little discussion of social implications of technology, but it'd be nice to see at least a little discussion of the technology itself.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
The only really major issue with NTP is figuring out the time offset due to network latency in asymmetric network environments. The NTP protocol itself can frequency lock down to a few ppm over the internet without any problem at all. Offset errors are another matter. Over the internet 10 ms is about the best you can do. Over a LAN with a local GPS signal driving the protocol you can reduce the offset error to less than a microsecond and can probably get it down to less than 100ns without too much trouble (and without needing any hardware time stamping or netif queue prioritization), and you can frequency lock down to a few nanoseconds.
Other issues include the fact that most motherboards do not have temperature compensated timebases, and they tend to float around a few ppm (for the better MBs) to a few hundred ppm (for the stupid MBs that put crystals next to heat sources), as well as bugs in the operating systems themselves which don't get noticed until you actually try to frequency lock your timebase. This creates a multiplicative effect if you are not connected to a stratum 1 or 2 server because each server in the chain is trying to do a frequency lock against a drifting MB timebase. The result is the leaf nodes in the chain can get frequency locks but the locks are to a drifted frequency instead of to the correct frequency because the server itself can't instantly correct for its drifting timebase. This is why you see the frequency lock jump around by +/- 50ppm in a seemingly uncontrollable fashion once you get past stratum 2.
NTPD was famous for not being able to frequency lock while doing an offset correction at the same time. The NTPD program has also had many issues over years, such as not using a proper dual-staggered linear regression to control offset and frequency corrections, which is why I wrote dntpd for DragonFly. I wonder how many of these old issues with NTPD have been fixed over the years, maybe they have.
Network jitter is largely irrelevant. It isn't a problem if you use a proper linear regression over a long enough period of time. The linear regression can be used to calculate what the jitter is and thus form a good knowledge of the baseline accuracy of the protocol. A medium sized run over a few minutes will get you down to 10-15 ppm on your frequency lock and you can get down to 1-5 ppm within about 10 minutes (assuming the motherboard has an accurate timebase of its own).
The sad thing about all of this is that it takes just a single resistor to temperature compensate a crystal, a resistor most MB manufacturers don't put in. Some don't even use crystals anymore for their PLL input (and the crystal on the RTC doesn't help much because the RTC doesn't have a fine enough grain timer to poll easily). Sigh.
-Matt
Although the summary and the article itself seem to take pains not to mention it, a visit to the RADclock homepage (http://www.cubinlab.ee.unimelb.edu.au/radclock/) will tell you that what's actually being offered here is an improved NTP client. No changes to the NTP servers, server software or NTP protocol are required or are proposed. The client improvements are in an improved filter topology (feed forward with quality assessment) and introduction of separate concepts of absolute and difference clocks optimally supporting the different ways that time is used by applications.
Examine the concept of "god" and you'll see the answer.
"god" does not exist in reality ; the concept exists perfectly well, but in physical reality there ain't no such puppy ; however even an convinced atheist can't deny that the non-existent entity "god" has real effects on life, the universe and 42. Richard Dawkins, amongst others, makes a steady income selling books pointing out how much damage this concept has caused.
The power of the non-existent thing "god" to have effects in the real world comes from the fact that some people think that "god" exists, and this changes their behaviour.
Similarly, the belief of market people, economists etc in the existence of "money" affects their behaviour and their behaviour has real-world consequences.
Birds are not dinosaur descendants;birds are dinosaurs, for all useful meanings of "birds", "are" and "dinosaurs"