The article opens with: Many successful academic and commercial projects use direct traffic measurements (such as ping, traceroute, and web page access data) to study the structure and dynamics of the Internet. Such efforts are inherently limited by the locations of probe points required to 'cover' the Internet meaningfully. Compounding the problem, there are no effective shortcuts - simply placing agents throughout the Internet's core, as done by several commercial services, only builds up a picture of core-to-core traffic latencies and losses that has no power to predict the true "Internet weather" that end users actually experience at the network edge.
This is just plain wrong. It is quite easy to obtain latency measurements of the edge starting from the core.
Let E1 and E2 be points on the edge. If you have enough agents in the core, you will find an agent A in the path from E1 to E2. Then you can easily compute the latency from E1 to E2 by ping from A to E1 and from A to E2.
Are they going to decrypt and listen in on every VPN tunnel?
You completely missed the point.
The NSA doesn't have to listen on every VPN tunnel. If GM has a VPN between Detroit and California, one can assume that it is unlikely two terrorists would communicate using that tunnel from within GM, so no need to monitor such traffic. Since most of traffic is corporate and legit, you can eliminate all but one or two terabytes of traffic a day.
That leaves 20K of data to be processed per day for each PC-unit-of-cpu power available to the NSA.
I think you are underestimating the amount of computational power available to the NSA. I believe the NSA has enough CPU capacity to analyze every single bit that traverses their network. Think about it, the cpu power of 100 million PCs are well within their budget. That many PCs running 24x7 would produce more CPU cycles than the rest of the world combined (due to subutilization of resources elsewhere).
Moreover significant portions of communications can be thrown out rather quickly such as regular backups from established corporations, usenet redistributions down the hierarchy and the umptenth access to slashdot's web page. (A trie structure works wonders for this, and it can be distributed rather easily). I would posit that 95-97% of the web traffic Joe User generates can be discarded in this step.
Once you culled out such data you are left with a few potentially suspicious messages.
Next you record all of those using an analog device.
Then analyze all recorded data for suspicious patterns.
At this point you just hope you get lucky. At first you don't need to break all messages in a sequence of communications, you only need to break one and then track back in time your archives for related communications and break those using communication specific learned information. Planning a complex operation such as bombing the NYC would normally require hundreds of message exchanges. If your chances of detecting a pattern in communication are one-in-one-hundred you are in business.
If CNN had kept Akamai, it certainly would have helped sustained the load. However do you thnk it would have been enough, or was the traffic so taxing that nothing could have possibly done the job?
They have an order of magnitude more visitors than/.
Try three orders of magnitude. Estimates of hits at CNN were over 50,000 per second.
/.ers: Don't get too cocky...
on
Handling the Loads
·
· Score: 5, Insightful
While my heartfell thanks go to/. for keeping this site up; others who are dissing the major news organizations must keep in mind that while
Slashdot was serving 50 pages per second, CNN was peaking at about an estimated 50,000 hits per second.
In light of this it was amazing that CNN was up at all, slow as it was.
The first time I heard about the price of Aerons I flipped out. Later on while on a consulting gig I was assigned a cubicle with an Aeron chair. Sat down, adjusted it, didn't feel any difference. So I just got on with my work.
Only twelve hours later did I realize that I was still sitting in the chair and that my buttocks didn't hurt. I had not twitched or slouched once in the whole session.
But you are right, Aeron chairs are a stupidity test. It tests those who think that just because they are expensive they cannot be worth the money.
Remember, defense always has an advantage over offense, in terms of time and effort -- the "battle" is on the defender's own turf. Offense can make up for this inherent disadvantage to some extent by, for instance, having the element of surprise.
Actually the entire field of guerilla warfare is predicated upon the complete opposite of your statement. Offense has the advantage over defense because they can choose the terms, time and location of engagement.
Or to quote Warren Buffet instead of Chairman Mao, investing is like a match of cricket, rather than baseball: there is no penalty for not swinging the bat.
The art professors are behind the times.
Two years ago, while talking to a couple of artists from the Paris art scene (who actually make a living out of creating art) they said:
"Yeap, it's amazing. Nowadays pretty much every piece of art our friends create starts by turning on the Mac."
"They first outline, sketch and play with it on the screen, and then proceed from there to whatever is the medium of their choice, including hitting the print button."
Google uses keyword searching. I've said it before, and I'll say it again. Keyword searching is a Dead-End Technology.
Well, as somebody who actually works in the field of web search engines, here's how keyword searches are seen:
"keyword searches suck, everything else sucks even more"
There is a pile of search engine carcasses proving it: the old lycos stemmer, the statistic excite approach, the OpenText structured data query, the Ask jeeves type-20-words to get the same old crappy answer you would get with a single keyword and the list goes on and on.
Yes, sooner or later researchers will find a way to go around keyword searches. But contrary to what you say, this is not bad news for Google. Nothing stops them from adopting the new technology when an alternative arises.
We should not get stuck in our mental image of ramps and countless slaves which isn't based on much concrete evidence, either.
Indeed. Ingenious labor saving devices are common across all cultures. Ramps and slaves sounds about the dumbest way to approach the construction task. Nobody is that dumb.
As a witness we have the frozen stone man which had an ingenious collection of tools and gadgets to fix and repair his arrows and bow.
If I had every version of a libary on my computer then my 4GB/usr directory would very quickly become bigger than my 40GB capacity
Versioned libraries does not imply having every version in existance. In all likelihood applications would support a range of versions, plus as you update to newer versions of each application, older libaries can safely be removed.
Well it'll be a lot safer to only use the libary which the programme was developed for but the development cycles are way too short.
I think this is a peculiarity of the immature state of Linux. As it ages changes will not occur as often, and improvements would be so minor that most people would hold on the upgrade until a full 1.0 change (only fresh installs would use the latest code)
What linux needs is just better scripts for creating packages and better error handling for when installing packages doesn't go right.
Those who do not learn from the past are condemned to repeat it. Automated replacement of shared libraries is the road to hell.
Automated replacing of shared libraries is the road to hell. Windows has proven that.
Each library must be versioned and each package must call explicitly a version of each library.
If the desired version is not available, the installer should deploy it.
Now that would be worthy of an advanced OS. Anything else is simply an open source implementation of a good but outdated OS (aka Unix).
How about just sitting down and explaining to her that there are a lot of sick people out there, on and off the net?
How about teaching her to make proper judgements, because if she's curious, she will have access to awful things one way or another (friends house, the library, even, gasp, in a book).
Plus sooner or later she will join the real world (TM) as a mature adult (C) and there will be no url log file or father sitting by her chair approving and disapproving of her choices.
How about teaching her something instead of tying a leash and threatening with a url log stick?
And what is even more amazing is the chorus of dittoheads recomending different strength leashes instead of suggesting teaching her the difference between good and bad.
While this news report is very likely just a measurement error, we must be reminded that the last time we discovered an error in a celestial body's trajectory we reinvented the notion of the universe.
I understand measurement error has been pretty much ruled out as well as another planet.
By the way, this news report first appeared in The Economist over a year ago.
Not the first time it has happened either. The Economist was the first non-technical journal to talk about the Internet in a general context, abck in 1991-1992.
Re:The problem with open source languages...
on
Apocalypse 2
·
· Score: 2
Is it better to have to somtimes think a bit to 'work around' flaws in the languge, or to change the languge, which makes everyone have to 'think a bit' to do anything?!?
If you consider that early in the lifetime of a language the majority of programmers have yet to learn it then it is well worth the extra moment of thought from those who pioneered programming in it.
Naturally this is somewhat subjective and it depends on the bug. I'd say "strict" in Perl is well worth the learning effort, considering how many bugs are obviated by it.
Re:The problem with open source languages...
on
Apocalypse 2
·
· Score: 3
The problem with open source languages...is that trying to learn them is a moving target.
I say that, to the contrary, the biggest problem with almost all languages, open source or not,is that they froze to early. Take any language (Java, C, C++, Perl, Python), and you can quickly name some real flaws that the designers readily admit to, but have left alone for "consistency" purposes.
I understand your frustration with having to relearn the syntax, but I believe that if the best thing to do is to byte the bullet as early as possible and fix the flaws.
Slashdot Mirror
As many others I too have lost confidence in the ability of W3C to direct the evolution of the Web.
HTML 3.0 and MathML are but two examples of how the corporate nature of the W3C has led to the delay and/or abandonment of sound technical proposals.
If you are interested in charting a new path for the web join the World Wide Web Standards Group (W4SG).
The article opens with: Many successful academic and commercial projects use direct traffic measurements (such as ping, traceroute, and web page access data) to study the structure and dynamics of the Internet. Such efforts are inherently limited by the locations of probe points required to 'cover' the Internet meaningfully. Compounding the problem, there are no effective shortcuts - simply placing agents throughout the Internet's core, as done by several commercial services, only builds up a picture of core-to-core traffic latencies and losses that has no power to predict the true "Internet weather" that end users actually experience at the network edge.
This is just plain wrong. It is quite easy to obtain latency measurements of the edge starting from the core.
Let E1 and E2 be points on the edge. If you have enough agents in the core, you will find an agent A in the path from E1 to E2. Then you can easily compute the latency from E1 to E2 by ping from A to E1 and from A to E2.
Are they going to decrypt and listen in on every VPN tunnel?
You completely missed the point.
The NSA doesn't have to listen on every VPN tunnel. If GM has a VPN between Detroit and California, one can assume that it is unlikely two terrorists would communicate using that tunnel from within GM, so no need to monitor such traffic. Since most of traffic is corporate and legit, you can eliminate all but one or two terabytes of traffic a day.
That leaves 20K of data to be processed per day for each PC-unit-of-cpu power available to the NSA.
I think you are underestimating the amount of computational power available to the NSA. I believe the NSA has enough CPU capacity to analyze every single bit that traverses their network. Think about it, the cpu power of 100 million PCs are well within their budget. That many PCs running 24x7 would produce more CPU cycles than the rest of the world combined (due to subutilization of resources elsewhere).
Moreover significant portions of communications can be thrown out rather quickly such as regular backups from established corporations, usenet redistributions down the hierarchy and the umptenth access to slashdot's web page. (A trie structure works wonders for this, and it can be distributed rather easily). I would posit that 95-97% of the web traffic Joe User generates can be discarded in this step.
Once you culled out such data you are left with a few potentially suspicious messages.
Next you record all of those using an analog device.
Then analyze all recorded data for suspicious patterns.
At this point you just hope you get lucky. At first you don't need to break all messages in a sequence of communications, you only need to break one and then track back in time your archives for related communications and break those using communication specific learned information. Planning a complex operation such as bombing the NYC would normally require hundreds of message exchanges. If your chances of detecting a pattern in communication are one-in-one-hundred you are in business.
If CNN had kept Akamai, it certainly would have helped sustained the load. However do you thnk it would have been enough, or was the traffic so taxing that nothing could have possibly done the job?
Don't forget the difference between pages and hits. A hit is anything - one of any ten-twenty images on the main page, for example.
Not in this case. CNN disabled all but one-two images per page.
CNN's main problem was that they had canceled their contract with Akamai a month or two ago to save money.
Are you sure about this?
As we speak the CNN page serves Akamaized content. Try it: open cnn.com in your browser, then say view source, and bingo: Akamai pics all over.
Remember that even if the contract was cancelled usually it takes time for the disconnect to take place.
I'm based in Montréal. Data routes to most US news site was either non-existent, or too painfull to use.
Montreal traffic is usually routed through a NY City peering point. This might have compounded your problems.
They have an order of magnitude more visitors than /.
Try three orders of magnitude. Estimates of hits at CNN were over 50,000 per second.
While my heartfell thanks go to
Slashdot was serving 50 pages per second, CNN was peaking at about an estimated 50,000 hits per second.
In light of this it was amazing that CNN was up at all, slow as it was.
Only twelve hours later did I realize that I was still sitting in the chair and that my buttocks didn't hurt. I had not twitched or slouched once in the whole session.
But you are right, Aeron chairs are a stupidity test. It tests those who think that just because they are expensive they cannot be worth the money.
My Latitude sucks. A coworker bought the another one and it sucks too. The fan and the hard drive are waaay too loud.
I guess all those watchdog boards that my next office neighbour sold in 1992-1994 for Linux systems were actually unnecessary.
Actually the entire field of guerilla warfare is predicated upon the complete opposite of your statement. Offense has the advantage over defense because they can choose the terms, time and location of engagement.
Or to quote Warren Buffet instead of Chairman Mao, investing is like a match of cricket, rather than baseball: there is no penalty for not swinging the bat.
"Yeap, it's amazing. Nowadays pretty much every piece of art our friends create starts by turning on the Mac."
"They first outline, sketch and play with it on the screen, and then proceed from there to whatever is the medium of their choice, including hitting the print button."
Well, as somebody who actually works in the field of web search engines, here's how keyword searches are seen:
"keyword searches suck, everything else sucks even more"
There is a pile of search engine carcasses proving it: the old lycos stemmer, the statistic excite approach, the OpenText structured data query, the Ask jeeves type-20-words to get the same old crappy answer you would get with a single keyword and the list goes on and on.
Yes, sooner or later researchers will find a way to go around keyword searches. But contrary to what you say, this is not bad news for Google. Nothing stops them from adopting the new technology when an alternative arises.
Indeed. Ingenious labor saving devices are common across all cultures. Ramps and slaves sounds about the dumbest way to approach the construction task. Nobody is that dumb.
As a witness we have the frozen stone man which had an ingenious collection of tools and gadgets to fix and repair his arrows and bow.
Versioned libraries does not imply having every version in existance. In all likelihood applications would support a range of versions, plus as you update to newer versions of each application, older libaries can safely be removed.
Well it'll be a lot safer to only use the libary which the programme was developed for but the development cycles are way too short.
I think this is a peculiarity of the immature state of Linux. As it ages changes will not occur as often, and improvements would be so minor that most people would hold on the upgrade until a full 1.0 change (only fresh installs would use the latest code)
What linux needs is just better scripts for creating packages and better error handling for when installing packages doesn't go right.
Those who do not learn from the past are condemned to repeat it. Automated replacement of shared libraries is the road to hell.
Each library must be versioned and each package must call explicitly a version of each library. If the desired version is not available, the installer should deploy it.
Now that would be worthy of an advanced OS. Anything else is simply an open source implementation of a good but outdated OS (aka Unix).
How about teaching her to make proper judgements, because if she's curious, she will have access to awful things one way or another (friends house, the library, even, gasp, in a book).
Plus sooner or later she will join the real world (TM) as a mature adult (C) and there will be no url log file or father sitting by her chair approving and disapproving of her choices.
How about teaching her something instead of tying a leash and threatening with a url log stick?
And what is even more amazing is the chorus of dittoheads recomending different strength leashes instead of suggesting teaching her the difference between good and bad.
So if created/written/composed by literate western man it deserves protection, if created by illiterate eastern tribe "it belongs to humanity"?
And before that, first living thing in space (a few plant specimens).
I understand measurement error has been pretty much ruled out as well as another planet.
By the way, this news report first appeared in The Economist over a year ago.
Not the first time it has happened either. The Economist was the first non-technical journal to talk about the Internet in a general context, abck in 1991-1992.
If you consider that early in the lifetime of a language the majority of programmers have yet to learn it then it is well worth the extra moment of thought from those who pioneered programming in it.
Naturally this is somewhat subjective and it depends on the bug. I'd say "strict" in Perl is well worth the learning effort, considering how many bugs are obviated by it.
I say that, to the contrary, the biggest problem with almost all languages, open source or not,is that they froze to early. Take any language (Java, C, C++, Perl, Python), and you can quickly name some real flaws that the designers readily admit to, but have left alone for "consistency" purposes.
I understand your frustration with having to relearn the syntax, but I believe that if the best thing to do is to byte the bullet as early as possible and fix the flaws.