I know AC I'm replying to was being funny, but major players like Akamai, which only uses multi-tenant data centers and sundry free colo, are also finding the ability to reduce greenhouse gases 90% per bit delivered over the last 6 years.
Argg - I accidentally moderated this point off-topic. Someone please up-vote. This post is to undo my vote.
(In my defense my browser was doing some funky pull-down menu flashing.)
1. Use a CDN and hope no one finds the origin domain or ips the CDN uses. Which as we can see from the article doesn't work due to the many ways they can be leaked. E.g., for www.example.com, try origin.www.example.com, ftp.example.com or IPs used in the past for www.example.com.
2. Have the origin servers only respond to white-listed IPs. That white-list needs to include those of the CDN. Still suspectible to a volumetric bandwidth attack. I.e., attacks with enough packets to overwhelm the origin server(s) or the ISP link to those servers.
3. Change your origin IPs periodically. Useless against a volumetric attack if they are just different IPs connected to the same uplink/router. Difficult to keep switching to use different ISP and each new provider brings its own problems.
4. Have origin(s) capable of withstanding a volumetric attack. Not cheap. The XOR DDoS botnet has recently produced DDoS attacks up to 150+ Gbps.
5. Use a BGP redirection service that routes all public internet packets whose destination IP address is the origin's through geo-graphically distributed scrubbing centers. Attackers sending traffic through the public internet to your origin are sending them to one of many scrubbing centers. The combined capacity on all these scrubbing centers can cope with volumetric attacks. The scrubbing centers will only forward desireable packets to the real origin using GRE tunneling.
The battleground — where this degradation takes place — is at ISP interconnection points. These are the places where traffic requested by ISP customers crosses between the ISP’s network and another network on which content and application providers host their services. This test measures whether interconnection points are experiencing problems. It runs speed measurements from your (the test user’s) ISP, across multiple interconnection points, thus detecting degraded performance.
What I don't understand is why people assume congestion is intentional throttling by ISPs for them to profit later with imagined fast lanes. Isn't the simpler assumption that it costs ISPs money to add interconnection capacity. And since their customers don't/can't choose ISPs based on the quality of their connection all the way to the popular content providers, the ISPs don't spend money on those upgrades? Usually the only thing customers have to go on and promised is the maximum download/upload speeds quoted by the ISP for the last mile.
If you are a windows user, Christian Zeitnitz offers a PC based Soundcard Oscilloscope free for non-commercial use. It also has a frequency spectrum waterfall diagram, x-y plots. Easy to install and run. Fun to speak into your microphone to test it out.
Only suitable audio speed signals like XOScope. I.e., 20-20000Hz from 44.1kHz sampling and 16-bit resolution. And without external hardware voltage dividers/protection the usual warnings about blowing up your soundcard if you feed in voltage outside of ±0.7V into it.
In the second-last paragraph the referenced article says:
Over the years and after doing several security assessments using social engineering techniques, nine times out of 10 we usually get caught when that one person says "I need to call someone about what you're doing."
Is that nine out of the 10 attempts they get caught?
A real intruder will not likely even try to break in with a 90% chance of going to jail.
Or out of 10 times they get caught, nine are due to this reason?
Since Akamai is the delivery platform for more than 200 global online retail companies, it has a unique view into the daily Web traffic of these sites. (Full disclosure: I work for Akamai Technologies, Inc.)
So far, according to the Akamai's real-time Network Usage Retail Index, the number of visitors per minute to online retailers is up 23% more than the average day today. Its also up 9% from the 5-month peak recorded earlier this month. Not a huge increase, but significant.
Regardless of whether you think Cyber Monday was a myth or a natural occurrence in the past, now we may be in for reality imitating the media. Consumers may expect more deals on Cyber Monday and will look for them. Retailers will respond by offering more deals. And because its a regularly scheduled event, its easy to plan press releases and advertising for.
But online visits doesn't mean more profit to retailers, or even more profit.
I wonder how many of those visitors are just buying the loss leaders, window shopping or checking the refund policy on the stuff they bought earlier in the week.
I often see statements like "I'd wager 90% of the functionality for X is only used by 5% of end users.", with the implicit assumption is that you could remove 90% and only upset 5% of the users.
Unfortunately the math is not that simple. Quite often single users do only use 5% of the features, but it not always the same features for all users. So its possible that when you remove the least used features to get almost every user lose at least one feature they must have.
What users really mean is: "I only use 5% of the features and I don't want to be affected by the ones I don't use." And "not effected" applies to UI design and security.
A system of minimal core functionality, plus opt-in, add-on plugins is a solution for this. The concept is applies to operating systems (e.g., microkernels like QNX) as well as applications. I'm not familiar with it, but I hope FireFox designed their plugin interface with the right balance of security and flexibility to achieve this.
NASA placed the reflectors on the moon and artificial satellites for this purpose since 1964. History, methods and equipment pictures are available from this pdf from
Goddard Space Flight Center who are still working at producing even more accurate equipment. The International Laser Ranging Service coordinates the data collection now from over 40 sites around the world. Many of them use existing observatories.
In 1980 I visited the MOBLAS-5 Yarragadee
station in the Western Australia outback, which was custom-built for this purpose.
MOBLAS meant MOBile LASer, and as you can see from the picture it is built in a trailer. But the equipment does not move. The site has clear night skies and no geological or human interference. And results from one location over time are valuable. It was beautiful and eerie to see the green laser beam. Even though the air was clear you could see it when standing around the pad. It seemed to last longer than the tiny fraction of a second it pulsed. At the time the operators would get a visual feedback from their instruments on who big the return signal was. And they would tweak the telescope tracking as required to get good returns. Tapes of the results were then sent back to NASA. I assume this is all automated now. Those guys looked extremely bored.
BTW, notice how big the empty tarmac is around the trailer. I was told NASA wanted a 100 by 100 foot pad and they mistakenly got a 100 by 100 metre pad, making it about 10 times bigger than they needed!
While this started out as funny, its a good example example of how unreliably misleading these search trends can be. Like a questionnaire, you can probably get any answer you want depending on the way you ask the question.
In Buenos Aires, Argentina, Finnish, Italian and Dutch it seems as if beatles are still more sought after than jesus. You might wonder how many of the hits on beatles where from bug collectors misspelling beetles? If you use their full more specific names, it seems as if The Beatles are more often searched for than Jesus Christ. http://google.com/trends?q=The+Beatles%2C+Jesus+Ch rist
aXis100 wonders why the programmers around my office read so many programming books. It seems strange that people so into technology like hunks of dead tree.
Case in point, "Programming Perl" the reference book seems to be completely reproduced in the online perldoc help supplied with Perl itself. But I end up using both. The online help is always at my fingertips when I'm programming, always up-to-date, easier to search and cut & paste. But I still use book occasionally, because I'm faster and more comfortable with a book when it comes to bulk reading.
PS: If you had to buy just one book on Perl and you already had old versions of the reference book, then I suggest the Perl Cookbook. It has covered an a lot of problems commonly encountered with excellent answers that go into great deatil about the tradeoffs of various solutions when warantted. It has solid dependable information that you won't find online - unless you buy the O'Reilly CD-ROM of the book of course.
How would you address the critique that you excessively extrapolate from a single data point to a generality?
In particular I refer to a show where you were examining fuel mileage on SUVs with windows open vs. air conditioning. As an engineer, I believe that you failed to conduct adequate experimentation to demonstrate anything other than results at a single data point and you didn't make that clear to your viewers.
Yes! Like the episode where you tried to test the myth of driving with the tailgate down increases gas mileage. After doing the parallel run of two new identical make pickup trucks, I thought you were going to repeat it by swaping the tail gate positions of the cars. One of the pickups could have had a defect or just normal variability could have led to the suprising difference observed. I know you can't test exhaustively and that's not going to make good television anyway. But remember to mention that proper science would have tried it again many times, but you didn't do it for reasons of cost or time.
That being said, you've taught us to love for science. Thank-you. And do keep doing myths about energy efficency. You have the power to reduce world pollution.
While consumers and business should be allowed to optimize their finances in the market place, it is the governments responsibility to see that all the true costs are reflected in the prices. Its a shame that this local goverment couldn't have worked in conjuction with the federal govenment to do something in the best interests of the citizens.
If all the real costs were included in pump prices then it would be a different story. Real costs like environmental cleanup and wars arising from foreign dependance.
The only way I see to incuding the real costs in the U.S. is a gradually introduced prices at the pump to pay for these. Other western countries get by with higher pump prices today. This would encourage less energy usage and investment in better energy sources which is good for the U.S. in the long term.
gru3hunt3r's says: Compaq smart array controller with battery backed write cache which used NVRAM to store data.
I can certainly back him up on that. We have them at work. Last week the disk performance went down a third on one disk array. On the BIOS bootup screen did we saw an error message that effectively said the battery had failed on the disk controller so it had to wait for each write to go to disk rather just return ok when it was written to cache. Things recovered when the battery was replaced.
And given current USB stick (which use flash) prices of $80 per GB, using flash RAM as a cache for disk seems like a good idea compared to building the whole disk out of flash.
I work for Speedera, the delivery network NASA uses for its main NASA TV live streaming link and the HTTP caching for its major web sites.
So you are right that assuming bandwidth isn't going to be a problem. Its just that NASA has to pay for these services and unlike TV, the more people that watch, the more they have to pay.
Collect lots of logs with client IP addresses and User Agents from various popular web sites.
Since www.visitorville.com is in the business of providing web stats, they are probably aggregating stats from many of their customers.
Get the mapping of which IP address blocks are owned by which companies.
You can get them the registries (e.g., ARIN, RIPE , APNIC)
by asking nicely and agreeing to use them for marketing.
Write some software that dissects user agents and OS from the User-Agent value and counts occurrences per per IP address block owner.
PS: www.visitorville.com is not currently down like intelligence.visitorville.com is.
And it was previously mentioned on Slashdot.
The original article says: We don't really stand to make any profit so we can't put a huge bankroll on this project, but we would like to have it up for holiday goodwill. That probably rules out Akamai. They seem to enjoy their reputation for having the highest priced bandwidth.
One way you can still make use of commerical Content Delivery Networks (CDNs) is by using refer blocking or a secure download service, where the file will be delivered from the CDN only by use of a time-expiring URL from your site. That way you can ensure that the media is only viewed from your site's pages and you can captitalize on any advertising.
If you move beyond Shockwave and to Flash, then Speedera is the only one of the other CDNs mentioned above that has annouced streaming from Macromedia's Flash Communications Servers.
You needed be concerned about the slashdot effect with Speedera. I've seen many Slashdot articles that link to media hosted on Speedera e.g., NASA stories). They turn out to be smaller blips that the corresponding surges from TV other major portals like Yahoo and MSN. Here's an example.
I know where SuperKendall is coming from when he talks about loss of efficiency. I myself am terribly frustrated by my reduction in productivity after was given a programer in India to manage. But that just my perspective. There's other andecdotes that show its still worthwile from a company's perspecitve. Here's one.
Last month I talked to a friend who is CEO of a company with about 100 software engineers in both Bangalore and Silicon Valley. He rated his India engineers better than his U.S. engineers. If he had to trim one location (he doesn't) it would be the U.S.
On the topic of costs, he said that while the current pay rates are 5 Bangalore engineers to 1 Silicon Valley engineer, the real costs are closer. After taking into account extra overheads (e.g., travel) and loss of productivity caused by poor communications the current overall costs are more like 3.5 to 1.
And due to rising salaries and costs in Bangalore he expects this to be 2 to 1 in a few years.
His key to making sure the loss of productivity on both ends didn't rise so far as to make it a negative sum game, was having good management at both ends capable of leading independent projects so less communications across the ocean would be needed.
BTW: USA Today just reprinted the story, so the Slashdot lead would have been better written as "The Christian Science Monitor reports..." even though the original article
is much the same.
In 1980 I remember my computer science professor saying his profession was the safest in the world - since no Professor of Computer Science in history had ever died. I believe it was true at the time.
I think a better link would have been to the Intel Research Paper. This paper describes an intergration of VMware, the Coda Distributed File System and a USB storage dongle.
The article link is light on details of the evolution algorithm, but Pablo Funes's home page has the text of his thesis on
Evolution of Complexity in Real-World Domains. It talks about his use of evolving algorithms on topics like designing the strongest lego brick structure and playing Tron. Very cool, but not its application to caching.
The is even a link to an online Tron game where us humans can play versus his evolving algorithms. The win/loss stats for his algorithm is approaching even. Given that humans can also evolve in the Tron game play, I imagine that the algorithm will have a head start over the new influx of slashdot visitors and start to win more often than not over the next week.
I never got to play though. The SQL db to mange the stats was already down then I tried.
Slashdot Mirror
I know AC I'm replying to was being funny, but major players like Akamai, which only uses multi-tenant data centers and sundry free colo, are also finding the ability to reduce greenhouse gases 90% per bit delivered over the last 6 years.
Argg - I accidentally moderated this point off-topic. Someone please up-vote. This post is to undo my vote. (In my defense my browser was doing some funky pull-down menu flashing.)
Which as we can see from the article doesn't work due to the many ways they can be leaked.
E.g., for www.example.com, try origin.www.example.com, ftp.example.com or IPs used in the past for www.example.com.
Still suspectible to a volumetric bandwidth attack. I.e., attacks with enough packets to overwhelm the origin server(s) or the ISP link to those servers.
Useless against a volumetric attack if they are just different IPs connected to the same uplink/router. Difficult to keep switching to use different ISP and each new provider brings its own problems.
Not cheap. The XOR DDoS botnet has recently produced DDoS attacks up to 150+ Gbps.
Attackers sending traffic through the public internet to your origin are sending them to one of many scrubbing centers. The combined capacity on all these scrubbing centers can cope with volumetric attacks. The scrubbing centers will only forward desireable packets to the real origin using GRE tunneling.
Akamai's BGP redirection service has some restrictions typical of other services. E.g.,
Yes, the article referenced doesn't point to the actual study directly, a but with a bit of goggling I found:
The battleground — where this degradation takes place — is at ISP interconnection points. These are the places where traffic requested by ISP customers crosses between the ISP’s network and another network on which content and application providers host their services.
This test measures whether interconnection points are experiencing problems. It runs speed measurements from your (the test user’s) ISP, across multiple interconnection points, thus detecting degraded performance.
What I don't understand is why people assume congestion is intentional throttling by ISPs for them to profit later with imagined fast lanes. Isn't the simpler assumption that it costs ISPs money to add interconnection capacity. And since their customers don't/can't choose ISPs based on the quality of their connection all the way to the popular content providers, the ISPs don't spend money on those upgrades? Usually the only thing customers have to go on and promised is the maximum download/upload speeds quoted by the ISP for the last mile.
If you are a windows user, Christian Zeitnitz offers a PC based Soundcard Oscilloscope free for non-commercial use. It also has a frequency spectrum waterfall diagram, x-y plots. Easy to install and run. Fun to speak into your microphone to test it out.
Only suitable audio speed signals like XOScope. I.e., 20-20000Hz from 44.1kHz sampling and 16-bit resolution. And without external hardware voltage dividers/protection the usual warnings about blowing up your soundcard if you feed in voltage outside of ±0.7V into it.
In the second-last paragraph the referenced article says:
Is that nine out of the 10 attempts they get caught? A real intruder will not likely even try to break in with a 90% chance of going to jail. Or out of 10 times they get caught, nine are due to this reason?
Since Akamai is the delivery platform for more than 200 global online retail companies, it has a unique view into the daily Web traffic of these sites. (Full disclosure: I work for Akamai Technologies, Inc.)
So far, according to the Akamai's real-time Network Usage Retail Index, the number of visitors per minute to online retailers is up 23% more than the average day today. Its also up 9% from the 5-month peak recorded earlier this month. Not a huge increase, but significant.
Regardless of whether you think Cyber Monday was a myth or a natural occurrence in the past, now we may be in for reality imitating the media. Consumers may expect more deals on Cyber Monday and will look for them. Retailers will respond by offering more deals. And because its a regularly scheduled event, its easy to plan press releases and advertising for.
But online visits doesn't mean more profit to retailers, or even more profit. I wonder how many of those visitors are just buying the loss leaders, window shopping or checking the refund policy on the stuff they bought earlier in the week.
I often see statements like "I'd wager 90% of the functionality for X is only used by 5% of end users.", with the implicit assumption is that you could remove 90% and only upset 5% of the users.
Unfortunately the math is not that simple. Quite often single users do only use 5% of the features, but it not always the same features for all users. So its possible that when you remove the least used features to get almost every user lose at least one feature they must have. What users really mean is: "I only use 5% of the features and I don't want to be affected by the ones I don't use." And "not effected" applies to UI design and security.
A system of minimal core functionality, plus opt-in, add-on plugins is a solution for this. The concept is applies to operating systems (e.g., microkernels like QNX) as well as applications. I'm not familiar with it, but I hope FireFox designed their plugin interface with the right balance of security and flexibility to achieve this.
In 1980 I visited the MOBLAS-5 Yarragadee station in the Western Australia outback, which was custom-built for this purpose. MOBLAS meant MOBile LASer, and as you can see from the picture it is built in a trailer. But the equipment does not move. The site has clear night skies and no geological or human interference. And results from one location over time are valuable. It was beautiful and eerie to see the green laser beam. Even though the air was clear you could see it when standing around the pad. It seemed to last longer than the tiny fraction of a second it pulsed. At the time the operators would get a visual feedback from their instruments on who big the return signal was. And they would tweak the telescope tracking as required to get good returns. Tapes of the results were then sent back to NASA. I assume this is all automated now. Those guys looked extremely bored.
BTW, notice how big the empty tarmac is around the trailer. I was told NASA wanted a 100 by 100 foot pad and they mistakenly got a 100 by 100 metre pad, making it about 10 times bigger than they needed!
While this started out as funny, its a good example example of how unreliably misleading these search trends can be. Like a questionnaire, you can probably get any answer you want depending on the way you ask the question.
h rist
In Buenos Aires, Argentina, Finnish, Italian and Dutch it seems as if beatles are still more sought after than jesus. You might wonder how many of the hits on beatles where from bug collectors misspelling beetles? If you use their full more specific names, it seems as if The Beatles are more often searched for than Jesus Christ.
http://google.com/trends?q=The+Beatles%2C+Jesus+C
Lies, Damn Lies and Statistics.
Case in point, "Programming Perl" the reference book seems to be completely reproduced in the online perldoc help supplied with Perl itself. But I end up using both. The online help is always at my fingertips when I'm programming, always up-to-date, easier to search and cut & paste. But I still use book occasionally, because I'm faster and more comfortable with a book when it comes to bulk reading.
PS: If you had to buy just one book on Perl and you already had old versions of the reference book, then I suggest the Perl Cookbook. It has covered an a lot of problems commonly encountered with excellent answers that go into great deatil about the tradeoffs of various solutions when warantted. It has solid dependable information that you won't find online - unless you buy the O'Reilly CD-ROM of the book of course.
That being said, you've taught us to love for science. Thank-you. And do keep doing myths about energy efficency. You have the power to reduce world pollution.
If all the real costs were included in pump prices then it would be a different story. Real costs like environmental cleanup and wars arising from foreign dependance. The only way I see to incuding the real costs in the U.S. is a gradually introduced prices at the pump to pay for these. Other western countries get by with higher pump prices today. This would encourage less energy usage and investment in better energy sources which is good for the U.S. in the long term.
I can certainly back him up on that. We have them at work. Last week the disk performance went down a third on one disk array. On the BIOS bootup screen did we saw an error message that effectively said the battery had failed on the disk controller so it had to wait for each write to go to disk rather just return ok when it was written to cache. Things recovered when the battery was replaced.
And given current USB stick (which use flash) prices of $80 per GB, using flash RAM as a cache for disk seems like a good idea compared to building the whole disk out of flash.
For some statistics here's some press releases and my previous posting.
For some statistics here's some press releases and my previous posting.
Live video feed from NASA TV too.
- Collect lots of logs with client IP addresses and User Agents from various popular web sites.
- Get the mapping of which IP address blocks are owned by which companies.
- Write some software that dissects user agents and OS from the User-Agent value and counts occurrences per per IP address block owner.
PS: www.visitorville.com is not currently down like intelligence.visitorville.com is. And it was previously mentioned on Slashdot.Since www.visitorville.com is in the business of providing web stats, they are probably aggregating stats from many of their customers.
You can get them the registries (e.g., ARIN, RIPE , APNIC) by asking nicely and agreeing to use them for marketing.
One way you can still make use of commerical Content Delivery Networks (CDNs) is by using refer blocking or a secure download service, where the file will be delivered from the CDN only by use of a time-expiring URL from your site. That way you can ensure that the media is only viewed from your site's pages and you can captitalize on any advertising. If you move beyond Shockwave and to Flash, then Speedera is the only one of the other CDNs mentioned above that has annouced streaming from Macromedia's Flash Communications Servers.
You needed be concerned about the slashdot effect with Speedera. I've seen many Slashdot articles that link to media hosted on Speedera e.g., NASA stories). They turn out to be smaller blips that the corresponding surges from TV other major portals like Yahoo and MSN. Here's an example.
Disclaimer: I work for Speedera Networks, Inc.
Last month I talked to a friend who is CEO of a company with about 100 software engineers in both Bangalore and Silicon Valley. He rated his India engineers better than his U.S. engineers. If he had to trim one location (he doesn't) it would be the U.S. On the topic of costs, he said that while the current pay rates are 5 Bangalore engineers to 1 Silicon Valley engineer, the real costs are closer. After taking into account extra overheads (e.g., travel) and loss of productivity caused by poor communications the current overall costs are more like 3.5 to 1. And due to rising salaries and costs in Bangalore he expects this to be 2 to 1 in a few years.
His key to making sure the loss of productivity on both ends didn't rise so far as to make it a negative sum game, was having good management at both ends capable of leading independent projects so less communications across the ocean would be needed.
BTW: USA Today just reprinted the story, so the Slashdot lead would have been better written as "The Christian Science Monitor reports ..." even though the original article
is much the same.
In 1980 I remember my computer science professor saying his profession was the safest in the world - since no Professor of Computer Science in history had ever died. I believe it was true at the time.
The beta doesn't have an advanced search tab.
How am I supposed to search just recent activity?
I think a better link would have been to the Intel Research Paper. This paper describes an intergration of VMware, the Coda Distributed File System and a USB storage dongle.
It also has a quote from Akamai saying "The attack was a result of a virus or worm infiltrating a system".
The is even a link to an online Tron game where us humans can play versus his evolving algorithms. The win/loss stats for his algorithm is approaching even. Given that humans can also evolve in the Tron game play, I imagine that the algorithm will have a head start over the new influx of slashdot visitors and start to win more often than not over the next week. I never got to play though. The SQL db to mange the stats was already down then I tried.