Instead of focussing on what format or server you need, just stop, take a step back, and ask yourself if you should be the one providing the service. Like television and radio studios of the past, we now have net-studios, places with A/V equipment and studios, video pumping gear, dependable dedicated operations staff and good connectivity. They were the pioneers of the road which you have probably not yet traveled.
InterVu (bought by Akamai) comes to mind. They were one of the first to directly broadcast live events and are now one of the best.
You can then have more time to focus on your web site and managing the content, not the service.
There goes the neighborhood. Just when there isn't enough power in PG&E territory, some feds are gonna suck even more power from the grid. Not in my back yard, dammit.;^)
When it's not crunching numbers at 1000 Tflops, will they at least shut down unused parts of the cluster to be wake-on-LAN?
Anyone notice how it's the spammers and spam supporters who like to whine openly about how evil a SPAM-advisory-service is? One is not known only by the company they keep, but the enemies they make. Paul's in good company. The methods might seem a little draconian to the rest of the people who aren't as adamant about spam, but that's EXACTLY what the subscribers of the information want.
A long time ago, Carl Malamud and Marshal Rose came up with a novel system for mapping phone numbers to hostnames. Go to www.tpc.int for more info. It was an internet-based free worldwide faxing service. A quick summary of the convention they use:
Phone numbers are heirarchical by country code, then by area code or city code in larger countries, then by phone number. The number of digits in a phone number aren't ever the same from country to country. The domain name system reverses heirarchy (right to left). Given these constraints, they came up with the convention of COVER_PAGE_INFO@PHONE_NUMBER_REVERSED.tpc.int. If I take a typical US phone number like
+1-212-555-6789, the host name mapping for it would be: 9.8.7.6.5.5.5.2.1.2.1.tpc.int .
Something similar could be used for addressing host names to people's cell phones for messaging or other mobile phone-addressable devices. I doubt people themselves will ever map a phone number to this reversed notation, but software could do it automatically for them.
/dev/earth is 98% full, please remove people immediately
Come to think about it, our ISP exchanges are pretty ludicrous. Miles upon miles of Cat5 and fiber linking large, power-hungry and expensive routers and switching equipment.
I really like the idea posed near the end of the article about a box of GSVs being used for exchanging telecommunications traffic.
It'd be interesting to have an exchange room filled with an inert gas where ISPs rent "wall space". One one side of the room, ISPs transmit data for other ISPs to collect on the other side. One ISP can transmit at some frequency where another on the opposing wall can receive the signals simultaneously with signals from all other providers. Multiplaxing and/or rerouting can be accomplished using prisms or mirrors inside the free space. For diagnostics, just add a tiny bit of air and smoke and watch the spectacular light show!
Perhaps those tailors for the Emperor's new clothes were innocent after all. Our ancestors of lowly tri-color ability couldn't see the fourth color in his wardrobe.
Jump on the stocks now! The Nokia faceplate, Crayola crayon, M&M candy, VW Beetle, and Apple iMac can now come out with another limited edition color for their produts. Sales will soar! Only those with distinguishing taste will be able to figure out it's not greenish red.
A recent survey of my (pretty popular) web site showed 78% IE-derived browsers and 18% Mozilla-derived. I wonder what percentage of the Mozilla users were actually from AOL. There's not much incentive for a Windows user (95% of our users) these days to actually go out and download a second bloated browser suite onto their system.
As I surf web sites, more are putting features into the browser that are best viewed on Windows boxes. The old standard for HTML-rich sites used to be "Our web page is best viewed with a Netscape or Microsoft browser". Now people mainly develop for IE first and work in compatability for Mozilla, AOL and/or WebTV (still a significant percentage, but dwindling into unimportance).
AOL bought Netscape, so AOL will keep it alive until it make more sense to put something else on their direct-mail CDs. Long live Mozilla.
What's up with Neoplanet? They used to be a cool alternative to Mozilla when it didn't crash my Windows PC.
I totally agree with this one. Online storage was so reliable and handy when I was in college. I could download my files to any workstation (Mac, PC, Unix), and I knew they were safe since the University did tape backups on a regular basis. Most students could use 5MB and be happy.
Instead of FTP (insecure) or SCP (clumsy to use, not supported well on every platform), consider a https web server as the main interface. There's an INPUT FILE directive that works well for uploads using any browser on any (popular) platform.
Back in the day (1991? 92?), batch.uu.net was an expensive Sun 640MP that just couldn't cut it. Pushing USENET through the box was hard enough. Compressing batches of news articles for dialup customers was more than it could handle. Instead of buying more hardware, our fearless leader came up with the idea of using all of the idle machines in the office at night (Sparc SLC/ELC/SS1/SS2) to run compress on them through rsh pipes.
Moral - a good sysadmin in your hand is worth two P2P sales reps in the bush.
For the most free computing power at your fingertips, hire script kiddies.
Anyone who lied on the InterNIC forms back in the early days got what they wanted. There was a time when you had to be a 503(c) non-profit ORGanization to get a ".ORG" domain. That's not the case now.
I suspect that anyone who can lie good enough to say they have accredidation can get a.EDU domain.
At least NetSol filters out the really stupid requests (SEX.EDU).
None of the subdomains suggested by that poster (".com.us", ".net.us") will actually help the system for ".US", just take it further down its path of commercial greed.
I've got 70 sq. miles of virgin Jovian-front property available, cheap. Spectacular views! Excellent place to get away from it all! PHONE HOME or e-mail for more details.
Also, contact Pizza Hut for possible franchising opportunities: "Pizza Hut is a pioneer in space commercialization" - Rick Hieb.
Could someone please setup a few root servers that adds "/" as a top-level hostname/domainname and gives me a CNAME to www.slashdot.org? I type "w-w-w-.-s-l-a-s-h-d-o-t-.-o-r-g-." way too often and I don't have good enough speech recognition software to save me the keystrokes. Thanks, EZ
There was a time at the beginning of 1996 when UUNET led the clamping down of peering. Their only true "peers" were SpirintLink and InternetMCI. Everyone else was a bottomfeeder that were destined to buy trasit from some larger Tier 1 provider like the "Big 3".
Within these three providers there was an oligopoly. They competed with each other - and when combined they competed with the rest of the ISP industry.
C&W doesn't have much marekting muscle to attact new providers. Their subscriber base probably isn't inscreasing as fast as it would have if it were MCI.
The same probably would have happened to SprintLink - having a good initial customer base, but not much left for marketing.
I have no idea what goes on behind the scenes at WorldCom these days (not employed by WCOM, but it's not like employees know anything either), but I suspect the withdrawal from EU consideration is either a tactical move (using the DoJ as an excuse to keep the EU from forcing an unfavorable vote) or a setback (maybe the DoJ was acting independently). Don't be surprised if something else surfaces. Execs at WCOM and FON were both committed to this deal at some point and so were the stockholders.
For wireless - I was a big fan of the Nextel merger. It fit well and made alot of sense. WCOM claims it was a difference over cash that made negotiations fall apart. I personally think it was ego. Now Nextel is way too expensive (more than double the value). Unfortunately, so is everything else. What's left? "Verizon"?:^( They'd be better off doing it themselves under the Skytel name.
While I don't have a specific answer to the question, I do have some thoughts on things I've learned along the way.
I once had a friend at a start-up ask me whether they should colo or do it themselves. I have an edited reply here: http://www.nspf.net/colo.txt. In it, I talk about several things that they or a colo provider would have to think about and plan for if they were building and managing a data center. As you look at a data center, you might want to think of some of these issues as well.
One of the older reasons that people started using colocation was that local loop charges from an office into the ISPs was expensive. By putting a server at an ISP, you avoided those charges. The data centers at the colo were better than one's typical office. Once you get past DS3 speeds, the LEC charges don't look as expensive as they used to. Smaller sites (<=10Mbps) are still better served at a colo facility.
Recently, the reasons for picking a colo provider have morphed into: "What am I willing to outsource?" When you pick a colo site, you're trusting your machines, networking, facilities management and physical security to another company. Are they better at doing it than you?
Some colo providers distinguish themselves in various ways. One might be better-connected into AOL or broadband networks. Some focus on connectivity. Some might be better aquainted with NT than others. Some might have a great daytime Sun or Linux staff. One might offer database outsourcing/management. One might offer backups. One might rent you EMC/SRDF disk space. One might offer managed servers instead of using your servers. These features could good reasons to select one over another, but only if they complement your operation. Make sure, though, that you don't get attached too much to any colo provider's value-adds. For example, if they have network problems every week and your data is trapped in their managed database, you'll have to live with it until you can duplicate the functionality that you'll have to leave behind.
Initially, in an R&D and trial/rollout phase, many companies can get away with outsourcing as much as they can to focus on their site's development, but as the site becomes more popular and as users depend on it more, the uptime and reliability of the site becomes much more important. You can't just let your colo or network provider screw up anymore. It's unacceptable. Decrease your dependence on any colo provider as your site becomes more important.
A very annoying feature of running a colocation site is working remotely. If you need to do anything more than hit a power button on a machine, you need to figure out a way to not have to do that. Don't change tapes, buy an autoloader with lots of tape capacity. Don't use machines that need keyboards/monitors - use serial console servers to access the serial consoles of your devices or use something like Citrix to manage your NT boxes. Buy reliable machines that don't crash as often. Inside a server, a 10000 RPM or 15000 RPM disk tends to fail more than a 7200 RPM disk. A beefier power supply running at half capacity runs longer than a cheaper one running at capacity. Buy more servers than you need for everything so that you can migrate your service from failed servers to standby ones. Don't run the latest version of an operating system. Run the most stable/patched version of the OS. Eliminate all single points of failure from the networking side (including having more than one upstream ISP if possible). Routers and layer4 gear need to reboot sometimes. Always buy more than one of each.
The best way to avoid failure is to have multiple data centers. You can care less about the reliability/availability of one data center because you can always direct traffic to another one. Many web sites make a mistake early on of building a single dependency into their site, whether it's a database or a filesystem, something keeps the site from running in parallel with a similarly-configured. Plan from the start by running your web site from 3 locations, and you'll be able to scale your site very well. You'll also be able to pick and choose from cheaper colo providers that don't do N+1 redundancy to help reduce your costs.
At some point, every colo provider will let you down. It's inevitable that something bad will happen. Picking a colo provider that learns from its mistakes can be better than one that strives to make none. We would expect the same of our employees if we ran a data center ourselves.
If you're good at managing servers remotely, the location of your site(s) becomes less important:
I have a friend who colocated his servers in San Diego while his team worked in the Bay Area. One of his justifications was that it was just as fast to drive from his office in SF to Oakland and fly a plane to San Diego than it was to drive from San Francisco to San Jose during rush hour. Becasue he had skilled remote hands, he didn't have to fly down often.
Some thoughts on networking for colo providers:
I used to think that I wanted to have servers closer to the exchange points. Exchange points are crowded places for traffic. If something happens elsewhere on a network, the traffic that is rerouted near your upstream routers might saturate your connectivity to anywhere. I'd now rather put my servers in a less-crowded area (St Louis? Denver? Dallas?) with good connectivity and let my bits get to most of the country/world without the congestion at the peering points.
A Tier 1 ISP may not be as good as a Tier 2 ISP with multiple transit paths into Tier 1 ISPs. I use SimpleNet in SanDiego as an example. They have six or more DS3/OC3 circuits out of their data center to different providers. Most other colo providers in our area have only one or two outbound paths. If connectivity out through one ISP starts to suck (fiber cut, BGP flapping), an ISP with multiple transit links can just shut off the bad ISP until it recovers.
If XX% of my traffic goes to a particular online service provider (AOL, CompuServe), a broadband network (@Home, RoadRunner), a WAP network (Sprint, others), through a portal (Yahoo, Excite), or to a dial-up audience (UUNET, AT&T), I'd want my servers to have really good conectivity to their network. I might choose a data center that has special peering or connectivity to my target network.
It really sucks when a colo provider oversells their capacity. Find out how much capacity a colo provider advertises. Then figure out how much 40% of that capacity is. Thell them you have BIG plans for a quick rollout. If they say they can handle it and don't suggest that they to order more capacity first, then they are likely to bump against their physical outbound limit which could mean packet loss for your site and every site withing the data center. You, already being a customer, would lose. If a colo puts you and everyone else behind bandwith limiters (like Xedia), that's good. They can rate-limit bandwidth hogs to keep everyone else running smoothly.
Random musings:
One set of colo providers that I'm gaining respect for are the ones that try to do as little as possible. Imagine a climate-controlled U-Haul storage facility with padlocks in front of 10x10 rooms. Add conduits for cables that go to other cages that contain LEC fiber or ISPs' routers. Even a high school dropout can reliably run such a facility. I can take care of my own racks and networking. I can deal directly with my upstream ISP's without the colo middle man. Equinix follows this model.
Another type of colo is gaining my interest as well. Look at rackspace.com. Dell and Intel and perhaps others have the rent-a-server model. Sure, it's managed-server like others offer, but they keep themselves on the physical/parts side and can custom-build and install servers to my specs within a couple days of lead time. They don't try to manage the server at all - just rent me assembled server parts and bandwidth. I would no longer have to buy servers, just rent them. If there's a problem with a server, I can have them repair the server or build me a new one right away. I don't have to wait for my vendor to ship me a spare part to install - it's their server, and they damned well better have spare parts.
Distributing static content through a CDP like SandPiper or Akamai can significantly help the scalability of your web site. The only machines that you'd have to colocate would be site-critical back-end servers and masters for your site's content. A serious mistake that sites make is pumping too much bandwidth out of their colo provider. The CDPs have gigabits of good bandwidth all around the world to rent you. Try not to use more than, say, 20% of your colo provider's available bandwidth. As your site grows, your colo provider (with 4-6 months lead time on new fiber uplinks) might not be able to keep up with you if you depend too much on them for bandwidth.
If you have servers in a remote colo facility, like on another coast, make friends with a local geek or have a really smart consultant available near that facility to save yourself 5-hour plane trips. Frequent flyer miles don't make up for the time you waste on airplanes, and it can get old fast.
You can never have enough tools or spare parts in your colo facility.
Just stuff to think about. -- Eric Ziegast
(PS: I used to work at ISPs and colo providers just like synx. I currently help run a very popular web site at several different colo facilities.)
Stripes writes: Netbooting is still important. If I want a quiet machine hooked to my stereo to play MP3s netbooting would let me avoid a hard drive (assuming I fetch the MP3s over the net, which is more then fast enough). Using a StrongARM, or other low power CPU would let me avoid the need for a CPU fan as well, so no moving noisy parts at all.
Yes, but you'd still be attached via a cable to your network. Those cableless players with disk drives are more appealing if you spend any amount of time on a plane or public transit.
On a plane ride back home recently, I picked up this month's Red Herring (May 2000). They had a really good read about wireless technologies. Of particular interest was an article about a relatively new technology called Bluetooth. Check it out. It could become as fundamental as ethernet to the way we network.
Check out www.bluetooth.com or look for Bluetooth in Google or Yahoo.
A poster asks: Just from a theretical point of view, how difficult do you think it would be to take those servers down from terrorist activity. I mean could the internet be taken down if 12 explosions at the right time/place where detonated?
Stripes starts his reply: Assuming you can figure out where they all are form the IP addresses in the root.cache file, and traceroute, or other similar tools, and maybe a bit of social engenering, it shouldn't be any harder then any other 12 randomly selected machines.
Define "explosions"
Stripes, The poster to which you responded did not specify what type of explosions were available to them. If they're nuclear explosions, they'd probably need only 8-10 strategically placed explosions to wipe out all of the current neameservers (with or without social engineering). If they're lucky, they might take out the "shadow root servers" as well. Given the location of some of the root servers, they'd probably cripple alot more than just DNS. They'd effectively take out a good deal of infrastructure as well as the Internet engineers necessary to repair it, not to mention start a worldwide panic.
The Internet would still recover though, much as you described in your post. Anyone can setup a redundant server cluster within a matter of minutes given a set of pre-staged root and first level zone data.
The more interesting problems are due to corrupted data rather than doing denial of service attacks on nameservers. Some bad data in Network Solution's database can make various interesting parts of the Internet suck really bad. When one root server has data corruption, the whole net feels it. Imagine if some NSOL staffer garbled the nameserver data for "Yahoo.COM." or "IN-ADDR.ARPA." to point to 255.255.255.255 instead of the real servers?
For anyone else interested in DNS DoS...
An easier method
One of the easiest way to kill DNS is to try a coordinated DoS attack against all of the nameservers. Each of the world hundreds of thousands of resolvers is configured to use any of 13 root nameservers. Just like a 15-year-old kid did with HTTP requests, one could probably start a distributed DoS attack against DNS. The "heftiest" root nameserver is rumored somewhere in this discussion to be able to handle 6000-8000 hits a second. With 13 published nameservers, one needs only about 100000 hits per second to saturate the current capacity of all of the servers. Let's say that I was a bright hacker (which I'm not) that I could find my way into 1000 machines around the world that each had a T1 connection or better. Can we agree that this is a difficult but not unreasonably impossible thing to do? If one were not smart enough to do it themselves, one could perhaps go to a hacker convention or local user group and bribe a script kiddie seeking infamy and fortune to go forth an find 1000 machines to hack. Another way is to unleash a time-dated virus onto the net that will do your bidding at a specific time. Each machine would gather a list of 100 addresses, perhaps starting with the history file of a user's browser to get a list of second-level domains. It could also look for addresses using a popular portal directory or search engine and interpret results to get domain names. With 100 domain names, it would query 100 names per second (less than one megabit) from each of the few registered root nmeservers. While the traffic isn't overwhelming, it will overload the root servers fo rthe number of transactions per second, and nothing short of hunting and killing half of the query servers would reduce the effectiveness of the attack. To make the attack harder to stop, one could double or quadruple the number of query servers or use methods of masquerading your attack (I won't go into detail here) to keep network administrators from being able to shut down query servers. Another way to scale the attack is to use they heavier TCP protocol for most of the queries instead of the lightweight UDP.
fin.
The technology needed to exponentially increase the ability of the root servers to perform is not out of reach. With the proper motivation (a DoS like I described), one million dollars of capital (compare $1m to the current valuation of NSOL), and perhaps 30 man-weeks of time, one can make a farm of servers able to handle two orders of magnitude more requests than the current set of servers.
The IBM server announcement by Network Solutions disappoints me. It's sad.
Any of the following are good candidates that I know about for scalably solving root DNS infrastructure problems...
UltraDNS - DNS service provider with an interesting spin on distributed scalability
Nominum - the knowledge and knowhow to make fast scalable DNS servers and software
Akamai/Sandpiper - a distributed operations infrastructure onto which one can install root clusters.
Hint: If one can make an application layer proxy host that takes inbound DNS requests and routes them based on a hash table of domain names to a set of back end nameservers (with only a fraction of domains loaded on each), one could have the start of a scalable solution. One can make a fast cheap BSD box to do this up to 5000 ops per second or better. I wonder if the skunk works at Novell can do ths faster. One can use some router technology (OSPF, trunking, or L4 switching) to spray UDP requests to a number of these appliction load balancer / DNS proxy servers.
One can also implement interesting filters on such a proxy server to reduce the effect of stupider resolvers or lame DoS attacks.
-- Eric Ziegast
PS: Slashdot probably isn't the best forum for this, but if you know a better forum, feel free to point them toward this post.
I agree that simple solutions may be better. I give an example of a site that uses two Oracle servers that act independently (seperate systems, seperate disk systems) from each other. One server is primary. The other is a backup. The clients are setup to read off the primary and writes/modifies are done to both. When doing backups, the backup server is used without affecting load on the primary server. If the primary server goes down, the clients start reading from the secondary. All you need to make this work is either better client access methods that you program, or you need a client to access a proxy server/service that forwards requests to the appropriate servers. While they used Oracle, this technique isn't hard to implement for other systems. The site is aware of technologies like Oracle Parallel Server and Veritas Cluster Server, but they like their method the best. Note for jrbrtsn: Disks are one of the first things I'd worry about with a database. If you switch disks from one system to another after a failure, there's no guarantee that the data on them won't have problems.
Slashdot Mirror
InterVu (bought by Akamai) comes to mind. They were one of the first to directly broadcast live events and are now one of the best.
You can then have more time to focus on your web site and managing the content, not the service.
There goes the neighborhood. Just when there isn't enough power in PG&E territory, some feds are gonna suck even more power from the grid. Not in my back yard, dammit. ;^)
When it's not crunching numbers at 1000 Tflops, will they at least shut down unused parts of the cluster to be wake-on-LAN?
-ez
Anyone notice how it's the spammers and spam supporters who like to whine openly about how evil a SPAM-advisory-service is? One is not known only by the company they keep, but the enemies they make. Paul's in good company. The methods might seem a little draconian to the rest of the people who aren't as adamant about spam, but that's EXACTLY what the subscribers of the information want.
- Phone numbers are heirarchical by country code, then by area code or city code in larger countries, then by phone number. The number of digits in a phone number aren't ever the same from country to country. The domain name system reverses heirarchy (right to left). Given these constraints, they came up with the convention of COVER_PAGE_INFO@PHONE_NUMBER_REVERSED.tpc.int. If I take a typical US phone number like
- +1-212-555-6789, the host name mapping for it would be: 9.8.7.6.5.5.5.2.1.2.1.tpc.int .
Something similar could be used for addressing host names to people's cell phones for messaging or other mobile phone-addressable devices. I doubt people themselves will ever map a phone number to this reversed notation, but software could do it automatically for them.Come to think about it, our ISP exchanges are pretty ludicrous. Miles upon miles of Cat5 and fiber linking large, power-hungry and expensive routers and switching equipment.
I really like the idea posed near the end of the article about a box of GSVs being used for exchanging telecommunications traffic.
It'd be interesting to have an exchange room filled with an inert gas where ISPs rent "wall space". One one side of the room, ISPs transmit data for other ISPs to collect on the other side. One ISP can transmit at some frequency where another on the opposing wall can receive the signals simultaneously with signals from all other providers. Multiplaxing and/or rerouting can be accomplished using prisms or mirrors inside the free space. For diagnostics, just add a tiny bit of air and smoke and watch the spectacular light show!
... just daydreaming.
Perhaps those tailors for the Emperor's new clothes were innocent after all. Our ancestors of lowly tri-color ability couldn't see the fourth color in his wardrobe.
Jump on the stocks now! The Nokia faceplate, Crayola crayon, M&M candy, VW Beetle, and Apple iMac can now come out with another limited edition color for their produts. Sales will soar! Only those with distinguishing taste will be able to figure out it's not greenish red.
A recent survey of my (pretty popular) web site showed 78% IE-derived browsers and 18% Mozilla-derived. I wonder what percentage of the Mozilla users were actually from AOL. There's not much incentive for a Windows user (95% of our users) these days to actually go out and download a second bloated browser suite onto their system.
As I surf web sites, more are putting features into the browser that are best viewed on Windows boxes. The old standard for HTML-rich sites used to be "Our web page is best viewed with a Netscape or Microsoft browser". Now people mainly develop for IE first and work in compatability for Mozilla, AOL and/or WebTV (still a significant percentage, but dwindling into unimportance).
AOL bought Netscape, so AOL will keep it alive until it make more sense to put something else on their direct-mail CDs. Long live Mozilla.
What's up with Neoplanet? They used to be a cool alternative to Mozilla when it didn't crash my Windows PC.
-- EZ
Instead of FTP (insecure) or SCP (clumsy to use, not supported well on every platform), consider a https web server as the main interface. There's an INPUT FILE directive that works well for uploads using any browser on any (popular) platform.
For example, 24.24.0.0 belongs to Time Warner cable.
Back in the day (1991? 92?), batch.uu.net was an expensive Sun 640MP that just couldn't cut it. Pushing USENET through the box was hard enough. Compressing batches of news articles for dialup customers was more than it could handle. Instead of buying more hardware, our fearless leader came up with the idea of using all of the idle machines in the office at night (Sparc SLC/ELC/SS1/SS2) to run compress on them through rsh pipes.
Moral - a good sysadmin in your hand is worth two P2P sales reps in the bush.
For the most free computing power at your fingertips, hire script kiddies.
Perhaps it was too popular. ;^)
Anyone who lied on the InterNIC forms back in the early days got what they wanted. There was a time when you had to be a 503(c) non-profit ORGanization to get a ".ORG" domain. That's not the case now. I suspect that anyone who can lie good enough to say they have accredidation can get a .EDU domain.
At least NetSol filters out the really stupid requests (SEX.EDU).
None of the subdomains suggested by that poster (".com.us", ".net.us") will actually help the system for ".US", just take it further down its path of commercial greed.
Also, contact Pizza Hut for possible franchising opportunities:
"Pizza Hut is a pioneer in space commercialization" - Rick Hieb.
Could someone please setup a few root servers that adds "/" as a top-level hostname/domainname and gives me a CNAME to www.slashdot.org? I type "w-w-w-.-s-l-a-s-h-d-o-t-.-o-r-g-." way too often and I don't have good enough speech recognition software to save me the keystrokes. Thanks, EZ
Within these three providers there was an oligopoly. They competed with each other - and when combined they competed with the rest of the ISP industry.
C&W doesn't have much marekting muscle to attact new providers. Their subscriber base probably isn't inscreasing as fast as it would have if it were MCI.
The same probably would have happened to SprintLink - having a good initial customer base, but not much left for marketing.
I have no idea what goes on behind the scenes at WorldCom these days (not employed by WCOM, but it's not like employees know anything either), but I suspect the withdrawal from EU consideration is either a tactical move (using the DoJ as an excuse to keep the EU from forcing an unfavorable vote) or a setback (maybe the DoJ was acting independently). Don't be surprised if something else surfaces. Execs at WCOM and FON were both committed to this deal at some point and so were the stockholders.
For wireless - I was a big fan of the Nextel merger. It fit well and made alot of sense. WCOM claims it was a difference over cash that made negotiations fall apart. I personally think it was ego. Now Nextel is way too expensive (more than double the value). Unfortunately, so is everything else. What's left? "Verizon"? :^( They'd be better off doing it themselves under the Skytel name.
--
Eric Ziegast
Disclaimer: I own some WCOM stock.
I once had a friend at a start-up ask me whether they should colo or do it themselves. I have an edited reply here: http://www.nspf.net/colo.txt. In it, I talk about several things that they or a colo provider would have to think about and plan for if they were building and managing a data center. As you look at a data center, you might want to think of some of these issues as well.
One of the older reasons that people started using colocation was that local loop charges from an office into the ISPs was expensive. By putting a server at an ISP, you avoided those charges. The data centers at the colo were better than one's typical office. Once you get past DS3 speeds, the LEC charges don't look as expensive as they used to. Smaller sites (<=10Mbps) are still better served at a colo facility.
Recently, the reasons for picking a colo provider have morphed into: "What am I willing to outsource?" When you pick a colo site, you're trusting your machines, networking, facilities management and physical security to another company. Are they better at doing it than you?
Some colo providers distinguish themselves in various ways. One might be better-connected into AOL or broadband networks. Some focus on connectivity. Some might be better aquainted with NT than others. Some might have a great daytime Sun or Linux staff. One might offer database outsourcing/management. One might offer backups. One might rent you EMC/SRDF disk space. One might offer managed servers instead of using your servers. These features could good reasons to select one over another, but only if they complement your operation. Make sure, though, that you don't get attached too much to any colo provider's value-adds. For example, if they have network problems every week and your data is trapped in their managed database, you'll have to live with it until you can duplicate the functionality that you'll have to leave behind.
Initially, in an R&D and trial/rollout phase, many companies can get away with outsourcing as much as they can to focus on their site's development, but as the site becomes more popular and as users depend on it more, the uptime and reliability of the site becomes much more important. You can't just let your colo or network provider screw up anymore. It's unacceptable. Decrease your dependence on any colo provider as your site becomes more important.
A very annoying feature of running a colocation site is working remotely. If you need to do anything more than hit a power button on a machine, you need to figure out a way to not have to do that. Don't change tapes, buy an autoloader with lots of tape capacity. Don't use machines that need keyboards/monitors - use serial console servers to access the serial consoles of your devices or use something like Citrix to manage your NT boxes. Buy reliable machines that don't crash as often. Inside a server, a 10000 RPM or 15000 RPM disk tends to fail more than a 7200 RPM disk. A beefier power supply running at half capacity runs longer than a cheaper one running at capacity. Buy more servers than you need for everything so that you can migrate your service from failed servers to standby ones. Don't run the latest version of an operating system. Run the most stable/patched version of the OS. Eliminate all single points of failure from the networking side (including having more than one upstream ISP if possible). Routers and layer4 gear need to reboot sometimes. Always buy more than one of each.
The best way to avoid failure is to have multiple data centers. You can care less about the reliability/availability of one data center because you can always direct traffic to another one. Many web sites make a mistake early on of building a single dependency into their site, whether it's a database or a filesystem, something keeps the site from running in parallel with a similarly-configured. Plan from the start by running your web site from 3 locations, and you'll be able to scale your site very well. You'll also be able to pick and choose from cheaper colo providers that don't do N+1 redundancy to help reduce your costs.
At some point, every colo provider will let you down. It's inevitable that something bad will happen. Picking a colo provider that learns from its mistakes can be better than one that strives to make none. We would expect the same of our employees if we ran a data center ourselves.
If you're good at managing servers remotely, the location of your site(s) becomes less important:
Some thoughts on networking for colo providers:
Random musings:
Just stuff to think about.
--
Eric Ziegast
(PS: I used to work at ISPs and colo providers just like synx. I currently help run a very popular web site at several different colo facilities.)
Netbooting is still important. If I want a quiet machine hooked to my stereo to play MP3s netbooting would let me avoid a hard drive (assuming I fetch the MP3s over the net, which is more then fast enough). Using a StrongARM, or other low power CPU would let me avoid the need for a CPU fan as well, so no moving noisy parts at all.
Yes, but you'd still be attached via a cable to your network. Those cableless players with disk drives are more appealing if you spend any amount of time on a plane or public transit.
On a plane ride back home recently, I picked up this month's Red Herring (May 2000). They had a really good read about wireless technologies. Of particular interest was an article about a relatively new technology called Bluetooth. Check it out. It could become as fundamental as ethernet to the way we network.
Check out www.bluetooth.com or look for Bluetooth in Google or Yahoo.
Just from a theretical point of view, how difficult do you think it would be to take those servers down from terrorist activity. I mean could the internet be taken down if 12 explosions at the right time/place where detonated?
Stripes starts his reply:
Assuming you can figure out where they all are form the IP addresses in the root.cache file, and traceroute, or other similar tools, and maybe a bit of social engenering, it shouldn't be any harder then any other 12 randomly selected machines.
Define "explosions"
Stripes, The poster to which you responded did not specify what type of explosions were available to them. If they're nuclear explosions, they'd probably need only 8-10 strategically placed explosions to wipe out all of the current neameservers (with or without social engineering). If they're lucky, they might take out the "shadow root servers" as well. Given the location of some of the root servers, they'd probably cripple alot more than just DNS. They'd effectively take out a good deal of infrastructure as well as the Internet engineers necessary to repair it, not to mention start a worldwide panic.
The Internet would still recover though, much as you described in your post. Anyone can setup a redundant server cluster within a matter of minutes given a set of pre-staged root and first level zone data.
The more interesting problems are due to corrupted data rather than doing denial of service attacks on nameservers. Some bad data in Network Solution's database can make various interesting parts of the Internet suck really bad. When one root server has data corruption, the whole net feels it. Imagine if some NSOL staffer garbled the nameserver data for "Yahoo.COM." or "IN-ADDR.ARPA." to point to 255.255.255.255 instead of the real servers?
For anyone else interested in DNS DoS...
An easier method
One of the easiest way to kill DNS is to try a coordinated DoS attack against all of the nameservers. Each of the world hundreds of thousands of resolvers is configured to use any of 13 root nameservers. Just like a 15-year-old kid did with HTTP requests, one could probably start a distributed DoS attack against DNS. The "heftiest" root nameserver is rumored somewhere in this discussion to be able to handle 6000-8000 hits a second. With 13 published nameservers, one needs only about 100000 hits per second to saturate the current capacity of all of the servers. Let's say that I was a bright hacker (which I'm not) that I could find my way into 1000 machines around the world that each had a T1 connection or better. Can we agree that this is a difficult but not unreasonably impossible thing to do? If one were not smart enough to do it themselves, one could perhaps go to a hacker convention or local user group and bribe a script kiddie seeking infamy and fortune to go forth an find 1000 machines to hack. Another way is to unleash a time-dated virus onto the net that will do your bidding at a specific time. Each machine would gather a list of 100 addresses, perhaps starting with the history file of a user's browser to get a list of second-level domains. It could also look for addresses using a popular portal directory or search engine and interpret results to get domain names. With 100 domain names, it would query 100 names per second (less than one megabit) from each of the few registered root nmeservers. While the traffic isn't overwhelming, it will overload the root servers fo rthe number of transactions per second, and nothing short of hunting and killing half of the query servers would reduce the effectiveness of the attack. To make the attack harder to stop, one could double or quadruple the number of query servers or use methods of masquerading your attack (I won't go into detail here) to keep network administrators from being able to shut down query servers. Another way to scale the attack is to use they heavier TCP protocol for most of the queries instead of the lightweight UDP.
fin.
The technology needed to exponentially increase the ability of the root servers to perform is not out of reach. With the proper motivation (a DoS like I described), one million dollars of capital (compare $1m to the current valuation of NSOL), and perhaps 30 man-weeks of time, one can make a farm of servers able to handle two orders of magnitude more requests than the current set of servers.
The IBM server announcement by Network Solutions disappoints me. It's sad.
Any of the following are good candidates that I know about for scalably solving root DNS infrastructure problems...
- UltraDNS - DNS service provider with an interesting spin on distributed scalability
- Nominum - the knowledge and knowhow to make fast scalable DNS servers and software
- Akamai/Sandpiper - a distributed operations infrastructure onto which one can install root clusters.
Hint: If one can make an application layer proxy host that takes inbound DNS requests and routes them based on a hash table of domain names to a set of back end nameservers (with only a fraction of domains loaded on each), one could have the start of a scalable solution. One can make a fast cheap BSD box to do this up to 5000 ops per second or better. I wonder if the skunk works at Novell can do ths faster. One can use some router technology (OSPF, trunking, or L4 switching) to spray UDP requests to a number of these appliction load balancer / DNS proxy servers.One can also implement interesting filters on such a proxy server to reduce the effect of stupider resolvers or lame DoS attacks.
--
Eric Ziegast
PS: Slashdot probably isn't the best forum for this, but if you know a better forum, feel free to point them toward this post.
I agree that simple solutions may be better. I give an example of a site that uses two Oracle servers that act independently (seperate systems, seperate disk systems) from each other. One server is primary. The other is a backup. The clients are setup to read off the primary and writes/modifies are done to both. When doing backups, the backup server is used without affecting load on the primary server. If the primary server goes down, the clients start reading from the secondary. All you need to make this work is either better client access methods that you program, or you need a client to access a proxy server/service that forwards requests to the appropriate servers. While they used Oracle, this technique isn't hard to implement for other systems. The site is aware of technologies like Oracle Parallel Server and Veritas Cluster Server, but they like their method the best. Note for jrbrtsn: Disks are one of the first things I'd worry about with a database. If you switch disks from one system to another after a failure, there's no guarantee that the data on them won't have problems.