My next best theory is that someone at LimeLight Networks(AS3549, a GLBX reseller) is sending poison BGP announcements, but I don't see any in looking glasses.
That kind of technically advanced activity, especially with the potential for huge economic losses, should trigger an FBI investigation. Of course, the FBI isn't going to admit anything or post updates on/. until they hand up indictments to the court and make some arrests.
now my head hurts.
limelight is not as3549. they are as22882.
limelight are not a global crossing reseller. they are a global crossing transit provider (with a pretty big network of their own)
limelight did not inject any sneaky advertisements about anyone related to this. first of all, they wouldn't. they're ethical, stand-up guys (i know some of their senior network engineers). secondly, i would have seen it. renesys, the company i work for, maintains a massive database of all routing changes affecting the global internet.
sigh. this is just more blathering nonsense, as with most of the rest of this thread.
My suspicion (since I don't have a looking glass with a historical search), is that someone with access to the main BGP reflectors inside of either UUNET or GlobalXing managed to make an announcement that they had a local router with a route to AS1680, and then that router just blackholed any traffic to those netblocks.
no.
that makes no sense. first of all, i see no evidence of AS3549 (global crossing) as a provider to netvision. second of all, even if they were and uunet and gblx both set a null route, traffic would still have come in via telia and btn.
the fact that the slashdot crowd seems not to know who btn (as3491) or telia (as1299) are, doesn't really matter. telia are one of the 10 or 15 largest networks in the world, depending on how you count. btn are top-25.
i'm going to go write this up with some details so that the network-clue-impaired can understand it. i doubt i'll succeed but i'll put results up over at the renesys blog later.
Bluesecurity (BS) are either confused or misleading people.
There is no way that a single "backbone" provider could have installed a null route to block all traffic to their network. Bluesecurity is served by a Haifa-based provider called Netvision (Autonomous System number 1680). Netvision buys internet transit from four providers:
--UUnet/701 (uunet north america) --UUnet/702 (uunet europe/middle east) --btn/3491 (beyond the network) --telia/1299 (telia sonera international backbone).
what the heck is BS claiming? that *all* of them installed a null route at once. do they even know what a null route is.
i'm getting annoyed enough at this nonsense to think about blogging about it in more detail over at www.renesys.com/blogs . perhaps later today.
This is, annoyingly, the *exact* same topic as the Verizon post earlier today and it is the exact same, terribly old news that people (including me have been discussing since back in November, when SBC chair Whitacre made comments to business week about forcing google to pay to access their network.
Don't get me wrong, this is an interesting topic. I'm in favor of continuing to discuss it. But the Businessweek, the nation and several other sources get it totally wrong by ignoring the technical realities (as I tried to explain earlier today).
There is already a two-tier network, people. There are legitimate questions about how much further (and in exactly what ways) it should be extended, but people who opine about these matters unhindered by data would do better to read and think a bit first.
Re:Verizon's recent purchase makes this subject mo
on
Is Verizon a Network Hog?
·
· Score: 3, Interesting
Ummm, no. In short, no. Also, no.:-)
Seriously, capacity is not some monolithic thing that you "have enough of" or "have too much of". Capacity is from a place to a place across a set of resources. VZ can have plenty of capacity from NY to VA but not enough peering to AS3356 (level3). Or They might have plenty of cross-country capacity until a train derails in Colorado causing a 3-4 day outage of the middle path and congesting some other paths. It all depends and the devel is in the detail.
Even using generous estimates of multicast efficiencies, video over packet (or IPTV) is going to consume a *lot* of resources. ~20-25Mb/s per channel. Right now, virtually no one has "enough capacity" for that.
This is the exact same issue (from the exact same source) as the interview with SBC Chairman Whitacre last fall. I covered the dispute in my blog last month. It doesn't seem to die.
The issue is clouded by fuzzy-headed thinking. Cable companies already do this. They "reserve bandwidth" (i.e. channels, frequencies, capacity) for their video content and only make a small amount of space available for Internet. The idea that ILECs would do the same when they roll out IPTV or other video-over-packet strategies, is so shockingly unremarkable as to not be news.
The only interesting issue here is whether VZ or SBC or others will restrict or degrade their existing Internet service in the process. I seriously doubt they will--the market would punish them for that. But if they were to, that would be interesting.
The ability to observe the outage (sharply) through routing activity is definitely the part that we thought was coolest.
People are saying two different things here: 1) well, duh, if power is out lots of people can't connect to the web; 2) if the core of the internet routes around that who cares. These are both interesting points. Here are some thoughts:
1) We agree. That's what I though. But read the keynote press releases. Or just google on 'blackout Internet' and you'll find glowing stories about how 'the Internet' didn't even blip under the blackout. We prove pretty conclusively that this is incorrect.
2) The core of the Internet did, indeed, route around the outage. This is good. What is less good is that thousands of networks within the outage area lost connectivity, either due to lost power themselves, or upstreams that lost power (or telcos who lost battery backup on csu/dsu units, or whatever). These are *not* DSL customers (or that grade, anyway). All of these are BGP-speaking networks with their own Autonomous Systems and their own prefixes.
The fact that so many networks went down is significant, given that many organizations are coming to rely on the Internet as a critical communications infrastructure.
This is certainly a topical comment, but it misses the point a little (I think).
A large number of organizations that were multi-homed, using BGP to announce routes out multiple upstream providers lost connectivity. This speaks to the situation that people who have spent a bunch of money on network infrastructure may not have spent enough on power (or may not have carefully evaluated their upstream providers).
One of the organizations located in the study had nine (9!) upstream providers and still went out. This is not a case of people on the far end of a DSL link; this is the case of people not being able to put together reliable network connectivity, even in the face of multi-homing.
Re:For the "Deregulation was the problem" nutjobs
on
Network Blackout
·
· Score: 2, Interesting
Security focus is saying that a previous outage wasn't deregulation, wasn't transmission, it was the slammer worm hitting Ohio's Davis-Besse nuclear power plant and disabling a safety monitoring program.
Interesting story. CNN and a bunch of people on NANOG (www.nanog.org) were speculating that this outage was caused by the msblaster worm. This story backs up at least the feasibility of that.
dan bernstein's position on this
on
DNSSEC: Good Enough?
·
· Score: 5, Informative
The summary: It's unfinished, the BIND company has poor implementations (like most everything else it implements), and won't provide a real increase in security. Interesting stuff.
The geolocation of networks is never a precise thing. Networks can be registered in the city of their corporate offices but deployed anywhere in the world (it's how the Internet works--cool, huh?).
There are some interesting, precise tricks that you can play by sending various kinds of packets (usually UDP) and using detailed latency information about each hop of those packets, along with a provider network map, to get closer to the physical location of a particular IP address. We didn't do that for these maps.
Re:Slashdotting animation
on
Network Blackout
·
· Score: 2, Informative
And yet, the webserver's fine (famous last words).
Did y'all notice *how* *small* that animation is? Someone else here put it together. I didn't know that animated gifs could compress that much.
Crap. The notes page points out several cool new packages (including acpid for power management and the dvd+rw tools for dvd mastering. But several important packages are falling by the wayside.
Among them:
pine (I've just finished, after 8 years or so) getting used to this thing after the migration from elm. Does this mean I finally have to use mutt?
tripwire. I know lots of people didn't actually use this, but it's really important and the fact that redhat integrated it really raised its visibility.
postgresql72. what's going on here? are they running a more recent version and simply removing the old one?
While I don't disagree, I would love to understand this better.
Unless I'm missing the boat here, nobody loves Exchange. What people like is Outlook for mail and scheduling, which I find incredibly primitive and unproductive, but there it is--people do seem to like it.
The problems with Outlook on Exchange are obvious--Outlook is better described as a sophisticated virus-distribution platform than it is an MUA or a scheduling client. But if people love Outlook, they can use it with a variety of other scheduling platforms, including Lotus Notes Domino servers (with an outlook plug-in thingy) or even the Byrani stuff (see www.byrani.com).
Am I missing something or is it really feasible to replace the Exchange server side with minimal client reconfiguration and still get what people want?
This misses an important point: circuit switched networks are inefficient; packet switched networks are efficient.
I sympathize with the notion that people are expecting something for nothing and that long distance data/voice will not continue to be move towards free forever. But I do think that it's important to understand just how much difference it makes to carry voice traffic as data packets on packet-switched networks.
Currently 20+% of all voice traffic on several US->other country routes (including US->israel, US->mexico and US->argentina) are carried not just as VoIP, but as packets on the public internet. Carriers do this to save money.
When a call on a circuit switched network is in progress, 100% of the 64Kb/s allocated to that connection is wasted, even if the two speakers are silent for the entire duration fo the call. When a call on a packet-switched network is in progress, only 6-12Kb/s is in use and even that can be reduced when the speakers talk less (or are more silent).
We use an older version of UW IMAP ad UW Pine both patched to use Maildir support (because they are too short-sighted to integrate such support themselves).
After the roll-over both programs started mis-sorting newly arrived messages to the top of the folder, rather than the bottom (but newly arrived messages are still sorted below older, within each category of 'before' and 'after' the 1 billion second point). Also getting 'mailbox changed unexpectedly, reloading' messages constantly.
Maybe I'm missing the point, or maybe I'm not clearly stating mine. DB2 does (and has done since the early 1980s) statistics-based optimization of queries *automatically*. Every time I talk to Oracle developers and DBAs they tell me things that sound like sheer insanity to me: what order you list clauses in your 'WHERE' dramatically change performance, you should do an EXPLAIN PLAN and then write your query accordingly.
I agree that the best optimizer is the human mind, but I have yet to meet database programmers who can bring themselves to regularly remember to care how big the table they are accessing is, or what fields are indexed or which part of their cartesian product is bigger. Programmers don't care. Oracle programmers have to (as far as I can understand). DB2 programmers don't.
This doesn't really suprise me. We've been using db2 in production on Linux at work (www.osogrande.com) for about 2 or 2 1/2 years (since the UDB 5 beta on Linux came out). It is easy to install (aside from some curses incompatibilities on RH7.1, which will get resolved shortly), easy to administer and performs well.
For free databases, I prefer postgres (transactional support, referential integrity, triggers, etc.). For commercial support, I have trouble liking Oracle (terrible query optimization for large queries, no statistics-based optimization like db2, much harder to administer, etc.). DB2 UDB on Linux is a low-cost, high-feature, high-performing dream.
(can't take credit for that one--one of our sys admins at work came up with it when we were discussing Citynet, who are laying fiber in Albuquerque right now).
Slashdot Mirror
That kind of technically advanced activity, especially with the potential for huge economic losses, should trigger an FBI investigation. Of course, the FBI isn't going to admit anything or post updates on /. until they hand up indictments to the court and make some arrests.
now my head hurts.
limelight is not as3549. they are as22882.
limelight are not a global crossing reseller. they are a global crossing transit provider (with a pretty big network of their own)
limelight did not inject any sneaky advertisements about anyone related to this. first of all, they wouldn't. they're ethical, stand-up guys (i know some of their senior network engineers). secondly, i would have seen it. renesys, the company i work for, maintains a massive database of all routing changes affecting the global internet.
sigh. this is just more blathering nonsense, as with most of the rest of this thread.
no.
that makes no sense. first of all, i see no evidence of AS3549 (global crossing) as a provider to netvision. second of all, even if they were and uunet and gblx both set a null route, traffic would still have come in via telia and btn.
the fact that the slashdot crowd seems not to know who btn (as3491) or telia (as1299) are, doesn't really matter. telia are one of the 10 or 15 largest networks in the world, depending on how you count. btn are top-25.
i'm going to go write this up with some details so that the network-clue-impaired can understand it. i doubt i'll succeed but i'll put results up over at the renesys blog later.
wow. you really have no idea how the internet works. so sad.
none. this is smoke and mirrors and clearly not what happened. the BS story doesn't make any sense.
Bluesecurity (BS) are either confused or misleading people.
There is no way that a single "backbone" provider could have installed a null route to block all traffic to their network. Bluesecurity is served by a Haifa-based provider called Netvision (Autonomous System number 1680). Netvision buys internet transit from four providers:
--UUnet/701 (uunet north america)
--UUnet/702 (uunet europe/middle east)
--btn/3491 (beyond the network)
--telia/1299 (telia sonera international backbone).
what the heck is BS claiming? that *all* of them installed a null route at once. do they even know what a null route is.
i'm getting annoyed enough at this nonsense to think about blogging about it in more detail over at www.renesys.com/blogs . perhaps later today.
foolishness.
Sigh.
This is, annoyingly, the *exact* same topic as the Verizon post earlier today and it is the exact same, terribly old news that people (including me have been discussing since back in November, when SBC chair Whitacre made comments to business week about forcing google to pay to access their network.
Don't get me wrong, this is an interesting topic. I'm in favor of continuing to discuss it. But the Businessweek, the nation and several other sources get it totally wrong by ignoring the technical realities (as I tried to explain earlier today).
There is already a two-tier network, people. There are legitimate questions about how much further (and in exactly what ways) it should be extended, but people who opine about these matters unhindered by data would do better to read and think a bit first.
Ummm, no. In short, no. Also, no. :-)
Seriously, capacity is not some monolithic thing that you "have enough of" or "have too much of". Capacity is from a place to a place across a set of resources. VZ can have plenty of capacity from NY to VA but not enough peering to AS3356 (level3). Or They might have plenty of cross-country capacity until a train derails in Colorado causing a 3-4 day outage of the middle path and congesting some other paths. It all depends and the devel is in the detail.
Even using generous estimates of multicast efficiencies, video over packet (or IPTV) is going to consume a *lot* of resources. ~20-25Mb/s per channel. Right now, virtually no one has "enough capacity" for that.
This is the exact same issue (from the exact same source) as the interview with SBC Chairman Whitacre last fall. I covered the dispute in my blog last month. It doesn't seem to die.
The issue is clouded by fuzzy-headed thinking. Cable companies already do this. They "reserve bandwidth" (i.e. channels, frequencies, capacity) for their video content and only make a small amount of space available for Internet. The idea that ILECs would do the same when they roll out IPTV or other video-over-packet strategies, is so shockingly unremarkable as to not be news.
The only interesting issue here is whether VZ or SBC or others will restrict or degrade their existing Internet service in the process. I seriously doubt they will--the market would punish them for that. But if they were to, that would be interesting.
or
i d= SD-032
6.0TB (5.25 usable) $12000
http://www.asaservers.com/system_dept.asp?dept_
The ability to observe the outage (sharply) through routing activity is definitely the part that we thought was coolest.
People are saying two different things here: 1) well, duh, if power is out lots of people can't connect to the web; 2) if the core of the internet routes around that who cares. These are both interesting points. Here are some thoughts:
1) We agree. That's what I though. But read the keynote press releases. Or just google on 'blackout Internet' and you'll find glowing stories about how 'the Internet' didn't even blip under the blackout. We prove pretty conclusively that this is incorrect.
2) The core of the Internet did, indeed, route around the outage. This is good. What is less good is that thousands of networks within the outage area lost connectivity, either due to lost power themselves, or upstreams that lost power (or telcos who lost battery backup on csu/dsu units, or whatever). These are *not* DSL customers (or that grade, anyway). All of these are BGP-speaking networks with their own Autonomous Systems and their own prefixes.
The fact that so many networks went down is significant, given that many organizations are coming to rely on the Internet as a critical communications infrastructure.
This is certainly a topical comment, but it misses the point a little (I think).
A large number of organizations that were multi-homed, using BGP to announce routes out multiple upstream providers lost connectivity. This speaks to the situation that people who have spent a bunch of money on network infrastructure may not have spent enough on power (or may not have carefully evaluated their upstream providers).
One of the organizations located in the study had nine (9!) upstream providers and still went out. This is not a case of people on the far end of a DSL link; this is the case of people not being able to put together reliable network connectivity, even in the face of multi-homing.
Interesting story. CNN and a bunch of people on NANOG (www.nanog.org) were speculating that this outage was caused by the msblaster worm. This story backs up at least the feasibility of that.
The summary: It's unfinished, the BIND company has poor implementations (like most everything else it implements), and won't provide a real increase in security. Interesting stuff.
The geolocation of networks is never a precise thing. Networks can be registered in the city of their corporate offices but deployed anywhere in the world (it's how the Internet works--cool, huh?).
There are some interesting, precise tricks that you can play by sending various kinds of packets (usually UDP) and using detailed latency information about each hop of those packets, along with a provider network map, to get closer to the physical location of a particular IP address. We didn't do that for these maps.
And yet, the webserver's fine (famous last words).
Did y'all notice *how* *small* that animation is? Someone else here put it together. I didn't know that animated gifs could compress that much.
Crap. The notes page points out several cool new packages (including acpid for power management and the dvd+rw tools for dvd mastering. But several important packages are falling by the wayside.
Among them:
pine (I've just finished, after 8 years or so) getting used to this thing after the migration from elm. Does this mean I finally have to use mutt?
tripwire. I know lots of people didn't actually use this, but it's really important and the fact that redhat integrated it really raised its visibility.
postgresql72. what's going on here? are they running a more recent version and simply removing the old one?
While I don't disagree, I would love to understand this better.
Unless I'm missing the boat here, nobody loves Exchange. What people like is Outlook for mail and scheduling, which I find incredibly primitive and unproductive, but there it is--people do seem to like it.
The problems with Outlook on Exchange are obvious--Outlook is better described as a sophisticated virus-distribution platform than it is an MUA or a scheduling client. But if people love Outlook, they can use it with a variety of other scheduling platforms, including Lotus Notes Domino servers (with an outlook plug-in thingy) or even the Byrani stuff (see www.byrani.com).
Am I missing something or is it really feasible to replace the Exchange server side with minimal client reconfiguration and still get what people want?
This comment is, in my experience, inaccurate.
I've logged onto Samba-controlled domains and printed to their printers from both NT4 workstation and NT4 server.
Not sure what bug you're referring to, but I'd love to see a reference.
The central point of this, though: that Linux can easily replace windows for many functions, is definitely still valid.
This misses an important point: circuit switched networks are inefficient; packet switched networks are efficient.
I sympathize with the notion that people are expecting something for nothing and that long distance data/voice will not continue to be move towards free forever. But I do think that it's important to understand just how much difference it makes to carry voice traffic as data packets on packet-switched networks.
Currently 20+% of all voice traffic on several US->other country routes (including US->israel, US->mexico and US->argentina) are carried not just as VoIP, but as packets on the public internet. Carriers do this to save money.
When a call on a circuit switched network is in progress, 100% of the 64Kb/s allocated to that connection is wasted, even if the two speakers are silent for the entire duration fo the call. When a call on a packet-switched network is in progress, only 6-12Kb/s is in use and even that can be reduced when the speakers talk less (or are more silent).
this stuff obviously matters.
Excellent! I did something similar:
/^9.*/ ){
#!/usr/bin/perl
#
# pinefix
# Fixing the maildir pine driver bug that occurred when the 1000000000
# second roll-over happened.
use File::Find;
$maildir = shift(@ARGV);
if (!(-d $maildir)){
die "$maildir must be a Maildir";
}
print "Working on $maildir\n";
find(\&pinefix, $maildir);
sub pinefix(){
$fullpath = $File::Find::name;
$dir = $File::Find::dir;
$filename = $_;
if ( $filename =~
$newpath = $dir . "/0" . $filename;
link $fullpath, $newpath;
unlink $fullpath;
print "In pinefix renamed $fullpath to $newpath\n";
} else {
print "$filename does not need renaming\n";
}
}
We use an older version of UW IMAP ad UW Pine both patched to use Maildir support (because they are too short-sighted to integrate such support themselves).
After the roll-over both programs started mis-sorting newly arrived messages to the top of the folder, rather than the bottom (but newly arrived messages are still sorted below older, within each category of 'before' and 'after' the 1 billion second point). Also getting 'mailbox changed unexpectedly, reloading' messages constantly.
Maybe I'm missing the point, or maybe I'm not clearly stating mine. DB2 does (and has done since the early 1980s) statistics-based optimization of queries *automatically*. Every time I talk to Oracle developers and DBAs they tell me things that sound like sheer insanity to me: what order you list clauses in your 'WHERE' dramatically change performance, you should do an EXPLAIN PLAN and then write your query accordingly.
I agree that the best optimizer is the human mind, but I have yet to meet database programmers who can bring themselves to regularly remember to care how big the table they are accessing is, or what fields are indexed or which part of their cartesian product is bigger. Programmers don't care. Oracle programmers have to (as far as I can understand). DB2 programmers don't.
This doesn't really suprise me. We've been using db2 in production on Linux at work (www.osogrande.com) for about 2 or 2 1/2 years (since the UDB 5 beta on Linux came out). It is easy to install (aside from some curses incompatibilities on RH7.1, which will get resolved shortly), easy to administer and performs well.
For free databases, I prefer postgres (transactional support, referential integrity, triggers, etc.). For commercial support, I have trouble liking Oracle (terrible query optimization for large queries, no statistics-based optimization like db2, much harder to administer, etc.). DB2 UDB on Linux is a low-cost, high-feature, high-performing dream.
The fiber is all put on the top of the sewer (the steel rings snap into place and have brackets for steel conduit above and the fiber goes in that).
This gives IP a whole new meaning! :-)
(can't take credit for that one--one of our sys admins at work came up with it when we were discussing Citynet, who are laying fiber in Albuquerque right now).