You are, of course, correct about the statements on Bugtraq and CERT - but I was mostly commenting on the AMOUNT of those posts. I also agree with your statement that there must be more than 50 servers out there that are compromised if the attack is a Trinoo variant. The attacks have been reported as coming from real world addresses, no spoofing. This means that it is very likely that these attackers have clients on 50+ servers - and would then have to switch at some point (well, if they really wanted to avoid detection).
Most of my earlier comments I stand by (specifically about vendors and bugs), but there have been updates with NANOG that have been very interesting, and may point to an actual concerted effort starting with Yahoo (sources were verified, some other ISPs admitted to seeing some strange events at the same time period). However, almost all of it is speculation still.
The comments I saw posted dealt with the estimated capabilities of the people involved, and how one might prevent his/her own network from participating in an attack. There is still no cure for proactively preventing a DDoS if you are the victim, AFAIK.
I guess my earlier post in last forum was ignored...here we go:
First off, you have to consider that most servers are NOT going to have the capability of participating in this kind of attack.
1. Bandwidth - um...50 servers, over t-1 or less links? Nope. They HAVE to be located at a Tier 1 provider (running on the Tier 1 provider's LAN, or on colo sites that are generally capped at 10 - 100 megs). That Tier 1 provider HAS to have private peering established over large pipes - this kind of attack would have melted down PAIX.
2. The colo customers would have to be completely blind to the fact that their sites are running up bandwidth charges (charged per meg/s), but getting NO hits for services offered. Also, their security would have to have been completely compromised - ie, bypassing load-balancing proxies in advance, compromising firewalls, bypassing access-lists.
3. ALL of the above would have had to have happened in a coordinated fashion, such that traffic would have to be sent to a DoS client on the servers in question, enable the attack, which said attack would bypass then aforementioned barriers and smack down Yahoo! for more than 1Gig of damage.
Now, how many machines do you have to compromise AND install clients on AND run without being caught, taking up sizable chunks of bandwidth which generally WILL be noticed, and still make the attack possible to occur without making yourself a huge effing target?
Possible, but not very credible - though my hat is off to anyone who could compromise much more than 50 sites and hide the massive amount of work that would have to be done to set this up and make this work. Of course, I don't think that it is likely, since we would have seen multiple reports at CERT and Bugtraq from pissed off sysadmins about some boosheet DoS client hidden on their systems.
Consider the alternatives instead. Consider that some of these outages -especially the eBay outage- were not caused by DoS attacks, but by faulty equipment/software from proprietary vendors - a certain network equipment manufacturer comes to mind on that one. Consider that none of these businesses have to suck up the cash damage if these were "unforseen" occurrences.
1. The Yahoo "DoS" attack may not have been the kind of attack they admitted to. There is always the possibility that equipment upstream was b0rked, causing packets to be sent promiscuously all over the network. I've seen it happen before, just not to Yahoo.
2. Consider that the eBay problem MAY have been a DoS attack, but not the kind you think. I know of at least one showstopper bug that has come up with no less than TWO different major router vendors that could cause the crash they had.
3. I've been able to reproduce similar problems in a lab environment with one vendor's equipment that I was demo'ing. Many of these "DoS attacks" can usually be chalked up to a configuration that the vendor never bothered to test or consider.
I am not calling ANY of the companies mentioned liars, or defaming their stories. I am just pointing out that they may be mistaken, or that their public relations people may be using "evil hackers" to point people away from problems that may have been alleviated but still exist. Please consider that these events could have been caused more by ignorance and greed than by a heretofor unknown elite cadre of super 'net ninjas.
Hmmmm...I won't say how or why (don't bother emailing me to find out, either), but it is possible to take out a high capacity colo site by hitting certain types of network equipment with the right kinds of packets. The vendors who make this equipment does not have a fix for this yet, AFAIK. I found this out from lab testing some eval equipment.
This may or may not relate to equipment used elsewhere, but I don't know, since I don't work for GC or Yahoo.
With that said, the event reported at Global Center seems credible. These kinds of bugs come up ALL of the time, depending upon the release of code by vendor to fix the last set of bugs on a platform (this is not a vendor specific problem - all the vendors are guilty of this). Consider the syslog bug in the 11.3 - 12.0 code of a certain large network equipment vendor. This is one example of such a showstopper that could take down a network (this bug is public knowledge, anyone with a clue has patched it). There have been other similar bugs with all network equipment vendors, so I wouldn't be suprised to see a new one.
There are still serious problems with the proprietary behavior of network equipment vendors, which is the likeliest culprit here. They all have their own implementations of RFCs that shoulda been followed to the letter. And on. And on. All in the name of market share and the mighty buck - and to the annoyance of many networkers.
That is, if it wasn't some kinda 1Gig+ DoS (also probable).
Um, after having read most of these responses...has anyone ever considered contacting their Congressional Reps and setting up a Slashdot style board for them? Or maybe how's about a PGP/GPG key server? I know there would be a lot of details to work out, but if tech savvy people can make an inroad where their.01% of the population's voices could be heard, why not?
I would probably put it this way- the files are written in another language, which the feds don't have translators for - lets call the language Ancient Panvrovian. They could figure out Ancient Panrovian on their own, but they either don't have the will or the means.
Kevin can provide them with the ability to translate the files into English, but that may incriminate him later. He himself can translate ancient Panrovian, given he knows the key phrase to do so under the right translation matrix.
Or, rather, he put a magic lock on the filing cabinet, and they won't give back some of his spellbooks and tomes until he gives them the magic word and the spell. Of course, the documents contain things that could be speech, or could be considered warlock's spells, or both. Unfortunately, I haven't heard of any caselaw that clearly determines whether something you have written could also be considered a tool. Doesn't mean it doesn't exist, but I think it is a new thingy. However, I am not a priest of the constitution; merely a lowly packet-herder teleporting some minor lord's sheep to market.
Filing pre-emptively in the other federal juridictions to test the law under the DMCA and to test the Cert Authority's parters' rights regarding claims on reverse engineering would have put the current defendants in a much more positive light, and probably would have impressed the court. Even if the courts declined to hear the test cases, the prior filings would have already been present, likely affecting much of the nature of any future case the MPAA would submit.
It would have also left the MPAA spin machine totally off balance. Rather than making themselves out to be defender's of content rights, they probably would have ended up looking fairly foolish to the average layperson. It's kinda difficult to characterize someone as a pirate when they have already taken the initiative to proactively prove the constituitionality of their rights (rather than merely defending them).
This quote from the Yahoo article made me think a bit. I mean, let's look at the UN's progress on other important human rights issues and then put these statements into perspective. I wonder how much they actually expect to accomplish:
Dreifuss said a seminar of international experts was scheduled to take place in Geneva on February 16-18 as part of preparations for the first United Nations World Conference on Racism, Racial Discrimination and Xenophobia in 2001.
She said a key objective of this seminar would be to prepare recommendations on international actions to counter racism on the Internet.
Also, the UN conference website (something Yahoo! did not include) has is a lot of stuff they plan addressing in three days, and anything relating to the internet seems to play a fairly minor role so far. Then again, they did say "Millenium" several times - that always seems to precede something inane relating to the Internet recently.
What bothers me about the DeCSS mess isn't that the geeks are mobilizing...it's that the geeks aren't learning. Let me clarify that statement...
The geeks have been cracked down on repeatedly the last 20 years with example cases. In just about every case, the govt and plaintiff were able to make their point (you are bad, you must be punished, we've taken your toys and your freedom; see how we punish). Whether the geek was exhonerated in the end didn't make nearly as much impact on the non-geeks as the crackdown did.
So the geeks now are better at mobilizing when the crackdown comes. So what! The point has already been made. It is already sticking in the backs of the non-geeks heads- geeks now have a two front battle of proving they aren't pirates to the court, and fixing their already dubious reputations with their own communities.
What the geeks need to start doing is heading off the problems when they see the problem at the outset - think of it as a good security policy for your community network (rather than your LAN/WAN). You see a nibble, you do a check, and if you find a hole, you plug it in as many places as possible.
The geeks, after the first filing, should have pre-emptively filed in EVERY federal jurisdiction. This would have headed off most of the mess going on now, and forced the MPAA off balance (disrupting their entired choreagraphed passion play to the media). What really gets me is this- nobody suspected that the first LAME attempt at a suit filed in California was just to test the waters, and to come up with more defendants for a case that HAD to be in the MPAA's works prior to the Copy Authoity's original filing.
So while everybody here is congratulating themselves on what the suits might think about how righteous we geeks may be, what is anyone doing to ensure we don't get a repeat? Volunteering to rebuild your local Congressperson's network/website to get their ear on tech issues? Finding new resources to work with on future problems that WILL come up? Even following current bills and measures being deliberated at the local level?
We may be laughing at how stupid the lawyers may look because they are technologically inept in regards to these proceedings, but how do you think the geeks look for leaving themselves open to this kind of stupid-ass suit in the first place? Do something, but don't just sit there.
At my previous company, a large-scale NSP, Unicenter and other CA products were brought in by upper management. The product had multiple bugs, and event the agents could poll properly on CPU load without maxing out the cpu. All of their interfaces, which were supposed to be configurable and intuitive were anything but that - no support for importing data, and obscure and deeply nested access via the GUI (checking a simple outage involved going through no less than 5 clickthrus, plus entering plenty of text). Demand on the management stations was VERY high, and the software did not share well with other processes - though orignally designed for NT, their servers ran Unix (as an option) but their management stations wanted win95. I know that this model may have been changed somewhat with the new features, but think of where they were at just 2 years ago...
Their support consisted of nothing at first, but then was scaled to 4 programmers living in our eng area. This was not because they wanted to do this- they had wanted to charge us ungodly amounts of cash for this privelige. The only reason we got them at all was because it would have violated a prior arrangement they made with us. The programmers, however, were uncooperative and generally did not want to work on anything but a very narrow set of parameters on the server side only. Getting anything done with them was about impossible, but we finally got them to compile a Unix client, which we could eventually compile under linux- neither were stable.
Bottom line is this- they did not have product or plan for product under Linux 2 years ago. Even under other platforms, they did not meet the "enterprise" standard of support (everything works, is fully interoperable, 99.9% of the time w/ comprable uptime). Considering how bad their previous "flagship enterprise" products were before, I can't begin to imagine all the hassles of dealing with their product on *nix, plus the added hassles of having to put up with their exhorbitant and lousy support (and VERY obtuse documentation). Maybe if you have only one person assigned for support it *might* work out, but it didn't work well in a multi-staff multi-hat environment.
They are excellent business people. They can sell to management like nobody else- strategic partnerships to increase their stocks has apparently been what they are best at. And they ALWAYS sell with binding, multi-year contracts that tie your hands while leaving them free to do as little as they wish...so I have to wonder if, beyond marketing hype, this is something I would really want associated with a quality product like Redhat, which is THE LINUX in the minds of most business-people and consumers.
Slashdot Mirror
You are, of course, correct about the statements on Bugtraq and CERT - but I was mostly commenting on the AMOUNT of those posts. I also agree with your statement that there must be more than 50 servers out there that are compromised if the attack is a Trinoo variant. The attacks have been reported as coming from real world addresses, no spoofing. This means that it is very likely that these attackers have clients on 50+ servers - and would then have to switch at some point (well, if they really wanted to avoid detection).
Most of my earlier comments I stand by (specifically about vendors and bugs), but there have been updates with NANOG that have been very interesting, and may point to an actual concerted effort starting with Yahoo (sources were verified, some other ISPs admitted to seeing some strange events at the same time period). However, almost all of it is speculation still.
The comments I saw posted dealt with the estimated capabilities of the people involved, and how one might prevent his/her own network from participating in an attack. There is still no cure for proactively preventing a DDoS if you are the victim, AFAIK.
I guess my earlier post in last forum was ignored...here we go:
First off, you have to consider that most servers are NOT going to have the capability of participating in this kind of attack.
1. Bandwidth - um...50 servers, over t-1 or less links? Nope. They HAVE to be located at a Tier 1 provider (running on the Tier 1 provider's LAN, or on colo sites that are generally capped at 10 - 100 megs). That Tier 1 provider HAS to have private peering established over large pipes - this kind of attack would have melted down PAIX.
2. The colo customers would have to be completely blind to the fact that their sites are running up bandwidth charges (charged per meg/s), but getting NO hits for services offered. Also, their security would have to have been completely compromised - ie, bypassing load-balancing proxies in advance, compromising firewalls, bypassing access-lists.
3. ALL of the above would have had to have happened in a coordinated fashion, such that traffic would have to be sent to a DoS client on the servers in question, enable the attack, which said attack would bypass then aforementioned barriers and smack down Yahoo! for more than 1Gig of damage.
Now, how many machines do you have to compromise AND install clients on AND run without being caught, taking up sizable chunks of bandwidth which generally WILL be noticed, and still make the attack possible to occur without making yourself a huge effing target?
Possible, but not very credible - though my hat is off to anyone who could compromise much more than 50 sites and hide the massive amount of work that would have to be done to set this up and make this work. Of course, I don't think that it is likely, since we would have seen multiple reports at CERT and Bugtraq from pissed off sysadmins about some boosheet DoS client hidden on their systems.
Consider the alternatives instead. Consider that some of these outages -especially the eBay outage- were not caused by DoS attacks, but by faulty equipment/software from proprietary vendors - a certain network equipment manufacturer comes to mind on that one. Consider that none of these businesses have to suck up the cash damage if these were "unforseen" occurrences.
1. The Yahoo "DoS" attack may not have been the kind of attack they admitted to. There is always the possibility that equipment upstream was b0rked, causing packets to be sent promiscuously all over the network. I've seen it happen before, just not to Yahoo.
2. Consider that the eBay problem MAY have been a DoS attack, but not the kind you think. I know of at least one showstopper bug that has come up with no less than TWO different major router vendors that could cause the crash they had.
3. I've been able to reproduce similar problems in a lab environment with one vendor's equipment that I was demo'ing. Many of these "DoS attacks" can usually be chalked up to a configuration that the vendor never bothered to test or consider.
I am not calling ANY of the companies mentioned liars, or defaming their stories. I am just pointing out that they may be mistaken, or that their public relations people may be using "evil hackers" to point people away from problems that may have been alleviated but still exist. Please consider that these events could have been caused more by ignorance and greed than by a heretofor unknown elite cadre of super 'net ninjas.
Hmmmm...I won't say how or why (don't bother emailing me to find out, either), but it is possible to take out a high capacity colo site by hitting certain types of network equipment with the right kinds of packets. The vendors who make this equipment does not have a fix for this yet, AFAIK. I found this out from lab testing some eval equipment.
This may or may not relate to equipment used elsewhere, but I don't know, since I don't work for GC or Yahoo.
With that said, the event reported at Global Center seems credible. These kinds of bugs come up ALL of the time, depending upon the release of code by vendor to fix the last set of bugs on a platform (this is not a vendor specific problem - all the vendors are guilty of this). Consider the syslog bug in the 11.3 - 12.0 code of a certain large network equipment vendor. This is one example of such a showstopper that could take down a network (this bug is public knowledge, anyone with a clue has patched it). There have been other similar bugs with all network equipment vendors, so I wouldn't be suprised to see a new one.
There are still serious problems with the proprietary behavior of network equipment vendors, which is the likeliest culprit here. They all have their own implementations of RFCs that shoulda been followed to the letter. And on. And on. All in the name of market share and the mighty buck - and to the annoyance of many networkers.
That is, if it wasn't some kinda 1Gig+ DoS (also probable).
Um, after having read most of these responses...has anyone ever considered contacting their Congressional Reps and setting up a Slashdot style board for them? Or maybe how's about a PGP/GPG key server? I know there would be a lot of details to work out, but if tech savvy people can make an inroad where their .01% of the population's voices could be heard, why not?
I would probably put it this way- the files are written in another language, which the feds don't have translators for - lets call the language Ancient Panvrovian. They could figure out Ancient Panrovian on their own, but they either don't have the will or the means.
Kevin can provide them with the ability to translate the files into English, but that may incriminate him later. He himself can translate ancient Panrovian, given he knows the key phrase to do so under the right translation matrix.
Or, rather, he put a magic lock on the filing cabinet, and they won't give back some of his spellbooks and tomes until he gives them the magic word and the spell. Of course, the documents contain things that could be speech, or could be considered warlock's spells, or both.
Unfortunately, I haven't heard of any caselaw that clearly determines whether something you have written could also be considered a tool. Doesn't mean it doesn't exist, but I think it is a new thingy. However, I am not a priest of the constitution; merely a lowly packet-herder teleporting some minor lord's sheep to market.
Filing pre-emptively in the other federal juridictions to test the law under the DMCA and to test the Cert Authority's parters' rights regarding claims on reverse engineering would have put the current defendants in a much more positive light, and probably would have impressed the court. Even if the courts declined to hear the test cases, the prior filings would have already been present, likely affecting much of the nature of any future case the MPAA would submit.
It would have also left the MPAA spin machine totally off balance. Rather than making themselves out to be defender's of content rights, they probably would have ended up looking fairly foolish to the average layperson. It's kinda difficult to characterize someone as a pirate when they have already taken the initiative to proactively prove the constituitionality of their rights (rather than merely defending them).
Dreifuss said a seminar of international experts was scheduled to take place in Geneva on February 16-18 as part of preparations for the first United Nations World Conference on Racism, Racial Discrimination and Xenophobia in 2001.
She said a key objective of this seminar would be to prepare recommendations on international actions to counter racism on the Internet.
Also, the UN conference website (something Yahoo! did not include) has is a lot of stuff they plan addressing in three days, and anything relating to the internet seems to play a fairly minor role so far. Then again, they did say "Millenium" several times - that always seems to precede something inane relating to the Internet recently.
What bothers me about the DeCSS mess isn't that the geeks are mobilizing...it's that the geeks aren't learning. Let me clarify that statement...
The geeks have been cracked down on repeatedly the last 20 years with example cases. In just about every case, the govt and plaintiff were able to make their point (you are bad, you must be punished, we've taken your toys and your freedom; see how we punish). Whether the geek was exhonerated in the end didn't make nearly as much impact on the non-geeks as the crackdown did.
So the geeks now are better at mobilizing when the crackdown comes. So what! The point has already been made. It is already sticking in the backs of the non-geeks heads- geeks now have a two front battle of proving they aren't pirates to the court, and fixing their already dubious reputations with their own communities.
What the geeks need to start doing is heading off the problems when they see the problem at the outset - think of it as a good security policy for your community network (rather than your LAN/WAN). You see a nibble, you do a check, and if you find a hole, you plug it in as many places as possible.
The geeks, after the first filing, should have pre-emptively filed in EVERY federal jurisdiction. This would have headed off most of the mess going on now, and forced the MPAA off balance (disrupting their entired choreagraphed passion play to the media). What really gets me is this- nobody suspected that the first LAME attempt at a suit filed in California was just to test the waters, and to come up with more defendants for a case that HAD to be in the MPAA's works prior to the Copy Authoity's original filing.
So while everybody here is congratulating themselves on what the suits might think about how righteous we geeks may be, what is anyone doing to ensure we don't get a repeat? Volunteering to rebuild your local Congressperson's network/website to get their ear on tech issues? Finding new resources to work with on future problems that WILL come up? Even following current bills and measures being deliberated at the local level?
We may be laughing at how stupid the lawyers may look because they are technologically inept in regards to these proceedings, but how do you think the geeks look for leaving themselves open to this kind of stupid-ass suit in the first place? Do something, but don't just sit there.
At my previous company, a large-scale NSP, Unicenter and other CA products were brought in by upper management. The product had multiple bugs, and event the agents could poll properly on CPU load without maxing out the cpu. All of their interfaces, which were supposed to be configurable and intuitive were anything but that - no support for importing data, and obscure and deeply nested access via the GUI (checking a simple outage involved going through no less than 5 clickthrus, plus entering plenty of text). Demand on the management stations was VERY high, and the software did not share well with other processes - though orignally designed for NT, their servers ran Unix (as an option) but their management stations wanted win95. I know that this model may have been changed somewhat with the new features, but think of where they were at just 2 years ago...
Their support consisted of nothing at first, but then was scaled to 4 programmers living in our eng area. This was not because they wanted to do this- they had wanted to charge us ungodly amounts of cash for this privelige. The only reason we got them at all was because it would have violated a prior arrangement they made with us. The programmers, however, were uncooperative and generally did not want to work on anything but a very narrow set of parameters on the server side only. Getting anything done with them was about impossible, but we finally got them to compile a Unix client, which we could eventually compile under linux- neither were stable.
Bottom line is this- they did not have product or plan for product under Linux 2 years ago. Even under other platforms, they did not meet the "enterprise" standard of support (everything works, is fully interoperable, 99.9% of the time w/ comprable uptime). Considering how bad their previous "flagship enterprise" products were before, I can't begin to imagine all the hassles of dealing with their product on *nix, plus the added hassles of having to put up with their exhorbitant and lousy support (and VERY obtuse documentation). Maybe if you have only one person assigned for support it *might* work out, but it didn't work well in a multi-staff multi-hat environment.
They are excellent business people. They can sell to management like nobody else- strategic partnerships to increase their stocks has apparently been what they are best at. And they ALWAYS sell with binding, multi-year contracts that tie your hands while leaving them free to do as little as they wish...so I have to wonder if, beyond marketing hype, this is something I would really want associated with a quality product like Redhat, which is THE LINUX in the minds of most business-people and consumers.