Researchers Probe Dark and Murky Net

umm qasr writes: "Security Focus has an interesting article on blocks of internet space that are hidden from most users, it is based on a survey by Arbor Networks. The most common 'invisible sites' being .mil, which seems is unintentional. The survey suggests others, which seem more sinister...using unused netblock addresses to send spam. It's a bit short on the details but interesting none the less."

Comments

Interesting

[rmadmin]

Kinda interesting what all is out there. Now, add on top of that all of those evil spam sending servers that are behind firewalls on 'reserved' ip blacks. Its kinda crazy thinking about all the stuff thats out there that no one will ever see. I always figured anything sensative for military use would be stored on a proprietary government network. But now that I think of it. If they put it on some obscure ip block and give it no hostname, who will ever find it? Wonder if they found my secret porn stash when they were probing all them blocks. =)

Spammers as Romulans?

[odaiwai]

"First Officer! Demurk!"
"Yes Captain Spamford."
"Prepare spam... Bulk Email!"
"Bulk Emailing sir! ... Finished!"
"Excellent, return to Murk space."

.
.
.
"Sir! it's an anti spammer!"
"What's he want?"
"He wants to shove our testicles up our noses and beat us to death with toner cartridges. He said something about sucking your eyes out with a penis enlarger as well."
"again?"

dave

Re:Spammers as Romulans?

[Anonymous Coward]

itsnt it great that +5 funny and trol can be the same thing :)

Re:Spammers as Romulans?

[Anonymous Coward]

It's more amusing that one of the +1's was an "Informative" Rating.

Maybe that moderator had never thought about sucking someone's eyes out with a penis enlarger...

and the Internet came from the military net...

[VDM]

It's strange to discover that a network born from military efforts is actually badly managed by its originators. This adds to the fact that the initial dream of having a network connected through multiple routes in case of attacks has been never fulfilled, or at least it is no more possible. Apart from those addresses, after the recent WTC attacks the Italian research network has been cut out from US networks because the backbone connection was passing under the towers. Some day was needed to find a backup connection, and we are still on backup at a lower bandwidth than usual.

Re:and the Internet came from the military net...

[clacke]

It's not the military's fault if the ISPs are too incompetent to be able to route to milnet adresses. It's just that nobody cares if their customers can reach milnet, just as long as they can reach goatse.cx...

Re:and the Internet came from the military net...

[oni]

In many cases, it is the military;s fault - not the ISP. Take for example, www.gordon.army.mil. You can't hit that site because the administrators are so paranoid they have closed it off to outside access.

Dark address space?

[gabriel_aristos]

So.. Does this mean that if they find enough "dark address space", the Internet will eventually stop growing, and someday, billions of years from now collapse back in upon itself to start the cycle all over again?

-j

Re:Dark address space?

[Tsar cr0bar]

Mod this freaking thing UP!

Re:Dark address space?

[Rasta Prefect]

No, no, you're thinking of the "Porn Critical Mass" the point at which theres so much pornography on the internet that every search returns only ads for things such as "The girl next door with a donkey!" and "Pamela Anderson does an entire boy scout troup!". This in combination with chain letters and spam will cause our minds to collapse, forcing civilization to start over with some more deserving species, like lemurs, who will have a flourizing culture until they invent their own packet switched networks, at which point its only a matter of time until "The lemur from the next whole over does it with humans!"

Re:Dark address space?

[G-funk]

Worse though is when dark address space and normal address space collide, releasing a ddos beyond compare to all hosts within 15 hops....

=)

Not much content in that article

[hackman]

And all that time I thought it was just my ISP that sucked when the "dark side" was taking over the address space. "Oh, now I get it. errrrrrr I think?"

It seems like the article could have had more explanaton and real information on what dark address space is.. I'm still not fully clear after reading. Is "dark address space" just unconnected networks or more subtle. I guess you really need to be a network person to understand fully.

Reminds me of the raging debate over dark matter in Astronomy, and how it accounts for the mass of the universe etc... The debates always involve crazy theories that pretty much contradict eachother until they finally high-enough resolution data..

Re:Not much content in that article

[clacke]

From the actual report:

Dark Address space

A Definition

• The range of topology accessible from one provider, but unreachable via one or more competitor networks
  
• In other words, the onesided differences in Internet provider topology.

Re:Not much content in that article

[kentrel]

I think the term is just used to make it seem deep and mysterious, and that the association with astronomical dark matter brings that to mind.

The Internet is dead! Long live the Internet!

[rhekman]

With all these secret netblocks with unaccountable traffic producers, various dead IP's and hackers gunning after the vulnerable DNS the Internet shall soon implode and the world shall be plunged into a great darkness, second only to the fall of the Roman empire!

Really, is this a huge surprise? Quality of service for unregulated CableCo's is an issue many have to deal with. Plus, human error is a big factor in DNS setups. Then you've got physical problems on end-point sites that don't have redundant connections.

I'd say 5% isn't bad.

Regards

To go where no man has browsed before...

[smaughster]

All this time I thought the slashdot effect was like the sword of Damocles, you never know when it might hit your site. This article shows that some sites can live their lives in oblivion...Anyone looking for the red and blue pill for his site, there you have it.

.info and other new TLDs in the dark, too?

[chrysalis]

ICANN is changing the domain namespaces by adding new TLDs like .info, and accepting new conventions like non-ASCII characters.
The problem is that many software, libraries, and hand-made filters validate domain names based on simple rules like "only 0-9, a-z, dots, and it should end by two characters or com/net/org/edu" .
For instance, I guess that many web forms are currently refusing mail addresses like "john@johncompany.info".
These new, non backward-compatible domain names will probably belong to the "dark and murky net" too.

Re:.info and other new TLDs in the dark, too?

[mbyte]

uh, that would be insanely stupid. what about the country tlds ? like .int .uk .us .ca, etc ?

Re:.info and other new TLDs in the dark, too?

[emag]

The problem is that many software, libraries, and hand-made filters validate domain names based on simple rules like "only 0-9, a-z, dots, and it should end by two characters or com/net/org/edu" .

That's where you get .uk, .us, etc from. And I certainly wouldn't be surprised if places didn't recognize .int. A lot of web forms don't recognize that "+" is a valid character in an email address either, and no amount of "your mail address validation [on this form where I'm trying to give you money] doesn't work" type mails can convince them to check the RFCs.

There will always be plenty of idiots in the world who think they know all valid addresses. Unfortunately, most can't be bothered to do a little research, and then I or someone else will come along, break the forms, and decide to go elsewhere. And unfortunately, the new TLDs also tend to break what for aeons of "internet time" was an acceptable TLD validation.

Re:.info and other new TLDs in the dark, too?

[robbyjo]

only 0-9, a-z, dots, and it should end by two characters or com/net/org/edu

No, you cannot enforce this. How about non-English character domain name? Say, Chinese / Japanese domain name?

Re:.info and other new TLDs in the dark, too?

[armb]

> > only 0-9, a-z, dots, and it should end by two characters or com/net/org/edu
> No, you cannot enforce this. How about non-English character domain name?

What part of "new conventions like non-ascii characters" don't you understand?

Re:.info and other new TLDs in the dark, too?

[Anonymous Coward]

You might have this all wrong?
Do you think?

The domain name maps to an IP address.
Doesn't it?

Once it gets there, who knows how large
the network there really is.

Computers can be attached to multiple networks.

But when we attach them all we tend to think of
them as all the same.

I don't think that the parsing issue concerning the characters of a URL is the issue here. It sounds more like the IP address as the issue.

Am I wrong?

Re:.info and other new TLDs in the dark, too?

[Anonymous Coward]

>No, you cannot enforce this

um... hello?

That's kind of his point, dimwit. He's saying there are sites using software that has an incorrectly narrow limitation on valid TLD's. duh

Re:.info and other new TLDs in the dark, too?

[Florian Weimer]

The problem is that many software, libraries, and hand-made filters validate domain names based on simple rules like "only 0-9, a-z, dots, and it should end by two characters or com/net/org/edu" .

This also explains why MILNET hosts are hidden in the dark.

Re:.info and other new TLDs in the dark, too?

[iso_bars]

hey hey just because my email address is john@johnsoftware..... :)

Re:.info and other new TLDs in the dark, too?

[Richy_T]

Just as bad, the Yahoo shopping application sees the .co.uk address I still use and refuses to accept my order as they "Dont ship to that location". Put in a generic .net or .com or .edu address and it goes through fine. Yet spookily enough, I have been sitting at the same location (Nashville, TN) the whole time.

What makes it even lamer is that the e-mail address entry is on the same page that you enter the shipping address so they should *know* that they won't be shipping abroad.

Just plain old stupid I guess.

Rich

Re:Interesting

[mccalli]

If they put it on some obscure ip block and give it no hostname, who will ever find it?

Not sure about the idea of an 'obscure' IP block? IP addresses follow defined patterns - you can scan for whatever range you like. Certainly the recent Nimda stuff isn't based on hostnames - it's based on going to nearby IP ranges. If you wanted to scan the entire net you could. Might take a while though...

Cheers,
Ian

IPv6

[Anonymous Coward]

Will there be more dark places in which to hide with IPv6? Sounds like a good argument for sticking to the 4 billion IPv4 addresses, and fix the shortage problem by allocating the space better.

Re:Interesting

[ShaunC]

>Its kinda crazy thinking about all the stuff thats out
>there that no one will ever see. I always figured
>anything sensative for military use would be stored on
>a proprietary government network

Might already be that way and we just don't know it. Talk about "dark netspace," nobody holds more of it than the US military... A bunch of class A's - 6.*, 7.*, 11.*, 21.*, 22.* - not to mention the smaller, uglier blocks. I imagine they could be running some sort of TOP-SEC-NET (or maybe SEC-PORN-NET) on one of these, unbeknownst to the outside world.

Shaun

Guess even the military's heard of /.

[Ramuh]

Many discussion sites have marginal value because it is difficult to sort through the background noise to find intelligent, meaningful dialogue. Slashdot is interesting because it resists the typical Internet qualities of anonymity and egalitarianism. ...next thing ya know, they're gonna be using slashcode for missles

Fire a tachyon pulse into the murk space

[yoink!]

We've been running a LAN out of our home for several years now and have never come across much of what is *lightly* discussed in the article. With a heavy user load (we have 1.3Mbps downstream capabilites, usually running at near full throttle) the only black holes we usually encouter are webservers that have crashed or simply gone down permanently; something which I believe deserves a lot more focus than a little bit of missing space. It is nice to know that there are still places to hide, but we didn't need the article to tell us that! :-P

It [the article] was so vague, in fact, that there was little reason wasting the time to read it. Murk space, dark matter, anti-matter, anti-time. I'm going to go back to downloading more STTNG episodes!

A lot of truth to this parent.

[Mustang Matt]

IPv6 could lead to a lot of new problems. I think it's necessary but even with IPv6 we need better methods of allocation.

Re:A lot of truth to this parent.

[dev!null!4d]

offtopic a bit... but I was thinking about IPv6... if every device has an address, mobile phone, mp3player, etc... when these devices break or get thrown away, the addresses will be lost

The real article

[clacke]

The report this article refers to is partially available as a pdf file

Sorry 'bout the whoring..

Invisible web?

[jmerelo]

What does anybody mean by this? Intranets? Bad term if I ever saw one...

If this mean things that, well, are closed to robots, let them be the way they are. Work a bit more, go to the site itself, and do a search.

If it means things in DBs, how come you prove that you've extracted everything in the DB?

In any case, has anybody seen one of those "dark" addresses sometime?

Re:Invisible web?

[clacke]

Not intranets, nets that are supposed to be globally reachable, but are unreachable from some places, due to ignorance or incompetence.

Definition of Dark Address Space

Re:Invisible web?

[richie2000]

In any case, has anybody seen one of those "dark" addresses sometime?

If you could see one, it wouldn't be dark. And if you did see one, They would have to kill you.

I think this is just another .mil conspiracy - those sites and addresses aren't just parts of badly managed webspace - they are websites of black ops, dark projects, stealth planes and hidden agendas. An intranet for the Anti-Illuminati - the Shadows. :-)

Re:Invisible web?

[supine]

Dark address space refers to globally unique IPs (ie. not private IPs as defined by the RFCs) that should be accessible from anywhere on the internet but are not due to one of many reasons. The two reasons I am most familiar with are:

Route filtering.

To reduce the size of the routing table in the memory of their core routers, some providers throw away announcements of small blocks (say /24 or longer masks). This means that unless there is an aggregrate route for that block that will get the packets there eventually, the IP is dark for people using that provider.

Some providers also filter blocks that are listed by the one of allocators (ARIN, RIPE, APNIC) as not being allocated or are reserved for special use. The article infers that this is what happens to lots of .mil

Black holed routes.

Sometimes, either intentionally or accidently, providers announce routes to blocks that they actually can't reach directly. This is usually a misconfiguration or done on purpose to null route blocks containing a host performing a DOS or some other network misdemeanour. This is usually a transient state.

hth
Marty

Re:Invisible web?

[Zocalo]

Yeah. I've seen spam purporting to originate from IP space that hasn't been allocated yet. Shame that having gone to all that trouble the spammer concerned hadn't checked what else was going into his SMTP headers...

Re:Invisible web?

[billn]

Consider the source they used for their data: Routing tables. Aside from announcing the main superblock that says 'Hey, I have these IPs', looking at a full routing table to find out where blocks really wind up isn't effective. I actually had this discussion with a colleague a few days ago. They may announce it, but that doesn't mean it's reachable.

The report cites .mil and broadband land as the largest 'offenders', for lack of a better term. Personally, I could care less if .mil hosts aren't world reachable. By and large, I know for a fact there's a lot that exist that you simply can't get to, or wouldn't want to anyway.

As far as broadband goes, as well as large NSPs, consider how much address space is simply lost to breaking /24's up into /30's for interface numbering. Doing this produces a herd of four IP subnets. You immediately lose two IPs to Network Address and Broadcast, leaving you with two usable IPs, one for each end of the numbered interface, against 254 for a full Class C allocation. Do the math, and that's 64 point-to-point circuits.

Companies like Cisco and Unisphere sport routers capable of numbering interfaces in the THOUSANDS. Even making efficient use of IPs when numbering ATM topologies (common for DSL implementations), you're still losing one IP per interface, in addition to whatever small block is allotted to the customer on the other end. In most cases, every hop you see in a traceroute is one IP of a four ip subnet (exceptions would be LAN topology based peers or transits). For the purposes of security, or simplicity, providers may simply choose to not announce routes to IP space allocated for interface use. Inside their own networks, interior protocols like IGP, ISIS and OSPF can handle local delivery, but the world doesn't really need to know how to throw packets at a router's interfaces.

Cable modems are less guilty of this than most, since they tend to allocate two or four class C superblocks to a neighborhood and mask them accordingly.

Re:Invisible web?

[evilviper]

I am a cable-modem user (don't believe the damn commercials) and recieve a broadcast address of /32, thereby using up less IP addresses than would normally be necessary. Not to mention that the DHCP server is not using a valid IP address. They obviously are doing a good job in changing their ways.

Just think of IPv6

[Anonymous Coward]

I wunder what will happen when IPv6 comes into its own with its billions of IP addresses. You think its hard to track these people now, just wait until people can hide in plain site with the reasonable expectation that nobody will ever come looking to allocate their ill-gotten address space.

Arbor analyzed ISP mail logs?

[ShaunC]

From the article,

Arbor Networks' researchers went to the mail logs of a local ISP and compared several thousand unique mail sources with "murky" addresses spotted in their monitoring.

Am I reading this right? If so, am I alone in feeling uneasy about it? It would be interesting to know what ISP allowed "some research company" to look through their mail logs. I suspect Arbor was only interested in source IP addresses, but it still smells.

Shaun

Re:Arbor analyzed ISP mail logs?

[Anonymous Coward]

The peso bank is now officially open.

Re:Arbor analyzed ISP mail logs?

[billn]

Okay, first off, slow down. I actually know of some of the people involved in that article. Odds are, your intuition is right: They simply used the source address IP data and nothing more. It's fairly simple to get that from a mail server without compromising any of the actual mail content.

The ArborNetwork's crew is as white hat as they come.

Re:Interesting

[Anonymous Coward]

Just think of what will happen when IPv6 comes into its own with its billions of IP addresses. You think its hard to track these people now, just wait until people can hide in plain site with the reasonable expectation that nobody will ever come looking to allocate their ill-gotten address space.

"invisible" sites

[dsinner]

Well, i'd rather have these blocked/unreachable hosts out there from the rest of the internet, if it's for military/government/research purposes. Some of these servers just shouldn't be accessed by the general public, and it keeps the script kiddies out. (well, at least it tries to keep them out.) Any explanation as to why cable users typically fall in to this shadow zone of addresses? that's probably what interests me the most. The gov't shadow zone i can understand, they've been hiding stuff for years, and will continue to do so, but wtf is with the cable users?

Re:"invisible" sites

[clacke]

I remember an episode a couple of years ago where one of the two major Swedish ISPs (Tele2) blocked access to their network for customers of the other major ISP (Telia), due to an argument (can't remember what it was about, though) between the two ISPs. If I understood the report correctly, some of the invisible nets in this case were a consequence of similar disputes, only on a smaller scale.

The cable users are a growing customer base, and everyone wants a piece. It's not surprising that one network would want to inconvenience users of a competing network.

Again? Yawn...

[O2n]

From the article:

Because routers don't normally log such activity, murky address space could hide the full range of antisocial or illegal network behavior, says Labovitz.

Oh no, here we go again. Just because it's about the internet and contains a lot of words that are a little bit different to what "normal" people use daily - like "router", "hosts" and "routable address space" - it doesn't mean it's something dangerous. Not even new.

Can you imagine someone getting funds to look into the origins of "paper spam"? "Oh no, the spammers are using bogus return addresses!" "Bad guys can communicate pretty safe and unhindered by putting their messages in envelopes, stamping them and sendim them by mail!"

I can understand that the guys had to show something for 3 years worth of "research", but unless the securityfocus article is a very-very short, abridged version for the masses, they have no results.

Re:Again? Yawn...

[db279]

Quite a bit of this data has been published by several of the principles involved for quite some time. They haven't been shy about answering questions and talking.

I have to sugest that you should re-read the sentence you quote in its context- I don't understand how the sentence quoted is in any way related to your comment about it. Labovitz is saying that it is difficult to accurately charachterize exactly all the bad things that are going on out there, in part because the bad things are happening in places that shouldn't exist and are therefore off of many peoples radar.

Re:Again? Yawn...

[markmoss]

You are correct that snail mail also makes it quite easy to send annoying anonymous mail, but there is one big difference between e-spam and junk mail: the junk mailer pays the full cost of delivery. Spammers pay less than half of the (much smaller) cost of mass emailings; they rely on intermediate servers to pass their stuff on for free, and finally it clogs up the bandwidth that recipients paid for. It's not bad at the office, where I get a share of a T1 line, but at home where 56K is the only affordable connection available (neither Verizon nor the cable company being ready for the 21st century), any spam that gets past the filters is a major annoyance.

On the other hand, why would I complain if someone pays to have free paper for lighting the wood-stove delivered to my home? 8-)

routing

[underpaidISPtech]

I'm with an ISP in Vancouver, and I can tell you that 1 out of 5 sites I try will fail. If a site cannot be reached, a quick traceroute reveals that UUnet is the culprit. Always a 152.158.xxx.xxx address.

Over the last 6 months or so, it definitely seems like the 'Net is .... not so reliable. Has anyone else noticed a slow degadation in the performance of the 'Net in general? Or is it the crack again?

Re:routing

[Pussy Is Money]

Yah, DNS queries r also SLOW

Re:routing

[Smuffe]

It's the crack. Stop using that shit! Its bad for your health, and can cause serious damage and addiction!

/Smuffe

Re:routing

[vrmlknight]

If it help i work at a co. that has 53 international offices and deal with UUnet/worldcom/alternet/ etc.. All the same (company) but they have horrible service it will cut out where if i dial in to netzero i can ping our office in Mexico or uk or wherever but with the UUNet connection (on both ends not leaving their network) i can't it sucks but that the internet (yes for those that will flame we are looking for alternatives but its not a quick switch)

Re:routing

[Calle Ballz]

I think the general slowdown has to do with the flood of worm traffic. I know that it's calmed down quite a bit, but I think the initial shock caused some backbone routers go into "skip every other packet" mode. I've noticed everything was great until code red hit, all hell broke loose, and things just haven't been the same since.

Re:routing

[Anonymous Coward]

I've heard that vancouver is the party capital of
north america. How do I get one of those cushy
underpaid isptech jobs?

Re:routing

[Anonymous Coward]

God that book kicks ass. Rest in Peace Douglas Adams :(

Re:routing

[dmelomed]

How is this offtopic?

The Details?

[phuqwit]

Does anyone have any more details on this? From what I can see, it appears that routers get compromised and certain addresses get re-mapped. Perhaps class A,B and C and other reserved addresses? This allows these addresses to be used as part of the internet. Then they are removed, hopefully before anyone notices, because the routers don't hold records of changes. Have I got this correct?
Seems like a clear cut case of bad administration.. again. Interesting Reg link on how to own a Cisco router..

The Cause..

[fwc]

The article doesn't really do a good job of saying what this is really about, and the report several people have linked to does provide detailed information, but again you need to have some context to understand it.

What they are really saying is that there are large chunks of the internet which can't talk to each other. This isn't because of firewalling or "hiding" behind a NAT box or the like, but is instead a result of the peering "politics" (which better describes what goes on than policies) between carriers.

Let me explain. If I am ISP A and I connect via peering to ISP B, I can't talk to ISP C's customers through B even if ISP B and C are connected. That is, unless I have an arrangement with ISP B to provide transit to ISP C. ISP C also has to agree to accept my routes even if ISP B provides transit to me.

Generally the big "Tier 1" ISP's peer with each other and generally don't exchange or buy transit from each other (except in some limited cases). Smaller ISP's generally buy transit from one or more Tier 1 ISP's. Some of the smaller Tier 1's both peer and buy transit.

It is not altogether unexpected that with hundreds of ISP's out there that certain ISP pairs just plain do not have connectivity between them. It would be almost impossible both economically, politically, and technically to insure that each ISP could talk to every other ISP out there.

Add on to that that there are some ISP's who set arbitrary limits on how many addresses you have to announce together in one chunk (prefix) before they will even listen to them. If you have a small ISP with insufficiently sized address blocks you may find that your connectivity to the internet suffers.

The other piece which WAS said fairly well is that most people don't notice the problem as 99% of the people out there don't use more than the most popular 1% of the internet. And THOSE sites are almost 100% connected (and if you ran an ISP which wasn't connected to the big sites, you would quickly find yourself without a customer base).

Note that I've taken some liberties with this description so there is some minor technical/political breakage in the description above. Or probably better put, this isn't meant as a technical reference piece on peering policies....

Re:The Cause..

[Anonymous Coward]

I agree... one of the best demonstrations of this is to install AOL in australia (its the Australian version by the way :P)... it works great on the "major" sites, but if it isnt a "big" site, it will take hours (literally) to get there, if at all... Many servers you trace and it dies at about the 16th hop (which is in the US)... even traceroutes to LOCAL ISPs often fail on the AOL network... now im not having a go at AOL or abusing it or anything, but it does happen... As you say, everyone can't be connected in a mesh topology... it just wont happen... I can visualise it now, routers with 65 million serial ports... yep...

The net isnt really a net at all, its more of an extended star topology (for all you networkers)... for those who havent got a clue what I mean is that you have the major servers in the US, and off them hangs other servers, and off them others, etc... Often, there just isnt a route to a server due to router downtime, malconfiguration, or intentional force editing of the routing tables...

so... my proposal is... scrap the name of the internet... i say we all call it ... THE INTERSTAR!!!!!!!!!

Re:The Cause..

[gerardrj]

The easier way to explain this is that it simply is not possible for every roadwayin the country to be connected every other highway. The resulting mess of infinite possible paths would ensure that no traffic ever got to its destination.
Sometimes you need to take smaller interconnecting roads, sometimes you just cant get there from here.
The latter is becoming more and more scarce in the real and digital worlds.

Routes withdrawn after spamming?

[cperciva]

While the proposed explanation is quite possible, there is a simpler explanation: The spammer's upstream ISP disconnected them. Cut them off, and their advertised BGP routes will automatically lapse -- resulting in the rest of the internet simply seeing a spam source followed by a withdrawn BGP route.

Re:Routes withdrawn after spamming?

[komet]

And how would you explain the route APPEARING just before the spam comes in?

And how would you explain the netblock in common not being registered with ARIN/RIPE/APNIC/etc.? Ok that one's easier, but this is more than coincidence.

Re:Routes withdrawn after spamming?

[Marcus Brody]

Um, they register with an isp. send a load of spam. get kicked off isp. QED

OK, this probably doesnt account for all of this, but i guess it accounts for some.

Re:Routes withdrawn after spamming?

[ameoba]

Sounds a lot like telemarketers. The ones that don't block caller ID return garbage numbers that the telco insists aren't connected. Obviously the ISP is taking kickbacks for covering the tracks of spammers.

Re:Routes withdrawn after spamming?

[haruharaharu]

Um, they register with an isp. send a load of spam. get kicked off isp

Nah, BGP routes need only enter into things when you multi-home (get a net feed from multiple upstreams) or carry your own net block around and they never show up for the average dialup/DSL user. Further, if i haven't explicitly negotiated BGP service with my ISP, I probably won't be able to propagate my routes.

It wasn't a military network!!!

[grid geek]

The Internet was never a military network. This seems to confuse many people buts its quite simple. ARPAnet was created to allow the computer science community to share resources since all the new CS departments in the 1960's were calling for more and more government funds to pay for bigger and faster computer systems. It was though that networking them would allow collaboration and sharing of big iorn machines. Futile hope I know 8)

The confusion is based on the fact that Paul Baran at RAND had designed a network which would have used inexpensive links with multiple redundancies to ensure that communications would not be disrupted in a command and control structure for the Nuclear deterant. This idea was also being developed seperately in the UK and called Packet Switching by Donald Davis at the UK National Physics Lab on the first system to use this technology. It was later used as a basis for ARPAnet.

The important point is that when the ARPAnet was created the inventors had never heard of the RAND report and the Air Force had turned down RANDs plan to build a test syestem. It was civilian to the core. However when the military absorbed ARPA to form DARPA the created a nonclassified system called MilNet. This came later and is not the same as saying the Internet is built on a military system

Ok that was my 2c's worth. Any comments?

Re:It wasn't a military network!!!

[billn]

I think a better clarification to say that portions of the network were simply owned by military research facilities. ARPA/DARPA was research labs holding hands and singing a binary chorus. That doesn't mean it was all private sector and universities.

Re:It wasn't a military network!!!

[gilroy]

Blockquoth the poster:

However when the military absorbed ARPA to form DARPA the created a nonclassified system called MilNet.

Um, ARPA was always in the DoD. The original offices were in the Pentagon. The shift to DARPA was just a name change to help refocus on defense projects, rather than civilian research.

Civilian research such as, for example, a vast interconnected computer network. :)

Researchers probe dark and murky net...

[smaughster]

In other words: science discovers goatse.cx

Re:Researchers probe dark and murky net...

[Anonymous Coward]

ah, i see i wasnt the first person to think of that after seeing the words PROBE, DARK, and MURKY

Shortest paths?

[reachinmark]

"...continuously gathering and analyzing core routing tables for three years. In the end, they found that for much of the Internet, the shortest path between two points doesn't exist."

But the longest path does exist? Do we blame the journalist, or the researchers who got paid for three years to conclude this amazingly useful fact?

Re:Shortest paths?

[actiondan]

I think it means that no path between the points exists. Much analysis of this sort is based on findinb the shortest path between two nodes in the network. If the shortest path cannot be found, the nodes are disconnected.

Re:Shortest paths?

[reachinmark]

Yes, thankyou for clarifying that. There I was thinking that "shortest path" meant the path between two nodes that is defined as being the shortest by some function based on weighted paths between nodes.

Or could it be that I was merely poking fun at the tendency to use "no shortest path" to mean "a path does exist, but it isn't short enough for us to call it a shortest path, therefore the two nodes are not connected"?

So spammers can grab anything they want?

[Anonymous Coward]

People with BGP clues, please throw some this way.

Let's say I'm an evil spammer (tm). I want to send out some spam that would be really hard to track down. So, I find a net block that's not being advertised by anyone, but isn't a part of a range that's "obviously" not allocated. Say, a piece of 64/8 or 65/8 that isn't being used yet.

OK, so I configure my spam pumping machine to be an address in that block, and start advertising it. Then I connect out, spew like nuts, and shut down. Once the routes disappear, you have *no idea* where I am or who my uplink is.

So, my request to those that know - is this possible? If so or if not, why?

If it is possible, just how much worse is it going to get when IPv6 starts getting widespread use and you can hide yourself anywhere?

Yes, I realize to do this I'd need a solid connection to lots of other well-routed ISPs. Assume that I do. Will it work? How can we stop it?

Re:So spammers can grab anything they want?

[billn]

You're close to right, that IS possible. The problem is, that someone has had the block allocated to them. It's a simple lookup to the IRRdb or various other registry's to find the owner of the block and contact them. It *is*, however, a pretty damn sneaky move, which fully thwarts the most common tool used to identify a spam source: traceroute.

As far as the IPv6 issue, a lot will depend purely on accounting: How is address space issued? Do you get an IP with your driver's license?

Accountability will be everything, at that point. IPv4, as it's designed, is based on trust. America, as it's designed, is based on civil disobediance. Stop laughing, I'm serious.

Re:So spammers can grab anything they want?

[db279]

In answer to your question- it depends, but certainly in some cases- yes.

Route-filters help address this, but many people don't do aggressive route filtering. Route filters, at least in this context, allow you to describe which route announcements you will accept from who. You typically write route-filters to *only* listen to route announcements for the networks that the person you are peering with owns. If its a multihomed connection then this can be a pain. If its an ISP (especially a multihomed one with multihomed customers) it becomes even more of a pain and becomes a matter of trusting your peers to enforce the right policies at the edge of their network. Some people do things with BGP communities to make this easier, but many folks do not have the clue to do so.

As mentioned earlier in the article, aggressive route filtering can actually increase the discontinuties in the network, but failing to do the right filtering can create opportunities for antisocial/malicious behavior.

There were attempts, with some success to create truly useful route registries- the radb's. MCI and someone else (I'm pretty sure it was the route-arbiter project folks- in which Abha [from this report] played a significant role) maintained these. Some people used these to auto-create route filters, but I think that all got just to darn complicated. I could be totally wrong about this, but that's my recollection.

Not to rant (to late), but to my way of thinking this all is rooted in a basic issue with large multi-entity IP networks- a peer isn't just someone you exchange traffic with for free [or with settlements] it really is a *peer*. By exchanging routing information (especially if you do something like accept/honor MED's) you really do have to trust these people- that means you have to believe they are as competent or moreso than yourself- in other works, a peer- in the truest sense of the word. With extremely democratic large scale IP networks (like the Internet) the meaning and usefullness of the term peer becomes significantly diluted- and this means that the network as a whole is likely to not function at a fully optimized state (or even a merely completely working state) all/most of the time. That isn't a horrible thing, but it certainly does make you reevaluate certain assumptions many people make about IP networks.

Further, I believe that most if not almost all of the "scaling" problems in the Internet today are not as much technical capability problems as configuration/design/education problems. We now have a giant, dynamic network that usually works quite well- can it fail catastrophically? I believe it *can*, but the size, interconnectiveness and diversity tends to locally contain failure conditions- events that would have been extremely catastrophic just a couple of years ago.

I'll stop "lecturing" now, except to say that it is great to see folks like these, CAIDA, Packet Design, and assorted others starting to really try to formalize analysis methods for networks of this complexity- its a great step forward from the cult-of-the-few-geeks (The Internet Routing Cabal wasn't that long ago- not to say they weren't great people who made lots of personal sacrifices to keep things working)

As a footnote, Craig L. and Abha A. have done other related work (before they were with Arbor Networks). I know they presented some of their work on BGP reconvergence time at the Montreal NANOG. I suspect they've presented since then.

http://www.nanog.org/mtg-9910/converge.html

Re:Interesting

[Anonymous Coward]

It's "with my pocket calculator". Or rather "ich bin der Musikant mit Taschenrechner in der Hand".

Re:Interesting

[Anonymous Coward]

informative!?

meta-mod that guy a ham-sandwich!

Five or ten web sites...

[philipx]

the phenomenon is generally not noticeable to average Internet users because most netizens only use a tiny portion of the Net. "Most people access five or ten web sites," Labovitz says.

Oh...(SHOCKED!) so does it mean out there are other sites besides slashdot...
Cool... do you need any special software to browse them ? I use K-Meleon. There's a green icon on my desktop - I double click it and it takes me to slashdot.org, where I read the coolest stuff and then I click the tiny X button ontop when I finished.

Heard about a proggie, though: Internet Exploder that would supposedly take you places where you wanted to go that thay - I always thought it's some travel/tourism/ticket booking application or stuff like that....

Gone researching how to get to the others 4 or 9 web sites...

Sad side commentary

[shani]

One of the people conducting the study, Abha Ahuja, has passed away.

There's another thing you operate

[Anonymous Coward]

My own, so very hard, long, thick cock.

Shut up, bitch, and keep sucking!

WTF?

[Anonymous Coward]

An AC that puts his name and phone number in his post? What are you looking for? Do you think someone will call you with a big dick for you to suck?

Re:Interesting

[Anonymous Coward]

Just think of what will happen when IPv6 comes into its own with its billions of IP addresses

IPv4 already has billions of addresses. 4294967296 of them, to be precise. Oh, I see, you are British, right?

Slashdot? Which Slashdot is that?

[mangu]

Lower down in the link: The higher a participant is rated by his peers at Slashdot, the greater visibility that person's comments will receive, and the greater clout he or she will get as a judge of others who post comments.

Perhaps this guy has accessed /. sometimes, but not for long and, obviously, has never read the FAQ.

Oh, yeah? Probe THIS!

[Anonymous Coward]

Can you find a dirtier story than this one?

DDOS network

[metlin]

I had posted this in an earlier discussion [http://slashdot.org/comments.pl?sid=23740&cid=256 1817] about DDOS networks being built.

Now one poster had suggested something about exchanging possibly "blacklisted" IPs. Perhaps we could build up a DB of such IPs and possibly compare these with those murkier IPs.

I'm almost certain that atleast some of the banned IPs would fall under the murkier regions. In fact, still worse is the fact that some of these come through wingates (as I found out), making it all the more troublesome :-/

Scary though...

Re:Interesting

[Anonymous Coward]

The military does lots of things (that people don't know about), but no porn unfortunately

private network? hidden network? who cares

[Anonymous Coward]

I have a hidden storage device right on my desk.
It isn't connected to the internet.
It is a spiral bound notebook.

Since when does every computer have to be on
the internet? Of course their are 'hidden
networks'.

The mythology that people believe about
computers astounds me. Are people really
this dumb about networks and network topologies?

What about those new toy messaging watches for
children? They are new this Christmas, just
saw them on a commercial yesterday.

They allow the children to send secret messages
to each other. Yet another 'hidden' network.

Sigh...

[MonkeyMadness]

By Definition, if its a part of the internet that is unconnected to the rest how do you expect to see it just casually browsing? The people who did the study had to consult ISP logs for months to understand the problem.

Not to mention...

[wirefarm]

Not to mention stupid things like "ZIP" codes.
Guess what - other countries may have postal codes, but they don't always fall into a format of five contiguous numbers...
Just today, Yahoo told me that I had an impossible 'zip' code, so I did what I usually do in that case - enter "02134", which as many of you know. is pronounced "Oh!, two-one, three-FOUR!", especially if it follows "Box 3-5-0, Boston Mass", which I fill in whenever some braindead php monkey has never heard of my particular prefecture...

Re:Not to mention...

[pomakis]

A couple of years ago I was in a Radio Shack in the Boston area. They wouldn't let me purchase my item (a $2 adaptor) until I gave them my mailing address. When they asked me what state I live in, I said that I live in the province of Ontario. They asked what the two-letter abbreviation for that was, and I said "ON". Their computer wouldn't accept that, saying "invalid state". So they just entered "CA" (for Canada) as the state. They then asked me what my zip code was, and I told them that my postal code was "K1N 1B7". They said that their computers didn't accept letters, only numbers. So I said "argh! Okay, try '12345'!". They tried, and the computer complained that the zip code didn't match the entered state (California). So I then gave them the only California zip code I know by heart - 90210. Thus, Radio Shack now thinks I live in Beverly Hills.

Re:Interesting

[cow ninja]

I always figured anything sensative for military use would be stored on a proprietary government network
It's called SIPRNET, and is well protected.

Yup, thats me

[Gaccm]

For a variety of reasons ranging from contract disputes among network operators to simple router misconfiguration, over five percent of the Internet's routable address space lacks global connectivity.

For weeks i've tried to get to somethingawful.com, i've pinged it, traceroute, i could never get anything. That is the only site that i know up which i can't reach.

Re:Interesting

[Proteus Child]

I always figured anything sensative for military use would be stored on a proprietary government network.

I don't think much of it (if any) is really sensitive information.. it wouldn't be surprising if they were just boxes that J. Random Military Sysadmin installed for a specific purpose (say, a temporary mail server, or a server which holds software to perform an FTP install of (insert system here) and forgot about. It might be documented and lost, it might not be documented at all, but no one's going to touch it because they don't know what it does.

If they put it on some obscure ip block and give it no hostname, who will ever find it?

People netmapping or portscanning entire blocks of IP addresses just to see what's out there? People tracerouting but a funky router returns some weird IP with no reverse record? Who knows.. maybe someone who's setting up /etc/hosts and makes a typo or two.

I know what they're talking about...

[Denor]

I've had a ton of problems getting to certain places on the internet. Whole IP blocks are giving me trouble. Some include:

• 10.x.x.x
• 172.16.x.x - 172.31.x.x
• 192.168.x.x

That's not even the strangest thing. I think I've discovered some sort of strange parallel universe gateway at 127.0.0.1! The computer there is exactly like mine!

Internet Black Holes that I have known

[ptomblin]

When I worked for a company that made routers and other networking equipment (Gandalf, now part of MIke and TErry's Lawnmowers), we had a very large address block. I forget how big it was, it might have been a class B or even an A. But I know we had assigned to our lab three class Cs, one that we used for computers we put on the internal net, and two that we used for computers we put on test networks. Usually the two class Cs on test networks were only connected to each other through a router or bridge that we were testing, not to the internet at large.

Actually, this was a pretty interesting project to many slashdot readers. Using an extremely early version of Linux (SLS 1.02 with kernel 0.99pl14e, I seem to recall), we had a laboratory full of 486s and 386s with two ethernet cards. One was a standard card that was connected to the company lan, and the other was a special programmable card that could be commanded to do stuff that ethernet cards aren't supposed to do, like short packets and bad ethernet headers and the like. This card was connected to one of the lans on one side or the other of the unit under test. There was an automated program running on each box under control of the master box, which ran a script in a custom scripting language that could tell one box to emit a packet, and another box on the other side to check if it got it, and more sophisticated stuff.

It was very cool, and a very early use of Linux in a commercial environment.

Re:private network? hidden network? who cares

[Jeld]

"It is you, who is mistaken about a great many things."

What they talk about is not computers-not-connected-to-internet, and not about general networks. The article is about ADDRESS SPACE. What they are saying is that there are legitimate internet address ranges owned and used by people to connect their systems to the internet which seem to be inaccessable from major parts of the internet due to misconfigured or restrictive or just plain mean routers.

Brooklyn bridge

[DaoudaW]

Pssst, buddy. You new in town. I got some great IP addresses I can sell you. Cheap...real cheap!

Here is an excellent site to monitor internet perf

[Malbosia]

Go here to check for yourself.

12345

[CaptainSuperBoy]

ZIP code 12345 is a special ZIP code belonging to GE in Schenectady, NY.

Re:12345

[Mignon]

Not to be morbid or anything, but the World Trade Centers had their own ZIP code, 10048. I wonder how long it'll be before Radio Shack's computer system stops accepting that one.

Dumb question... mostly OT

[cr0sh]

Please don't mod down though, I am sure others here probably have the same question!

Can you explain (or better, point me to a source explaining) what is meant in networking terminology when you say /24's, /30's - and definitions of classes (A, B, C, etc), as well as what it means when you see like an IP followed by a /nn (like, oh, 27.141.102.18/24 or similar).

This is something I have been curious about for a long time, and would like to learn more (whether it would be useful to me or not).

Thank you for any help you or others can provide...

Re:Dumb question... mostly OT

[Anonymous Coward]

Classes A, B, and C (also D and E) are an older way of announcing groups of IP Addresses. The problem with the system was that it wasn't very flexible and led to organizations wasting lots of address space.
Class A Range 0-127.x.x.x (16.7 million IPs per block)
Class B, Range 128-192.x.x.x (65.5 K IPs per block)
Class C, Range 192-223.x.x.x (254 IPs per block)

If an ISP had a whole slew of Class C addresses to be used for their customers, their routers had to announce each Class C individually, and those announcements had to be carried by all the other backbone Internet routers, leading to very large routing tables that weren't efficient.

Conversely very large organizations like HP and Stanford had Class A addresses, that only required a single route announcement, but even they couldn't really use 16 MILLION IP addresses.

Classless (CIDR) system was implemented in 1994, to ease the problem.

/8 = Class A of address space
/16 = Class B of address space
/24 = Class C of address space
/30 = subnet of 2 usable addresses

The slash refers to the number of significate bits that constitute the unique network address (of the 32-bit IP addresses, with the rest of the bits identifying the host)

This means that if and ISP has 4 contigous IP addresses he can announcing as 1 /22 instead of 4 separate /24s (under the old classful system)

The change to CIDR has kept the Internet from running out of IP addresses for the next number of years. One downside of this system, is that Internet users can't keep their IP addresses when they change their providers. That's because the user may have a /24 but in reality it may be part of a /19 that the ISP is announcing for all its customers.

Re:Dumb question... mostly OT

[billn]

Okay, slash notation for subnets.

When you look at the subnet mask for a network, say 255.255.255.0 for a Class C, break it up into the appropriate bit segments:

255 255 255 0 converted to hex is FF FF FF 00. Each value of FF is 8 bits. 255.255.255.0 is a 24 bit mask, or /24. 255.255.0.0 is a 16 bit mask, or /16. 255.0.0.0 is an 8 bit mask, or /8. A single host mask of 255.255.255.255 is a 32 bit mask, or /32.

You can determine the size of a network block (or subnet) by subtracting the bitmask from the maximum possible value (FF FF FF FF). A /24 would contain 256 ips. A /30 would contain 4. A /23 is two class C's, but a /22 is four. I leave the math as an exercise because I'm poor at it.

Re:Dumb question... mostly OT

[[amorphis]]

/XX notation is known as Classless Inter-Domain Routing, and as always, consult the Oracle

Dumb question... mostly OT

There are no stupid questions; only stupid answers.

Re:Dumb question... mostly OT

[haruharaharu]

Okay, the class A/B/C issue has been covered, so I'll tackle the /n thing

When you see an address A.B.C.D/n, the high (first) n bits of the address are network bits and the rest are the host bits. This is usually used to route traffic - compare the network bits and, if they match, do this. They can also be used to aggregate network blocks or to subdivide them (they are usually allocated in blocks of n*class C networks. when you subdivide them (getting shings like /28 or whatever), it's called subnetting, and when you aggregate them, it usually gets labelled as CIDR, as it no longer follows traditional netmask boundaries.

Within each network, there are two speciall addresses - setting the host bits to all 0 or all 1. If all of the host bits are 0, you have the network address. If they are all 1, you get the net broadcast address.

In a /30 network, there are 2 host bits, leaving 4 possible hosts. Since two of those are reserved, you have two usable hosts. This makes /30 really inefficient and also the smallest possible network.

Re:Interesting

[Amazing Quantum Man]

By definition, any classified machine CAN NOT be connected to the Internet. Try it, and you could be looking at a lifetime vacation in Leavenworth.

When I worked for a defense contractor, we were exceptionally paranoid about this sort of thing.

What is it made of...

[Amazing Quantum Man]

Is the "dark address space" made up of strange websites? Or perhaps charmed ones?

murky net spaces eh?

[zoftie]

Dark matter for physicists, murky patches of net for CS types?

> Arbor Networks' researchers went to the mail logs of a local ISP and compared several thousand unique mail sources with "murky" addresses spotted in their monitoring. They found that 30 of those addresses sprang into existence shortly before sending the email, and quickly vanished afterwards.

Murky alright, frickin' SPAMMERS using dialup accounts. Article emphasizes obvious, rides on ignorance of uncouth. UUCP is of same type, does
it mean that net was not connected in those days
either? How about that one: http://www.blug.linux.no/rfc1149/writeup.html

p.

Re:Interesting

[david duncan scott]

Is that what they told you?

Anyone tried a massive nmap?

[josquint]

might be somewhat offtopic, but...
anyone actually try an nmap -sS -O 0.0.0.0-255.255.255.255
i think it'd be an intresting project anyway. your ISP'd be pissed i'd think and it'd take FOREVER...
but it think it'd be interesting to see the log file... make for some interesting data mining maybe too

You all don't understand IPV6

[Anonymous Coward]

While I'm far from an expert, AFAIK:

IPV6 is more than just "more addresses". The addresses are created by concatenating several different numbers such as your providers network string and your NIC's MAC address. So one immediately can tell who the ISP is (although the specs allow one to "create a MAC" on the fly). Another feature is that routing is integral to IPV6 so that that it is not an ad hoc add-on. Additionally, routing can be done with multicast messages to improve reachability.
Finally, one of IPV6's key benefits is that it eliminates the need for NAT'ing. So, it is likely that IPV6 would be the solution to this problem.

Re:Interesting

[Fortmain]

FWIW, the REALLY sensitive stuff is only on internal nets, air-gapped from the internet. An outsider can't break in and look at your files if there is no connectivity.

That's why I always laugh whenever I read about some some 'l33t d00dz' hacking into military computers and compromising all our secrets. They may get some semi-sensitive, For-Official-Use-Only type crap, but they're not going to get the true classified stuff.

Someone below mentions the SIPRNET. Yes, it exists for lower-classified stuff, but it has very few connections to the general internet, and those that exist are VERY tightly controlled. If you try to slip in through one of them, you will have the OSI, CID, FBI, and a bunch of other letters knocking on your door. (Yes, the government does have a bunch of very intelligent, capable computer security guys. No, they don't noise it around - better to let the 'l33t d00dz' _think_ they are getting away with it.)

One viable explanation for hidden nets

[per unit analyzer]

There is at least one somewhat plausible explanation for why some hidden networks are sucking up valuable address space from the global connected Internet.

There are what I would call "confederations" of sites and networks which maintain connectivity through private networks, most likely research-community and government oriented. e.g. Abilene(Internet2), CA*Net, APAN, ESnet, etc. The members of these confederations may be different research labs, universities, etc which have need for complex routing policies based on endpoint and which private network to take. Unfortunately, the tools for implementing such policy are weak and often fall back on making decisions based on IP address. This in turn means that certain IP addresses are used to cause traffic to flow in a certain way and must be blocked to the public Internet.

Now with all of that said, one would naturally assume this could be accomplished with RFC 1918 (private) address space and shouldn't require using up valuable public address space. This is true if there was only one confederation, but many of these semi-private groups exist and many of the individual organizations participate in multiple confederations simultaneously. This means if RFC1918 address space were used for each confederation, someone would need to be responsible to make sure no conflicts existed in the variuous private address spaces. This would be problematic becuase 1) the confederations generally don't cooperate with each other (not in an antagonistic sense, more like ships-in-the-night) and 2) this would take take up someone's time which even in the research community is genereally not free (as in beer). Some confederations don't even know others exist. Furthermore, even if such a project were undertaken, all of the participants would need to agree on a common chunk of the RFC1918 space. This would be hard to do as many organizations probably have already used varying parts of this space for their own purposes. (again, none of which were coordinated.) Some people would not be happy about having to renumber.

So in order to maintain unique address space amongst this web of semi-private networks, the particpants simply use additional addresses out of the global Internet address space but only announce it amongst themselves. i.e. The global Internet registry is used to also coordinate use of addresses across these multiple, private interconnections of (usually) public institutions.

Now, I don't think this is the main cause of hidden address space. In fact it's probably so small compared to other causes that it is probably not necessary to address at this point. However, I wanted to offer it up as a legitimate reason some parts of the global Internet are not reachable from commodity ISPs.

-z

Re:Interesting

[nathanm]

Someone below mentions the SIPRNET. Yes, it exists for lower-classified stuff, but it has very few connections to the general internet, and those that exist are VERY tightly controlled.

No, the SIPRNET is not physically connected to the internet anywhere. Unclassified info only is sent over the internet (NIPRNET in gov't terms).

See my post in this earlier /. story for more info.