Fragmentation is a problem, however, I am not seeing it as fargmented as I thought it could be. I would say about 99% of the ip's in/24's route to the same last two hops. Some isp's break it up a lot, but that's on their small part of their networks. When that happens it is relfected in the image with "less resolution" of the edges.
The scanning is done using random/16's that are checked out from the valid address space, they are scanned by 3 different nodes on different backones in different parts of the world. Soon they will be done from maybe about 6 different backbones.
Also, using the old term, "Ceci n'est pas une pipe" (this is not a pipe), the "map" is a working image of data, nothing more. It's art, math, whatever you want to call it, but I never say "this is the internet."
The best image I have now is stated as: "This graph is by far our most complex. It is using over 5 million edges and has an estimated 50 million hop count."
If you want to see the data for yourself and play with it, you should download Alex's LGL view.jar file and load the data. You can zoom in on every single route on any point and then turn on the ID (or IP of that point) run some traceroutes of your own and you will see that your traceroutes match the data in the images.
Anyway, back to my point: No image can be "the Internet", it is just an image.:) Depending on what philosophy you subscibe to.
This was a small project that grew, we are prducing neat maps and we are solving issues with them as the days go on. This is about 4 weeks of work and it's neat, but not anywhere near done.
I'll be looking forward to help for anyone.:)
I have herd of CIDR, but I don't have a complete list of every single subnet. If you know a.b.c.0/24 routes to a specific edge router, then you can assume the rest of the subnets route there as well. It should be a safe guess.
If you don't like the way I do things, feel free to do it yourself or submit code to the project. Keep in mind this project is about 4 weeks old now. Mapping the/24 space seems to be doing the trick.
Read the FAQ, I do map RFC1918 stuff that is used to route Internet packets. However, I am not mapping PRIVATE networks that are not part of the Internet. We all know that private networks are huge as well. So one can assume that on the edge of every node on my maps, there exists a potential huge cluster of networks... big image. If I had enough cpu and memory I would play with that a little.
-Barrett (opte guy)
200 million hosts, we are not mapping hosts, we are mapping networks. We don't really care where the hosts are because if you know where every single class C is located then you know where very host is located with-in that route. It may have one of the end hops wrong if someone is using RFC1918 or subnets, but that's not important for a huge map and it is easy to clean up for research as well.
If you want a map of the entire internet I can do that, send me a high-end sun system so I can process this stuff.
We do not use domains, DNS information would be another map all together. Rather, we are using route information based off of specific tools (Dan Kaminski's scanrand2.)
The data is then pushed into a db and processed by any graphing tools you wish. I like LGL because it does a good job with the hardware we have.
That is the very point! This information changes so fast that a single map a year is worthless. If we are collecting data all day every day and producing maps daily, along with diff maps, and animations, this stuff becomes something rather neat.
Slashdot Mirror
Fragmentation is a problem, however, I am not seeing it as fargmented as I thought it could be. I would say about 99% of the ip's in /24's route to the same last two hops. Some isp's break it up a lot, but that's on their small part of their networks. When that happens it is relfected in the image with "less resolution" of the edges.
The scanning is done using random /16's that are checked out from the valid address space, they are scanned by 3 different nodes on different backones in different parts of the world. Soon they will be done from maybe about 6 different backbones.
Also, using the old term, "Ceci n'est pas une pipe" (this is not a pipe), the "map" is a working image of data, nothing more. It's art, math, whatever you want to call it, but I never say "this is the internet."
The best image I have now is stated as: "This graph is by far our most complex. It is using over 5 million edges and has an estimated 50 million hop count."
If you want to see the data for yourself and play with it, you should download Alex's LGL view .jar file and load the data. You can zoom in on every single route on any point and then turn on the ID (or IP of that point) run some traceroutes of your own and you will see that your traceroutes match the data in the images.
Anyway, back to my point: No image can be "the Internet", it is just an image. :) Depending on what philosophy you subscibe to.
This was a small project that grew, we are prducing neat maps and we are solving issues with them as the days go on. This is about 4 weeks of work and it's neat, but not anywhere near done.
I'll be looking forward to help for anyone. :)
I have herd of CIDR, but I don't have a complete list of every single subnet. If you know a.b.c.0/24 routes to a specific edge router, then you can assume the rest of the subnets route there as well. It should be a safe guess. If you don't like the way I do things, feel free to do it yourself or submit code to the project. Keep in mind this project is about 4 weeks old now. Mapping the /24 space seems to be doing the trick.
Read the FAQ, I do map RFC1918 stuff that is used to route Internet packets. However, I am not mapping PRIVATE networks that are not part of the Internet. We all know that private networks are huge as well. So one can assume that on the edge of every node on my maps, there exists a potential huge cluster of networks... big image. If I had enough cpu and memory I would play with that a little. -Barrett (opte guy)
200 million hosts, we are not mapping hosts, we are mapping networks. We don't really care where the hosts are because if you know where every single class C is located then you know where very host is located with-in that route. It may have one of the end hops wrong if someone is using RFC1918 or subnets, but that's not important for a huge map and it is easy to clean up for research as well. If you want a map of the entire internet I can do that, send me a high-end sun system so I can process this stuff.
We do not use domains, DNS information would be another map all together. Rather, we are using route information based off of specific tools (Dan Kaminski's scanrand2.) The data is then pushed into a db and processed by any graphing tools you wish. I like LGL because it does a good job with the hardware we have.
That is the very point! This information changes so fast that a single map a year is worthless. If we are collecting data all day every day and producing maps daily, along with diff maps, and animations, this stuff becomes something rather neat.