Ask and you shall receive:-). I have more information on that than you'd probably like to know. The back-end is actually quite similar to the PunkSPIDER project's back-end and uses all of the same principles, most of the same open software as its base, and even reuses some of the code (in fact, once it's done I'll probably make the back-end of web 3.0 a part of PunkSPIDER 2.0 - free and open source of course). So with that said here's info on how PunkSPIDER was built, which should give you a solid start to how we're building the web 3.0 back-end:
And last but not least, if you want to know even more feel free to contact Hyperion Gray at punkspider@hyperiongray.com or follow me (Alejandro) at @DotSlashPunk on Twitter. Oh and thanks for the feedback on the buzzy name, it's meant to be a little over the top, but we'll keep your comment in mind!
1) The statement that we "just run Nessus" is incorrect. We wrote our own scanner that works on a Hadoop cluster. Why is this important? It means that we can handle a lot more scans than anyone else (several thousand per day with a small cluster) and it's also specifically made for mass scans. This is important in point 2 below.
2) The process you're describing is for finding a vulnerability in a piece of software in general (e.g. a common CMS), not a specific vulnerability in an implementation of a piece of software (e.g. a specific website). That's a huge difference. You wouldn't put a CVE up for a SQL injection bug in a specific implementation of a site (you would only if it was common to an entire CMS for example). Anyway, what we hope is to build a community of like-minded security folks that can help those website owners fix their *specific issues* first and if applicable go through the process you describe when needed. We also want to provide this for free.
3) What if the vulnerability is in a custom built site that no one cares enough about to do security research on. Who's letting them know their issues? We hope to provide a view of this to the website owner and yes, push them a little to get their security ducks in a row.
4) We're not attention whores or jackasses. Calling people names isn't nice and makes us sad.
So one thing that we've been trying to make clear is that the project is *on track* to scan the entire Internet, we haven't scanned everything yet. We have scanned about 70k sites and have under 4 million indexed. Our next version is going to be clearer on what is and is not scanned - currently we just say 0 vulnerabilities if we haven't scanned it, indicating that we have not found vulnerabilities in it yet - not necessarily that it doesn't have any.
This was all part of our ShmooCon presentation which just hasn't been released to everyone yet! The system is self-sustaining at this point so these numbers are constantly going up. The "not pretty" comes from the fact that we have over 100,000 vulnerabilities from just scanning about 70,000 sites (some sites have multiple vulnerabilities).
Slashdot Mirror
Free and open source, dude. Just not released yet because it's funded by DARPA CFT and still ongoing research: http://www.wired.com/dangerroom/2011/11/darpa-fast-track/
Erm. Unity is a well-known 3D gaming engine, dude.... http://unity3d.com/
Ask and you shall receive :-). I have more information on that than you'd probably like to know. The back-end is actually quite similar to the PunkSPIDER project's back-end and uses all of the same principles, most of the same open software as its base, and even reuses some of the code (in fact, once it's done I'll probably make the back-end of web 3.0 a part of PunkSPIDER 2.0 - free and open source of course). So with that said here's info on how PunkSPIDER was built, which should give you a solid start to how we're building the web 3.0 back-end:
(1) A link to the talk at ShmooCon on PunkSPIDER which gives more info than you'd ever want to know about the back-end: http://www.hyperiongray.com/shmoocon
(2) If you're in a rush you can read some basic stuff about it here: http://www.hyperiongray.com/node/18
(3) If you really want to get into it you can download PunkSCAN (the PunkSPIDER back-end) on bitbucket and take a look: https://bitbucket.org/punkspider/punkscan
And last but not least, if you want to know even more feel free to contact Hyperion Gray at punkspider@hyperiongray.com or follow me (Alejandro) at @DotSlashPunk on Twitter. Oh and thanks for the feedback on the buzzy name, it's meant to be a little over the top, but we'll keep your comment in mind!
Alex
Hmm, a few issues with this...
1) The statement that we "just run Nessus" is incorrect. We wrote our own scanner that works on a Hadoop cluster. Why is this important? It means that we can handle a lot more scans than anyone else (several thousand per day with a small cluster) and it's also specifically made for mass scans. This is important in point 2 below.
2) The process you're describing is for finding a vulnerability in a piece of software in general (e.g. a common CMS), not a specific vulnerability in an implementation of a piece of software (e.g. a specific website). That's a huge difference. You wouldn't put a CVE up for a SQL injection bug in a specific implementation of a site (you would only if it was common to an entire CMS for example). Anyway, what we hope is to build a community of like-minded security folks that can help those website owners fix their *specific issues* first and if applicable go through the process you describe when needed. We also want to provide this for free.
3) What if the vulnerability is in a custom built site that no one cares enough about to do security research on. Who's letting them know their issues? We hope to provide a view of this to the website owner and yes, push them a little to get their security ducks in a row.
4) We're not attention whores or jackasses. Calling people names isn't nice and makes us sad.
So one thing that we've been trying to make clear is that the project is *on track* to scan the entire Internet, we haven't scanned everything yet. We have scanned about 70k sites and have under 4 million indexed. Our next version is going to be clearer on what is and is not scanned - currently we just say 0 vulnerabilities if we haven't scanned it, indicating that we have not found vulnerabilities in it yet - not necessarily that it doesn't have any. This was all part of our ShmooCon presentation which just hasn't been released to everyone yet! The system is self-sustaining at this point so these numbers are constantly going up. The "not pretty" comes from the fact that we have over 100,000 vulnerabilities from just scanning about 70,000 sites (some sites have multiple vulnerabilities).