The Dark Side of the Web
Barence writes "Beneath the web pages indexed by Google lies an online world that few know exists. It's a realm of huge, untapped reserves of valuable information containing sprawling databases, hidden websites and murky forums. It's a world where academics and researchers might find the data required to solve some of mankind's biggest problems, but also where criminal syndicates operate, and terrorist handbooks and child pornography are freely distributed. Interested? You're not alone. The deep web and its 'darknets' are a new battleground for those who want to uphold the right to privacy online, and those who feel that rights need to be sacrificed for the safety of society. The deep web is also the new frontier for those who want to rival Google in the field of search." The melodrama is tempered, though: "The deep web isn’t half as strange or sinister as it sounds. In computer-science speak, it refers to those portions of the web that, for whatever reason, have been invisible to conventional search engines such as Google."
TFA:
>terrorist handbooks and child pornography are freely distributed. Interested? You're not alone.
No, actually, speak for yourself!
It's called the Metaverse, created by Neal Stephenson in Snow Crash. We now know it as Second Life.
I find it ironic that in the online world there are places where the Terrorists and those who support them are calling us, the Freedom Loving People, terrorists !
By the original Terrorists I mean those who strap bombs to themselves and go KABOOM ! taking themselves with innocent people around them.
What I find ironic is THEY call us terrorists !!
Muchas Gracias, Señor Edward Snowden !
http://www.robotstxt.org/
Says it all really, no need for a melodramatic "article" trying to draw parallels between the non indexed and page ranked portion of the net, and kiddie porn.
Some of us just don't want google indexing out stuff on general principles.
FX, types "brain tumour" into google
up pops page full of links asking me if I want to buy a brain tumour on fleabay...
http://slashdot.org/~GuyFawkes/journal
There is no dark side of the web really
Matter of fact its all dark.
From what I've seen and heard this 'hidden' information is hidden on a purpose - most such sites I've ever encountered are trafficking (child) porn, software, audio and video - there's next to zero informational value in this undernet. As someone once said "Information wants to be free" and if it isn't let it die.
The real problem which this article doesn't even touch is that sometimes it's getting very hard to find the information buried in millions of pages Google returns to your query.
The author explains that there is a lot of content behind password protected sites. I had no idea that google didn't know your password!
I heard something about a robots.txt file somewhere before, but I thought that all robots where smarter than me anyway.
I also learned about something called freeweb that may or may not be used for good or bad things. I then learned about TOR which also may or may not be used for good or bad things.
This article really opened my eyes to the vastness of the Internets
[ominous music, thunder and lightning] ...
A dark and foreboding place with over 5000 pages of scientifically useful information with more than 60000 links between them, all craftily hidden
[suddenly the music stops] ... behind a robots.txt file. Hmm. Not so interesting.
Actually, it's not that dark or foreboding, it would just be a pain if Google tried to index it all because the algorithm they use would mean the key pages for a particular item (i.e. where items were actually defined) would end up down the ranking list because of the way that the links work in this particular dataset. The top pages of the site are indexed and people can use the specialized search tools there to find what they are looking for, or they can hop directly to an entry using a predictable URL if they know its name.
Oh, and if a bot wants to ignore the robots.txt file they can get away with it for a little while ... and then they get tossed into a tarpit. [Queue the ominous music again ... BWHAHAHAHA!]
The article says it was "Posted on 3 Sep 2010 at 15:47". Unless I've missed something, we're still in March 2010...
there is no mystery to this 'deep web'. massive data reserves? quite likely. personal, but unsecured servers hosting copyrighted content? even more likely.
This kind of article will only make things worse for a future defendant trying to explain he wasn't coordinating with 'the deep' in the distribution of his movies from his computer to his Mythbuntu box.
Just maybe not everyone in the world wants to be Google's bitch and allow them to mine their precious information for profit. Information may want to be free, but information is also power. Secrets are valuable to those who hold them, and in a near future world where information becomes increasingly more valuable, those who hold the secrets will be the most powerful.
Now just ask yourself, are you willing to submit to the likes of Google and give up the right and freedom to decide what to do with your information? Your secrets?
Do we continue to sell our individuality, our identity so cheaply?
Darknets are a concern. What is the link with the Deep Web? The only connection seems to be that they're both unindexed by search engines.
I thought the article might get to the point by the last page, but it was still talking about child protection and terrorism (in company databases???) I had wondered whether this confusion was down to an incautious academic, but the doesn't seem to suggest it: http://ai.arizona.edu/research/terror/
The deep web and its 'darknets' are a new battleground for those who want to uphold the right to privacy online, and those who feel that rights need to be sacrificed for the safety of society
Corporations, wealthy individuals and people in power keep their right to privacy. That's not good for the "safety of society". See the ACTA negotiation. Most of the calls about the future of society are made in a non transparent way, by corporations, the psychopaths that run them and corrupt politicians. If I don't keep my right to privacy ( and this looks like a lost cause) then I want them to lose it as well. I want a full public database with detailed information about every dollar owned and every move made by politicians and members of a corporation board. I want every government contract to be published on an easily searchable database. I want all meetings between corporation boards and/or government officials transcribed and published on another publicly search able database.
When his defense asked, "Which computer has Jon Johansen trespassed upon?" the answer was: "His own."
> In computer-science speak, it refers to those portions of the web that, for whatever reason, have been invisible to conventional search engines such as Google."
Spare me your mumbo jumbo, doctor!
What have you Done!
http://xkcd.com/591/
Now the darknetz are doomed !11!!!!1!
Vast collections of literature (many, many LoCs worth) exist outside amazon.com containing esoteric theories, morbid historical narratives and subversive ideas.
Set your phasers on "funky"!
If child pornography is being freely distributed amongst anonymous networks of paranoid people, what is the problem?
The vast majority of people who use onion routing are very cautious people, so very few will be stupid enough to leave a trail which could identify them (such as a payment) as doing anything which is seriously controversial or illegal. It would be absurd to suggest that anybody is going to profit from producing child pornography and distributing it through anonymous networks.
If somebody produced child pornography as a "hobby" (instead of for profit, which would result in a swift arrest anyway), it's pretty obvious that the producer would produce the pornography for themself regardless of whether they distributed it. So again, anonymous networks are not contributing to a problem, nor is the alleged availability of child pornography.
The majority of perpetrators of child sexual abuse are the parents of the child. If people genuinely wanted to stop child abuse, they would focus on protecting children from abusive parents. Instead, politicians and police chiefs tend to focus on matters which score politicial points and win votes; parents are not an acceptable target because they constitute a major component of the electorate. Claming to fight child pornography is much easier for politicians and police chiefs, as they will not lose significant support and they can easily claim a victory without any risk of being exposed as liars; after all, who is going to check the evidence?
"To the future or to the past, to a time when thought is free" ~ Nineteen Eighty-Four
... I came across to this nice article regarding deep web. It has some techniques on how to search, access and exploit deep web. It is worth to look also the other articles of this site...
Until the skies turn blue...
Until the air of freedom strikes us...
I was thinking about exploring those 'darknets' but child porn? I don't even want my browser pre-caching its way into those websites much less directly stumble onto one. At least I was warned. Anyone foolish enough to go there can't expect to feel like a victim if they get caught in a dragnet for showing up on a bad site's web access log.
--- Grow a pair, liberals... stop letting the Republicans bully you!
Remember, this is an analogy using an unrelated but example with hindsight already established, unlike what this article is about.
The idea of the stock market was to allow people in invest in companies they believed in, sort of a put your money where you mouth is vote.
But due to the higher and higher levels of abstraction and manipulation at these higher levels of abstraction, putting your money in companies you believed in, is no longer what moves the market. Instead there have been some very bad things resulting in the manipulations of the markets at these higher abstract levels.
You might not even know where your investment is being put, had you handed it over to someone else to do for you, financial institutes and packages. Examples of very bad results include The Trillion Dollar Bet that drained Southeast Asia, including Indonesia (cia reported 88% Muslim) and gave rise to the repercussions of 911 (don't argue - follow the money instead, Worldcom, Enron, etc. some of the losers in the gamble) and Bernard Madoff's multi-billion dollar Ponzi scheme.
So this deep dark net.... it may not be so bottom line (money) connected, or maybe some of it is with insider trading info, but the results eventually show in the open.
And clearly the saying, "what we make we can break" applies here regarding anonymity, a lock that can be broken, as locks are for honest people.
Its not like governments don't have their own means of inside communications, of which if the public knew what all was communicated, there would be opposition as well.
It comes down to what humans will do, good or bad, where the only difference is in what tools they use to do so.
And the higher the level of abstraction being used ... the more damage it can result in. Good things generally don't require such sneaking around, spy vs. spy... And in the virtual world, what does it matter really matter, unless it has real world results?
At what point in technology advancement will thought police enter the picture? Or perhaps that question should be in the past tense.
What does the article really accomplish? More players in the spy vs. spy game. Absolutely nothing more! Advertising the dark web.
Why increase the number of players?
The solution to the trillion dollar bet was to release the formula used into the public hands, effectively nullifying its ability to be used in such a damaging manner.
It is like going to garage sales. You can expect to see a ton of the same old garbage, but usually someone has 'stored' something they don't care about but you really want. Occassionally there are priceless buried treasures that should be shared with the whole world. If marketers could regularly go through them all, it would provide unparalled insight into what we buy and store. Unfortunately, (or fortunately depending on your point of view) everyone doesn't get to ramble through our garage unless we give them access or they break-in.
They are not really "darknets", a more accurate name is NOYBnets. (None Of Your Business)
If you still remember how to use those BBSs, the IRCs, then you can get back into the 'darknet' world I guess, though today it is also about appearing/disappering websites and botnets. Botnets like Zeus got that whole 'FreeNet' idea and have their own implementation, only it's not exactly free.
You can't handle the truth.
I was recently told by an academic in computer security that only about 13% of the web are indexed by search engines such as Google. Does anyone know anything about this?
the search engines don't crawl the alt roots either! that means that there are AT LEAST FIVE WHOLE PAGES they've missed there!
Holy cow, people have been 'mining' the 'deep' 'dark' web for years. It's been the 'next frontier' for a decade. 'Darknet' has been a mysterious and edgy term for a private network for a long time. The VPN I setup last week is a darknet, I suppose ...
http://www.google.com/search?num=100&hl=en&safe=off&q='deep+web'+site%3Aslashdot.org
I made my forum/website invisible to search engines because I only want members coming from one single place which I have a link on. Then only certain members join and it's actually quite nice and peaceful. That's really all there is to it. No terrorists or anything.
Google's Super Secret Search Algorithm: SELECT @search_results FROM internet WHERE @search_results = 'good'
So the bottom line is there is more private data then public data floating around.
Go figure. *snooze*
---- Booth was a patriot ----
It must be a slow news day for something so technical to show up as a news article. But then again, the article has all of the necessary alarmist features, so I guess they had to let it through.
Every 12, 16 months or so I go "Oh yeah, Freenet exists...I should check it out again." But every time, I can't answer the question: Is there anything actually HERE?
Freenet: Is there actually anything worthwhile on it?
"These people look deep within my soul and assign me a number based on the order in which I joined" --Homer re:
Darknets are not new for me. Every porn site I visited had a dark background thus helped me to "penetrate" into it further.
I thought Darknets were private and encrypted vpn based file sharing systems. They work based on invites and trust.
I think the author meant Darkweb
Not always a good thing, as http://www.goatse.fr/ proves quite plainly.
http://en.wikipedia.org/wiki/Six-Day_War
So this is the new term for IPv6?
-- I have a private email server in my basement.
Google has built quite an empire on making people believe that they are the defacto standard for search. They should be commended for the quality of their applications but sadly the marketing has led people astray. I actually took a trip to my local University to do some research. A day login gave me access to thousands of Scientific papers that I would otherwise have to pay hundreds of pounds for. Doing real research takes footwork and hardwork. The web can do a lot but you have to know where to look. See http://narconews.com/Issue64/article4073.html , http://deepwebresearch.blogspot.com/ , http://society.guardian.co.uk/e-public/story/0,13927,1195901,00.html
"The deep web isn't half as strange or sinister as it sounds. In computer-science speak, it refers to those portions of the web that, for whatever reason, have been invisible to conventional search engines such as Google."
In reality, we might be better off if more of the Web were "dark" in this sense. Google (and the other major search sites) are fairly good at indexing text in English (and other human languages). But there's a lot of data online in those "sprawling databases" that are not encoded in human language, and are much better indexed by separate software that is designed to do that job right. One of the very real problems with google is that it so often returns sites that seem to contain strings of "words", but in fact aren't human language. I'm sure most people here are quite familiar with the results of "matches" on such data.
This isn't any sort of new observation. There are by now several thousand projects around the Net that deal with distributed databases of information encoded in various ways that are relevant and important to the people who use that data. One very well-known example is the GIS data that's used by the mapping software in your GPS gadget. This is very useful information, but for google to index it and return it when you google for "Brittney Spears" is not really a good idea. Similarly, there are large online databases of such stuff as astronomy and DNA data, and the person looking for the info on Madonna or Shakespeare is not happy with getting strings of parsed DNA.
I've been involved in building a search facility for several kinds of technical data, whose nature is uninteresting to nearly everyone here, but would server as one of many such examples. There are a thousand or so sites that use these formats. The data is "encoded" in plain-text form, mostly for ease of sending via any available method such as email. But most of the sites use robots.txt to tell the search bots to ignore the directories that contain this data, because the people using it are themselves annoyed by getting their own data back in google searches. The data contains shorts bursts of letters, so search bots treat it as text and index all the "words". The result is worthless to nearly anyone, including those who are familiar with and using this data encoding.
As an example, I found a file in my site that contains the string "red haired girl", did a google search, and it found a match on the file. People who click on that match, perhaps due to a redhead fetish, might be disappointed to find that it contains lines of text like "E3 ECE | FAB Ace | fec ecA | B2 e efg | age f2 a |fec ecA |BAB cAE |1 FAF FAF :|2 FAF". That's probably not what most people would expect for such a search. (Of course, if you follow irtrad-l, you'll probably recognize that notation and start humming along. ;-)
A much more rational approach would be to develop separate search sites for each such kind of specialized data. The google gang does sorta understand this, of course. Thus, Google Maps uses the GIS data, but Google Search doesn't return GIS data (very often ;-). They have software for searching GIS data, as do all the mapmakers, but they understand that there's little benefit in trying to merge GIS and human-language searches. Share some of the software, yes, since net navigation is a common task in all of them. Present the GIS addressing info in text form, too, since it's useful for humans to search that. But the text analysis and indexing schemes for the basic data have little in common, and need mostly separate software to do a good job of searching the data.
Anyway, it's no doubt true that there is a lot of "suspicious" information (semi-)hidden online. But it's also true that a lot of data is hidden from the major searchers simply because they don't do much that's useful with the data, and there's no reason to waste a server's CPU time servicing requests from googlebot or the other human-language search bots.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
It's more commonly known as 4chan... or, "the bottom of the internet."
where all the junkies go.
Where everything is fast
And everyone gets slow.
I see you never tried the -robot-follow option in google...
The dark web also refers to items hidden behind password controls, like your bank account details, which google/bing cannot index. More than often, they are 'dark' for a very good reason.
It doesn't always have to mean nefarious purposes. Do you want Google indexing your bank account, or your online xtube accounts?
That only applies for gay marriage.
Oh so you're the lunatic that acts first thinks later - the next time you swerve watch out for the kid on a bike you did not see the first time you looked.
Google is actually a good way to find scientific papers, their search engine gives better results than the academic databases offer most of the time... the problem is you usually have to pay. The best thing to do alot of the time is google what you want then log in through your schools VPN or go their library to get the paper without paying.
Fravia's Research:
http://www.searchlores.org/
http://www.searchlores.org/indexo.htm
He was a great guy - it's his "fan club" I couldn't stand.
Watch yourself on the links to some of the "discussion boards".
There's some really good knowledge in what he left us.
~hylas
http://www.pcpro.co.uk/features/356254/the-dark-side-of-the-web/print
Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).