Slashdot Mirror
Making Freenet Find Stuff Faster
Sanity writes "Many probably saw the recent announcement of Freenet 0.5.2. This release represented a vast amount of work - primarily in reducing Freenet's CPU and memory requirements. However, streamlining Freenet's current functionality isn't all we've been working on. I just finished an article that describes the most fundamental improvement to Freenet's core algorithm since its original design over three years ago, it is called "Next Generation Routing" and has the potential to dramatically increase the speed with which Freenet retrieves information. It could even make Freenet faster than the World Wide Web in many circumstances, all without compromizing anonymity and while remaining immune to the /. effect."
Freenet is an awesome idea, and very rapidly becoming one that is neccesary to ensure your protection. Although it is a double edged sword (It can help both good, and bad people), I think it's one that is neccesary. And, if it becomes speedier than the web at large, it'd be just freaking awesome. Now, no one needs to fear censorship, nor do they need to fear the government shoving them into a database.
Now if only I could get it running on my Mac OS X box...
"immune to the /. effect."
:-)
If this isn't a challange I don't know what is
-traskjd
My blog [.net, rants, general IT]
> ... and while remaining immune to the /. effect
Said the author of the slashdotted article.
Those using browsers that support the "mng" animation format (such as Mozilla) can see an animation of a node's datastore specializing over time here.
That's not true anymore, communists Mozilla maintainers removed mng support to save a 'whopping' 100k download.
Other than the fact that most infringers do not like to use Freenet because it is too clunky for them to get their quick hit of free music, it is no more of a threat than any of the popular P2P services.
Translation: "Oh Lord, I hope Freenet is inherently unable to have robust search functions, because if it ever develops these, we're hosed. But in the meantime, we can dismiss this software as being a big POS."
Now, less than two weeks after the interview, it seems the one aspect of Freenet that Oppenheim wanted to write off at is on the brink of being fixed.
I'm generally "Interesting," "Insightful," and even "Funny" here. What the hell happens to me at parties?
Instructions for windows and linux and linux compatables.
Windows : Right click the rabbit icon in your system tray, then click upate to latest snapshot build.
Linux : run update.sh in the freenet directory.
Nero-burning ROM for Linux!
You know what'd be really impressive? Finding a way to make FreeNet slower. It'd be so slow you could make a Beowulf cluster of FreeNet nodes and use it as a time machine. Personally, I'd use it to go back to Ian Clarke's dorm room and convince him to get drunk and high rather than wasting his life making a P2P system that will be useful around the same time we have to start worrying about being censored by the United Federation of Planets. But that's just me.
What I find interesting about this algorithm is that it is applied individually by each node; there seems to be no need for nodes to share data over some complicated protocol as in many distributed systems. Yet (I think we can believe Clarke) this change improves response time through the system as a whole. It's a validation of the basic Freenet model of systems acting alone but providing a service greater than the sum of its parts.
has anyone ever tried peekabooty, esp. under wine? The reflections on open source development the developer(s) feature on their website sound kinda depressed..but then again, the honesty factor speaks for them. Are there any deep flaws in the idea? I personally like the simplicity of their design, but since I'm not a design guru, I may be utterly wrong.
...at least not keyword searching as you find in Google and Kazaa. When they refer to searching they mean given a key (a very large number), finding the corresponding data.
Immunity to ignorant masses of /. users it is not.
/. crowd of joining, and here is the etiquet/advice I have.
:)
/1/ /2/ /3/ and link to images from the future. If the image loads, you know there is a more recent revision. date based must be activated every time interval, or they die. Be very careful with these.
/. effect immunity, they mean linking to a site will only make it stronger. Everyone on /. joining freenet is just going to slow it down, because basically, you are creating a great suction on the net without any data to give back. Even worse, when you quit off of freenet, everyone will be looking for you from their cache and not finding you. This is going to cause the most problems, but surely not everyone on /. are going to quit on the same day. ;)
I was in the first
Things to do if you plan on playing with freenet:
1. Set it up properly.
1a Set your IP in the config file, read the site for details, but it's freenet.ini
1b Try to use DynDNS if you have a dynamic IP
2c Leave it up 24/7 for a few days before you judge speed. You need to let the blood circulate
2. Install a proper version of Java. I recommend the 1.4.2 beta. IBM may work better, I haven't tried.
3. Fix your browser.
3a Your browser will crash on some sites (even Mozilla not Opera) because of a GIF bug.. patch it.
3b Set your number of simultaneous connections up a lot. You request a file from your local store, then it downloads it. You need to request as many in parallel as possible.
Now, on to advice.
Get Frost! Frost is like the news groups of the freenet. It's a great place to read interesting ideas.
If you want to make a site, check out Fish tools, Fuqid and FIW.
Be aware that there are 3 different kinds of sites, and two modes of getting information
3 types include interval based, revision, and static. Static sites are one time shots. Revisions you create directories like
There are SSK and CHK linking methods, which I still don't know a whole lot about, but maybe someone will reply and explain them.
By
Get IIP, so you can realtime chat with people that run some sites on freenet. #freenet is dedicated to freenet chat and issues.
Have fun!
(Posting anonymously in respect of the freenet principals.)
Makes you wonder if Freenet gained popularity over the web whether all "official" transactions would be web-based, leaving Freenet to misc. web sites that are completely information/communication based. The reason I wonder is because if someone gets their login/password stolen from some random service on Freenet which they invested mucho time in, how will anyone else know the difference? That would really irk me.. (Yes, I know the web is vulnerable to this as well, but at least it requires a user have an IP address -- whether or not it's actually legit.)
This is incorrect. For networking related stuff, Java is efficient. It will use some more memory, and it will use (a bit more) CPU power but there are many reasons to go with java for such a project:
- easier language to pick up / understand (this is a collective effort)
- little to no chance of buffer overruns, making the node much safer against attacks)
- runs on Linux, Apple, Sun, Windows, FreeBSD without - any - porting
- java was more or less created with projects like these in mind, so most functionality will be readily available in the default libraries
Nowadays CPU and memory are commodities that can easily be come by. I see it taking about 32 MB right now, but that is out of a single 512 MB pool that can be upgraded to 1 GB for virtually free. My processor usage is max 25%, but note that the freenet guys set the priority to low themselves.
Java means a shift to better programming, with better runtime information and safer programs. This will take CPU and memory, but this is an offer you should consider very well.
This same discussion went on between assembler and C programmers. Look at it now. I think the progress of object oriented, garbage collecting, more secure platforms are as important as that paradigm shift.
Warper
Seriously. It is a bit ironic that the Freenet project doesn't run on a free system like Debian GNU/Linux. So there is an effort underway to Free Freenet! See the developer mailinglist archive. Please donate (Matthew Toseland - Toad - is the "Official Codemonkey" of the Freenet Project).
They thought it would be cool to design a censorproof network. They weren't interested in supporting what was already in development, namely Freenet, after all - where is the publicity in being part of someone else's project?
The only problem was that they dramatically underestimated the difficulty of pulling it off - the result? Peekabooty was, is, and probably always will be, vaporware. The design they do have is a primitive HTTP proxy network last time I looked, and it doesn't solve any of the difficult problems of circumventing censorship (just ask them how the poor little Chinese dissidents are supposed to find their HTTP proxies).
Amuzing, after draining the concept of a censor-proof network for all it was worth (without actually building one) - they then did their best to drain publicity from their failure to build it!
Freenet answers those questions, and has done so since its original design in 1999.
the nice thing about the current ng routing scheme is that there's plenty of room for research on how to tune it even further.
/me wanders if embedding fortran in java makes sense ;))
Note: if you haven't read the article, this won't make much sense to you.
For one, the number of reference points doesn't have to be fixed; if/when memory and cpu power allows us, we could have variable number of reference points per node. This opens the door to other decisions, such as whether we encourage clustering reference points. If yes, we add new ref points closer to others. If not, we remove a ref point the density within some keyspace interval gets too big. Another option is to add a new ref point whenever the n previous estimates turn out to be more than x% correct, and remove one if otherwise.
Another direction to go into is curve fitting. If cpu power allows us, we could use various techniques of polynomial or Fourrier interpolation within the existing reference points to draw more accurate curve of time vs. keyspace.
Don't go silently into that peaceful night
It takes some time to build up information on how to get around. It gets faster, be patient
Freenet is about Freedom of Communication, not Free Software. Just because there is significant overlap between those that advocate each - does not mean that Freenet should spend its resources advancing the Free Software/Open Source agenda at the expense of its own.
Freenet is NOT immune to the /. effect currently. Every time /. runs a Freenet related story, loads of new users seem to get on the Freenet and it just collapses, meaning response times go way up and many freesites become unreachable. I'm sure NGRouting will take care of this, but it's not honest to say it will help Freenet "remain immune" to the /. effect, because it's not immune.
"Oppression and harassment is a small price to pay to live in the land of the free." -- Montgomery Burns.
Uh, your really off your mark here. The Freenet web interface thingy comes with it's own mini webserver and the functionality to turn any non-transient node into a freenet distribution center. From the Freenet web interface, there's a link called Spread Freenet. (Link only works if you have Freenet installed and running.)
Even if the main Freenet site got taken down, things would still be just peachy...
While we're at it, what's this about the Bittorent mainpage going down? I know that a few popular tracker sites went down, but I've never heard of the main BitTorrent site going down. Click the link; it's up right now.
Moderators: How the hell did the parent get modded +2 Insightful?
No, you are referring to the startup time of the VM. Once started, the memory pages will have settled and the response gets better and better. The same thing happens when you use the menu's the first time. Once the classes have loaded, the program has fine responsiveness. Actually, you can preload classes with Java, but not too many developers use that particular feature (it will add to the startup time anyway).
IMHO, the Java VM should be loaded at startup, and a single VM should be used to launch multiple applications. When used like that (together with an efficent GUI startup process) much of the gripe against Java applications should be gone. Obviously the firewalling between programs should be maintained. Alas, this is not currently so.
To come back to freenet: it doesn't incorporate a GUI in normal use (using the web interface is not the same as launching a Swing application) and for networking speed: the speed of the connection will be the bottle neck, not the Java application.
It must be said that the current implementation will scare away programmers that are looking towards efficency. For most programs you should n't though. Look at the architecture before trying to get something more efficient by changing languages.
ps. for an ample showcase of efficiency, try Eclipse from IBM. Check the features first before posting though.
FreeNet will have problems for the forseeable future because the average joe can't easily install it and make it work.
Who will take FreeNet to the masses?
In other words, who will make a simple, usable client/server program that works on FreeNet? (Think Napster/KaZaA/Gnucleus)
Will it be KaZaA? BearShare? Will it be some Open Source project?
How long until somebody with the right skill set takes this to the "next level" so that it's actually usable to people other than geeks?
I have no problem with your religion until you decide it's reason to deprive others of the truth.
A "rights" holder knows the freenet key of certain material. Can the holder not hust write a script hop onto Freenet, request that key (and only that key), and fire off C&Ds to all the ISPs whose allocations include the addresses that respond? Seems simple enough--even with blinding of requests, the intellectual "property" holder can still point to the nodes that respond as having distributed the material--just as the exit server from Mixmaster (or freedom.net, before it became a casualty of 9/11 hysteria), etc. is vulnerable to legal attacks.
This might be able to be foiled with some kind of chaffing in which nodes respond even if they don't have a piece of the data in question, but that would introduce more inefficiency.
In particular, those who are "willfully blind" to infringement losing safe harbor provisions, I don't see how Freenet will survive as a means of propagating "questionable" material. And since that's it's raison d'être, then it probably won't survive at all in the U.S.
CEE5210S The signal SIGHUP was received.
When I first tried Freenet a year or so ago it defaulted to be a transient node.
I noticed the lastest versions default to permanent node and the Windows version also puts itself in your startup folder.
I don't think a few hundred or thousand transient nodes coming onto and off of Freenet would hurt it, but I think permanent nodes frequently hopping on and off will slow it down. I wonder why they changed the default to permanent?
If I understand correctly, a transient node doesn't store data, respond to data requests from other nodes or get put in the routing table, while a permanent node does. A full-time permanent node will make your local browsing faster as well has help out Freenet, but a sporadically on permanent node would cause delays I suspect.
The reason that Freenet is supposedly free from the Slashdot effect is because a greater demand for a freesite naturally causes it to be available from more nodes. The supply scales to demand.
freenet still isint there yet, but feel free to tell us when.
This is just the attitude that is delaying the adoption of many new technologies (IPv6, for one). "Early adopter, what's that? Just tell me when it's done!" How do you expect it to "get there" if no one uses it? Take a chance. You might be pleasantly surprised.
Of course, it might.
More likely, Congress will order the FBI to use Carnivore (or whatever it is called now) to track people downloading a particular file on Freenet, and to try and find out who they are. I don't remember how Freenet works, or how Carnivore works, but I'm sure with total control of the router infrastructure you could figure out who was downloading what, eventually..... although, every control message for freenet is encrypted, huh? Well, anyway they'd try.
Then, the RIAA will demand that congress give them the power to open up carnivore boxes and track down "pirates" without judicial oversight.
Our legislators have such a poor idea of how freenet works (worse even than mine,) that I don't think they *can* write a law against it. A law against software that enables two remote computers to connect to each other without both of them knowing who the other is?
The good and new comes from no quarter where it is looked for, and is always something different from what is expected.
freenet is a *protocol*, not a client (though they do ship the http proxy client with the main distro). just like the http protocol doesnt have any search functionality built into it, neither does freenet. you can, and people do in fact use regular old web spiders to create searchable indexes of freenet.
tasty electronic music vittles
Take a chance. You might be pleasantly surprised.
I'd love to. You show me where I can download the features that they're announcing, and I'll try it. That's the point: This stuff doesn't exist in Freenet yet; they're talking about their wishlist. This is news?
The text in the FAQ is mine, I don't think Ian is claiming freenet will get better throughput/latency for browsing random websites, however freenet can be faster for downloads from websites, websites with flash animations or big applets, those kinds of things. (At current the anonymity filter (a piece of code that filters potentially anonymity-compromising parts from freenet websites) will remove all plugins and applets, but we're talking middle to long-term future here).
:-))
Basicly, freenet latency is bad, freenet throughput is good. (and freenet reliability is different
Data in Freenet is split into small pieves, and those pieces are hashesd to make the keys. When you specialize, you are more likely to store keys in some short range of prefixes than you are for any keys outside your specialization.
No type of content is more likely to have keys starting with a certain prefix than any other. So you can't "specialize" in child porn, or any other content <i>type</i>.
Yes this is right, specialization occurs via key names, not content.
This is good, since keys are a random sampling of content, so if a node goes down then no specific type of content is lost. (Not putting all your eggs in one basket idea.)
I've been reading about Freenet, and I'm trying to imagine how a potent search engine could be implemented on top of Freenet. Ideally it'd be great to use meta tags and such to index pages, but then how do you find the files if you do not know their keys in the first place?
Yes, I have heard about Frost. As far as I understand, it's some sort of anonymous newsgroup. I guess a search engine could harvest the keys posted on Frost, and index them after retrieving and analysing the content and possibly the meta tags. But then the question becomes: how do you host such a search engine anonymously? Aren't you liable/vulnerable if your search engine is known to help you retrieve questionable content? Can't Frost be attacked ultimately for that same reason? Or is it distributed/anonymous? Am I missing something? Should I RTFA?
"In our tactical decisions, we are operating contrary to our strategic interest."
Yeah, we here at the RIAA would very much like you to keep using Kazaa. It's fast for you, and we get to track you down. You see, it's a win-win situation!
If you download porn, the spyware programmed into Freenet will foward your IP to the RIAA, FBI, NSA, and then post it to a few hacking/warez newsgroups and forums.
Freenet is NOT a pornster program.
If you use Linux, please help development of Autopac
1. Freenet does work, its slow, but it works, I run it on dialup, all you people with bband stop moaning.
2. Whatever connection you use give it time to integrate into the network.
3.Stuff you may not agree with can and probably will be stored on your node.
4. You cant be done for 3. Unless certain western goverments get really upset with freenet users.
5.Download it. Run it. Leave it as long as you can. Repeat. Eventually it will work ok.
6.Remember its worth it. Support this project you might need it.
1. Its theorethic. The original freenet concept contained simulation to show that it should work. I am missing this here.
/no hops it took because it leads to security problems.
2. It does not fit really well in the freenet sources. In the current freenet implementation the network layer and routing layer are split. Unless you develop it yourself this will not be implemented in freenet (soon).
DNF: estimate if they are legimate by estimating their time. This does not work on a saturated network. And freenet is always (by design i think ) full.
There are some asumptions here that do not work. Also there will be things in freenet that will try to hide the location
Inherited Knowledge:
Make nodes learn faste by assuming some kind (vague!) of trust between nodes. read: create trust by an estabished node and new (unreliable?) node. This is against the freenet paradigma and creates all kinds of security problems. This kind of thing should not be implemented in freenet where the 1st priority is security.
The only positive thing this article is suggesting is to time the data and so optimize the flow of messsages according to the internet structure. In freenet this is an implementation problem.
There were more of these kind of suggestions on the freenet tech mailing list. I unsubscribed it (too much spam, too much interesting ideas from people who had no clue)
If you write such articles please investigate other p2p solution as well! (gnet/gnunet india network and many others.)
I've said this before!
The only problem is that there's no one-click tool to mirror a website into Freenet, yet. Freenet's gateway has an anonymity filter which prohibits out-of-freenet links, and it also disallows a lot of things. If someone wanted to write a simple tool to clean up a site and hack the links to work in Freenet, it would make this a lot easier.
By the way, using the http://127.0.0.1:8888/KEY@whatever style links is discouraged, because not everyone's freenet node is localhost, and not everyone runs it on port 8888! The preferred format is freenet:KEY@whatever which can then be handled appropriately by your browser.
You're part of the problem! The reason Freenet sucks for a little while after each release is that there's a huge influx of empty datastores joining the network. The network bounces back pretty quickly, as data gets passed around and as routing tables hone themselves, the network gets a lot better.
Then a day or two later, you and 90% of the other slashdotters drop off, and leave holes in everyone's routing tables. All the contribution that your nodes were just starting to make, gets undone. All the copies of content that got replicated into your datastores vanish. All the routing optimizations that were just sorting themselves out get broken again.
Tourists hurt the network. If you're judging Freenet based on it's performance the day after a slashdotting, you're not getting a full or fair picture. Come back and stay a while! Let your node run for a week and I think you'll be impressed.
When they say Freenet is slashdot-resistant, they refer to content within the network. Any piece of data, be it a single file or a whole freesite, will simply propagate more as more people request it. The network itself definitely labors a bit as empty datastores dillute it. The best way to improve Freenet's performance is to encourage those tourists to stick around, so they and the network will benefit the most.
Lets see if we can use them all...
Freenet is now being 'fixed' like a leaky faucet is fixed.
The RIAA wants the digitial audio/video market 'fixed' like a crooked horse race is fixed.
With the new Freenet the RIAA is about to be 'fixed' like your dog at the vet's is fixed.
I think that about covers it.
It's Christmas everyday with BitTorrent.
One aspect of Freenet is that the content reflects what the community puts out there. If you want to see stuff other than porn, put in up on the network. In fact, it will help to put even more stuff as long as it is of value to other people.
"If you make it, they will come" is all to important with Freenet.
Another point to make: If you view the porn and try to download it, you are also spreading this content to other nodes. If you don't want it on the network, don't view it or use it. Indeed, Freenet is very democratic in this sense, and you "vote" on each key each time you use it. These votes are actually used to determine if data is kept or discarded once the data store if filled, and old seldom used data is dumped routinely.
> I'm going to wait for NGrouting.
N Grouting? Does that mean I can seal my bathtub remotely? Cool feature!