One way this has been done successfully is to embed the perl source, encrypted, in a small C program which feeds it to a perl interpretter (statically linked against the binary) at run time.
The C program can then check a licence key, decrypt the embedded blob of Perl in memory, and eval the perl code.
The code to do this isn't that complex (see perlembed)
For this to work on Windows, you may need to replace any XS based Perl librarys you ship with it with pure Perl. Any constants you use for socket based stuff, and sysread/syswrite you will need to find their Windows equivalents - I'm afraid I can't help you much there.
This may be something like what perl2exe does, although they may use the Bytecode interface or the modern equivalent of undump.
Having a select/poll based main loop doesn't stop the dynamic content APIs from being handled simply. It could be done by a pool of FastCGI processes, CGIs, or perhaps a high performance threaded ISAPI runner a la IIS & Zeus.
There aren't that many tasks that must be in the same process as the main connection handling loop - and the state engine. The open spec for ISAPI filters/extensions allows for a lot to of module-like things to be done via a simple, well defined API.
I think the main obstacle to Apache users moving to 2.0 will be that there is too much in-process code and it has been built with the implicit assumption that the process is single threaded and short lived. Testing beyond what was actually in the dominant, stable versions of Apache will not have been a priority. Reviewing and fixing that code for 2.0 is going to be a very nasty job.
I don't think it attempts to filter out all of the DDOS traffic specifically. I think it just tries to ensure that close to the maximum rate of requests can still be served under extraordinary load by quickly binning excess connections. It may even have some prioritisation of who gets binned.
It can also bin requests with particular signatures associated with known attacks.
The effectiveness of the source control system used to develop any non-trivial, multi-developer project is critical.
Creating a good one isn't trivial.
In my opinion, CVS and RCS are far from the best solutions available. CVS behaves like an automation layer on top of the single file version control of RCS, not a change management system for a whole project. It can do most things other systems can do, but you may have to waste effort working around its weaknesses.
I've not used BitKeeper but I have used Perforce, another commercial product which can be used for free on Free software projects, and found the benefits to the development process significant. Making simple things trivial and hard things quite easy helps developers to do the right thing without getting in their way.
Minimising the cost of using the source control system to its fullest effect leads to many improvements throughout the software development process. I'd argue it leads to better, more maintainable code.
As a Linux user and free software advocate I'm glad that Linus is using BitKeeper, rather than CVS. I'd love to see a better free software source control system, but until we've built one I'd rather not sacrifice quality and efficiency in other important projects.
It's very different. WebBrain looks like a directory based system such as Google directory or Yahoo. In response to a query it gave an unstructured list with references to the parent directory nodes.
From what I saw of Kartoo, it partitions the query results. For example I asked for chocolate cake and it gave me the opportunity to refine my search to recipes, cake shops etc as well as example sites of each and nice summaries of those site's content.
I know everyone is cursing Flash, Javascript and Java but client side UIs are a sensible tool in search result presentation if you want to present something more complex than a list.
Whilst Google and others identify and rank results very well, they can't spare the time to do expensive post-processing like partitioning the results. ( It seems that Kartoo can't either can't either. ) 90% of the time what you wanted is on the first page, but sometimes more structure would be useful to dig out the page you actually want. e.g. "get rid of all pages like this one" where there is no obvious "-" term.
A result grouping system on the client side might be more successful. It could get a list of results with summary data from a search or meta-search system (Google's API would be ideal) and could then do any amount of extra work for presentation in a scalable way.
Has anyone had a chance to try the clustering option ? Is this the first full web search tool to display results using dynamic clustering ?
I only had a brief play with the non-clustering version prior to/.ing (using "chocolate cake") and thought it summarised the options very well. The speed wasn't great but not too bad either for the work I think it was doing. If they can work on scalability a bit more perhaps they'll have something.
A while ago I implemented a dynamic clustering search/refine system based on a Xerox PARC idea called "Scatter-Gather". Potentially, it gives good results, but since clustering is naturally an O(n^2) operation, you need to find shortcuts to make it quick enough.
Clustering is finding groups of documents in a collection which interrelate more to each other than to the other documents. e.g. the results for "chocolate cake" would hopefully partition into cake recipes, cake shops, cake mix, diet tips, chocolate appreciation societies etc etc..
From what I've seen, my guess is that with clustering off (the default) it's doing some sort of pseudo-clustering a bit like this:
* Starts with an indexing search engine's results. * Re-indexes these or their summaries, binning the usual stop words like pronouns and weighting by frequency in the sub-collection. * Picks some distinctive/distant vectors, i.e
documents which contain few overlaps with each other.( picking a few samples O(nlogn) ? ) * Labels them with their distinctive terms and displays. * Allows the initial "Google" search to be refined +/- these terms.
With clustering on, I'd guess the main differences to be in the document sampling and query refinement. I'll take another look in a couple of days time. Has anyone tried it out ?
Clustering is expensive, but I still think it's a useful tool for presenting and refining results. This is the best example I've seen so far. The graphical presentation feels fairly natural and intuitive.
It would be a nice option to have on Google to say "cluster my results", when you notice there are distinct classes of result you want to isolate. I'd wait 30s for that.
Slashdot Mirror
One way this has been done successfully is to embed the perl source, encrypted, in a small C program which feeds it to a perl interpretter (statically linked against the binary) at run time.
The C program can then check a licence key, decrypt the embedded blob of Perl in memory, and eval the perl code.
The code to do this isn't that complex (see perlembed)
For this to work on Windows, you may need to replace any XS based Perl librarys you ship with it with pure Perl.
Any constants you use for socket based stuff, and sysread/syswrite you will need to find their Windows equivalents - I'm afraid I can't help you much there.
This may be something like what perl2exe does, although they may use the Bytecode interface or the modern equivalent of undump.
However it is also possible to achieve near ideal
scalability over multiple CPUs with a state engine based model:
Zeus CPU Scalability
Having a select/poll based main loop doesn't stop the dynamic content APIs from being handled simply. It could be done by a pool of FastCGI processes, CGIs, or perhaps a high performance threaded ISAPI runner a la IIS & Zeus.
There aren't that many tasks that must be in the same process as the main connection handling loop - and the state engine. The open spec for ISAPI filters/extensions allows for a lot to of module-like things to be done via a simple, well defined API.
I think the main obstacle to Apache users moving to 2.0 will be that there is too much in-process code and it has been built with the implicit assumption that the process is single threaded and short lived. Testing beyond what was actually in the dominant, stable versions of Apache will not have been a priority. Reviewing and fixing that code for 2.0 is going to be a very nasty job.
http://www.zeus.com/news/articles/020307-001/
I don't think it attempts to filter out all of the DDOS traffic specifically. I think it just tries to ensure that close to the maximum rate of requests can still be served under extraordinary load by quickly binning excess connections. It may even have some prioritisation of who gets binned.
It can also bin requests with particular signatures associated with known attacks.
The effectiveness of the source control system used to develop any non-trivial, multi-developer project is critical.
Creating a good one isn't trivial.
In my opinion, CVS and RCS are far from the best solutions available. CVS behaves like an automation layer on top of the single file version control of RCS, not a change management system for a whole project. It can do most things other systems can do, but you may have to waste effort working around its weaknesses.
I've not used BitKeeper but I have used Perforce, another commercial product which can be used for free on Free software projects, and found the benefits to the development process significant. Making simple things trivial and hard things quite easy helps developers to do the right thing without getting in their way.
Minimising the cost of using the source control system to its fullest effect leads to many improvements throughout the software development process. I'd argue it leads to better, more maintainable code.
As a Linux user and free software advocate I'm glad that Linus is using BitKeeper, rather than CVS. I'd love to see a better free software source control system, but until we've built one I'd rather not sacrifice quality and efficiency in other important projects.
It's very different. WebBrain looks like a directory based system such as Google directory or Yahoo. In response to a query it gave an unstructured list with references to the parent directory nodes.
From what I saw of Kartoo, it partitions the query results. For example I asked for chocolate cake and it gave me the opportunity to refine my search to recipes, cake shops etc as well as example sites of each and nice summaries of those site's content.
I know everyone is cursing Flash, Javascript and Java but client side UIs are a sensible tool in search result presentation if you want to present something more complex than a list.
...
Whilst Google and others identify and rank results very well, they can't spare the time to do expensive post-processing like partitioning the results. ( It seems that Kartoo can't either can't either. ) 90% of the time what you wanted is on the first page, but sometimes more structure would be useful to dig out the page you actually want. e.g. "get rid of all pages like this one" where there is no obvious "-" term.
A result grouping system on the client side might be more successful. It could get a list of results with summary data from a search or meta-search system (Google's API would be ideal) and could then do any amount of extra work for presentation in a scalable way.
I might give this a go
Has anyone had a chance to try the clustering option ?
/.ing (using "chocolate cake")
Is this the first full web search tool to display results using dynamic clustering ?
I only had a brief play with the non-clustering version prior to
and thought it summarised the options very well. The speed wasn't great but not too bad either for the work I think it was doing. If they can work on scalability a bit more perhaps they'll have something.
A while ago I implemented a dynamic clustering search/refine system based on a Xerox PARC idea called "Scatter-Gather". Potentially, it gives good results, but since clustering is naturally
an O(n^2) operation, you need to find shortcuts to make it quick enough.
Clustering is finding groups of documents in a collection which interrelate more to each other than to the other documents. e.g. the results for "chocolate cake" would hopefully partition into cake recipes, cake shops, cake mix, diet tips, chocolate appreciation societies etc etc..
From what I've seen, my guess is that with clustering off (the default) it's doing some sort of pseudo-clustering a bit like this:
* Starts with an indexing search engine's results.
* Re-indexes these or their summaries, binning the usual stop words like pronouns and weighting by frequency in the sub-collection.
* Picks some distinctive/distant vectors, i.e
documents which contain few overlaps with each other.( picking a few samples O(nlogn) ? )
* Labels them with their distinctive terms and displays.
* Allows the initial "Google" search to be refined +/- these terms.
With clustering on, I'd guess the main differences to be in the document sampling and query refinement. I'll take another look in a couple of days time. Has anyone tried it out ?
Clustering is expensive, but I still think it's a useful tool for presenting and refining results. This is the best example I've seen so far. The graphical presentation feels fairly natural and intuitive.
It would be a nice option to have on Google to say "cluster my results", when you notice there are distinct classes of result you want to isolate. I'd wait 30s for that.
Colin
Just to add to the catalogue: worked for me with Galeon 1.2.0 & Flash 5.