For web cache benchmarking they are the gold standard and everyone in the industry (even the much maligned IBM) shows up to the IRcache events and use Polygraph results in their ad copy.
If you do it right, open source benchmarks can become the standard in an industry...in fact, I think if you do it right, they will become the standard in an industry. Good companies want a level playing field. Bad companies will be weeded out eventually...
Forget all the scientology nonsense (we all know the story of LRH telling Heinlein "If you want to make a lot of money, you have to start you're own religion").
The book was GREAT FUN! When I was in the 5th grade, I read this book and was introduced to a whole new world (Science Fiction)...Sure it's a total space opera, but boy, what a fabulous space opera.
Without this book I might have discovered Heinlein, or Asimov, or all of the hundreds of others out there. I for one will be going to see the movie (it cannot be worse than Starship Troopers and I gave my 7 bucks for that one!).
I'm not going because I'm a big fan of the Scientology...I couldn't care less what hobbies LRH happened to have (hey, starting a cult, collecting stamps, surfing for internet porn...what difference does it make? He never hurt anyone...all members joined voluntarily).
Nice try. Look into the squid source and FAQ. Kernel threads are required for an async io compile of squid. I'm not knocking FreeBSD by any means. I've had to learn to love it while doing these benchmarks (the testing environment only runs properly under BSD). It's a great OS.
But squid currently performs better under Linux, when proper tweaks are made. If you can make a BSD box do 110 reqs/sec from a K6 and 2 IDE hard disks with a stable version of squid, then I'd love to hear about it. It simply isn't going to happen currently. It's not even easy to do with Linux.
When the new squid filesystems come online, all the rules may change. But currently linux is the speed king for Squid.
I was the tech at the bakeoff for Swell's entry, and after having met everyone during the first week and taking a look at the entries, I have to say the vendors at this bake-off were pretty much all above average.
To find the below average vendors you'll need to look to the folks who didn't show up this time.
While I have an obvious interest in promoting the Swell entry (which did quite well...but not as well as we expected due to some unresolved bugs), I spent a lot of time talking to the other vendors techs. There were some very smart people pushing very good products at this bake-off. I wouldn't hesitate to recommend (for customers that need more than our boxes provide and can spend the extra dough, of course) many of the products tested.
I know it sounds rather flowery to say that "All the girls in the pageant were very pretty", but in this case, I think the bake-offs are really separating the chafe from the wheat. Look at previous bakeoff numbers and prices and compare to this time around. Keeping in mind that this bake-offs workload was MUCH harder than previous workloads (the Polymix-1 or Datacomm-1 workloads found in previous comparisons), even so the price performance has improved markedly from all vendors. And the price/performance also-rans from previous events just didn't show up this time.
That's why the polygraph guys deserve such praise. They allow cache users to really know what they are buying. And the companies that don't show up or don't provide a good value just won't sell as many boxes. (And I strongly recommend against buying an untested cache product...there are some real stinkers out there and you don't always get what you pay for.)
I've seen several comments about Squid not being the best, or what have you. Squid made an admirable showing at the bakeoff. If you only look at price/performance you are not seeing the whole picture, or even most of it.
Squid showed perfect cacheability (why buy a cache except to cache?), whereas some others in it's price range (except the Swell box also running squid) displayed much lower cacheability. Response times from a lot of boxes were not so good either, while squid's was excellent (the other reason to cache...browsing speed). When you see a box with long response times and low cache hit rate, you are looking at a box that was being pushed WAY too hard. You would not run a cache with 30 or 40% DHR and mean response times of 2 seconds...ideally, you run it such that cacheability is near perfect and response times are very very fast. Squid did that. Microbits didn't.
The Squid team have done a great job with Squid, and it gets better every time around. Even compared to the ICS products (many of which are very very fast these days...but you pay the price for them...ICS on low end boxes suffers a bit), Squid didn't do so bad at all.
Anyway, if you'd like to see some more Squid numbers, we've got a $2139 squid box in the lab doing 110 reqs/sec from dual IDE drives, whereas the Squid team got 160 from a $4k box with 6 SCSI 10k drives. We will be posting pretty specific specs for it sometime in the future so that others who want to roll their own can do so (it takes a lot of work). Some of our recent benchmarks (using Bake-off rules and benches) are posted on the Swell Technology web page. Currently, the posted benches are for a run at 100 reqs/sec. The 110 run will be posted sometime soon.
Those interested in caching should check out the squid devel list lately. Discussion has centered on a couple of new filesystem ideas that should improve squid performance markedly. Fascinating stuff. I suspect the ICS guys will be a little more worried come next bake-off.
BSD is great for Squid because of the excellent stack and reliability (and it's the platform of choice for Duane and most other leads developers), but Linux is better if you want performance. Async IO is only available under Linux and Solaris and it makes a HUGE difference (look at the Swell results at the bakeoff without threads and our results more recently with threads--77 reqs/sec at the bakeoff, 110 running in our labs now--from a $2139 dual IDE drive box! Performance is on the Linux side).
Recent benchmarks of a very tweaked linux/squid box are posted at the swell technology website doing 100 req/sec.
I think it's more fair to look at the Swell entry (also using Squid, except on Linux). It's price is very similar, though it has much beefier hardware.
You should further look a little deeper into the results. Microbits box was only caching about 44% of web traffic and getting rather slow response times. So while they got 120 reqs/sec, no sysadmin in their right mind would push that box that hard. To compare apples to apples with the Squid entry or the Swell entry (both had nearly ideal cacheability and excellent response times) you should think of the Microbits box as being more along the lines of 95 or 100 reqs/sec.
To see Squid results in more favorable light, check out the more recent results on the Swell web page:
Our test box at the bake off was having fits using async io...so we disabled it in order to get a clean run. However, performance suffers markedly without it. Those async issues have been resolved...Our boxes are running in our labs at 110 reqs/sec right now (we have a 100 reqs/sec run benchmark online...you can note that response of squid is still excellent at that load).
Anyway, given the proper tweaks, Squid can really scream on a low priced box. (Our $2139 unit is the one included in the bakeoff and our more recent benchmarks.)
I/O driven means that is relies on a select or poll loop to wait for file descriptors to be ready for processing.
The async io compile option allows what you've described. And it does wonders for performance. The Swell results from the bakeoff compared to our current results published on our page are a good example of this (we rated 77 req/sec at the bakeoff...we're testing 110 req/sec on less hardware in the lab right now). The difference was that we were experiencing problems (system lockups) with async io compiles at the bakeoff, so to simply get a solid run in we ran without async...performance was less than ideal because of it.
Those issues have been resolved, and the async io builds are significantly faster.
This adversely effects web caching technologies as well. Any dynamic content is uncacheable and unsearchable due to the inability to know if the content is specific to a query or simply a love of the concept of "dynamic content" on the part of the page designer.
It's clear that many aspects of a webpage that could be pregenerated every time the information is updated are not being done that way. Slashdot is a prime example. Presumably, thousands of people visit Slashdot anonymously everyday. Even though they see the same content, the page is regenerated with unsearchable/uncacheable content. Shouldn't it be a simple matter to have a script choose between a dynamic page for logged in users and a constantly up to date pregenerated page for anonymous users? Saving CPU cycles for the servers, allowing indexing by search engines, and speeding up accesses for users behind a cacheing proxy. Sounds like only good things to me.
Obviously this won't solve all of the problems, but many websites front pages are the same for every user. Wouldn't it make sense to pregenerate it as static content? This could be taken much further by news sites that provide the same story content to every user, but use a database frontend for simplicity anyway. This doesn't preclude use of a backend database for information storage and organization, but it does impose quite a lot of complexity in the implementation of a system to index all of the pages as they become available and make them into static, numbered pages.
I tend to fall into the category of folks who believe that site designers should be a little more aware of the outside world and making their content accessible via every possible means. I don't think it makes sense to prevent search engines from finding ones content. If you've put it up, you want people to find it. Why turn down that extra banner display simply because someone doesn't check your headlines and instead searches Google or Alatvista?
I'm sure there are other issues involved and I'm glad this was brought up...I've been trying to figure out solutions to these problems myself while implementing a company web page with our web designer. Being a cache server company, we've got to make sure our own pages are completely cacheable whenever humanly possible. Not to mention that when someone does a search on any engine we want our URL to come up if we've got something to say on the subject of the search. It just makes sense to be as openly accessible as possible.
So, is this a problem that should be addressed mainly by the search engines, or should web designers be thinking ahead to such concerns when they are building a site with dynamic content?
I asked this same question of Ask Slashdot about 6 months ago when I found an old fixed freq. monitor...Never got posted, so I dug it all up myself. I've documented my whole experience with it.
Has a bit of a howto on the subject. It also includes documentation that I recieved from a video card hacker on how to squeeze normal text modes out of some monitor/card combinations. It's a scary, very cool hardware hack that I haven't delved into yet. It involves flashing your video card a modified BIOS.
Last night I just setup the fb console device in the 2.2 kernels to allow all of my virtual consoles to display. I'll be documenting that as well, sometime in the future. A legible 154x98 text mode is a rather amusing sight to behold.
It's not all that difficult (X was working within 30 minutes for me) and it's well worth it, as these old workstation monitors were made like tanks.
Slashdot Mirror
"IRCache Polygraph Homepage"
For web cache benchmarking they are the gold standard and everyone in the industry (even the much maligned IBM) shows up to the IRcache events and use Polygraph results in their ad copy.
If you do it right, open source benchmarks can become the standard in an industry...in fact, I think if you do it right, they will become the standard in an industry. Good companies want a level playing field. Bad companies will be weeded out eventually...
And that's all I have to say about that.
The book was GREAT FUN! When I was in the 5th grade, I read this book and was introduced to a whole new world (Science Fiction)...Sure it's a total space opera, but boy, what a fabulous space opera.
Without this book I might have discovered Heinlein, or Asimov, or all of the hundreds of others out there. I for one will be going to see the movie (it cannot be worse than Starship Troopers and I gave my 7 bucks for that one!).
I'm not going because I'm a big fan of the Scientology...I couldn't care less what hobbies LRH happened to have (hey, starting a cult, collecting stamps, surfing for internet porn...what difference does it make? He never hurt anyone...all members joined voluntarily).
And that's all I have to say about that.
But squid currently performs better under Linux, when proper tweaks are made. If you can make a BSD box do 110 reqs/sec from a K6 and 2 IDE hard disks with a stable version of squid, then I'd love to hear about it. It simply isn't going to happen currently. It's not even easy to do with Linux.
When the new squid filesystems come online, all the rules may change. But currently linux is the speed king for Squid.
To find the below average vendors you'll need to look to the folks who didn't show up this time.
While I have an obvious interest in promoting the Swell entry (which did quite well...but not as well as we expected due to some unresolved bugs), I spent a lot of time talking to the other vendors techs. There were some very smart people pushing very good products at this bake-off. I wouldn't hesitate to recommend (for customers that need more than our boxes provide and can spend the extra dough, of course) many of the products tested.
I know it sounds rather flowery to say that "All the girls in the pageant were very pretty", but in this case, I think the bake-offs are really separating the chafe from the wheat. Look at previous bakeoff numbers and prices and compare to this time around. Keeping in mind that this bake-offs workload was MUCH harder than previous workloads (the Polymix-1 or Datacomm-1 workloads found in previous comparisons), even so the price performance has improved markedly from all vendors. And the price/performance also-rans from previous events just didn't show up this time.
That's why the polygraph guys deserve such praise. They allow cache users to really know what they are buying. And the companies that don't show up or don't provide a good value just won't sell as many boxes. (And I strongly recommend against buying an untested cache product...there are some real stinkers out there and you don't always get what you pay for.)
Squid showed perfect cacheability (why buy a cache except to cache?), whereas some others in it's price range (except the Swell box also running squid) displayed much lower cacheability. Response times from a lot of boxes were not so good either, while squid's was excellent (the other reason to cache...browsing speed). When you see a box with long response times and low cache hit rate, you are looking at a box that was being pushed WAY too hard. You would not run a cache with 30 or 40% DHR and mean response times of 2 seconds...ideally, you run it such that cacheability is near perfect and response times are very very fast. Squid did that. Microbits didn't.
The Squid team have done a great job with Squid, and it gets better every time around. Even compared to the ICS products (many of which are very very fast these days...but you pay the price for them...ICS on low end boxes suffers a bit), Squid didn't do so bad at all.
Anyway, if you'd like to see some more Squid numbers, we've got a $2139 squid box in the lab doing 110 reqs/sec from dual IDE drives, whereas the Squid team got 160 from a $4k box with 6 SCSI 10k drives. We will be posting pretty specific specs for it sometime in the future so that others who want to roll their own can do so (it takes a lot of work). Some of our recent benchmarks (using Bake-off rules and benches) are posted on the Swell Technology web page. Currently, the posted benches are for a run at 100 reqs/sec. The 110 run will be posted sometime soon.
Those interested in caching should check out the squid devel list lately. Discussion has centered on a couple of new filesystem ideas that should improve squid performance markedly. Fascinating stuff. I suspect the ICS guys will be a little more worried come next bake-off.
BSD is great for Squid because of the excellent stack and reliability (and it's the platform of choice for Duane and most other leads developers), but Linux is better if you want performance. Async IO is only available under Linux and Solaris and it makes a HUGE difference (look at the Swell results at the bakeoff without threads and our results more recently with threads--77 reqs/sec at the bakeoff, 110 running in our labs now--from a $2139 dual IDE drive box! Performance is on the Linux side).
Recent benchmarks of a very tweaked linux/squid box are posted at the swell technology website doing 100 req/sec.
You should further look a little deeper into the results. Microbits box was only caching about 44% of web traffic and getting rather slow response times. So while they got 120 reqs/sec, no sysadmin in their right mind would push that box that hard. To compare apples to apples with the Squid entry or the Swell entry (both had nearly ideal cacheability and excellent response times) you should think of the Microbits box as being more along the lines of 95 or 100 reqs/sec.
To see Squid results in more favorable light, check out the more recent results on the Swell web page:
http://www.swelltech.com
Our test box at the bake off was having fits using async io...so we disabled it in order to get a clean run. However, performance suffers markedly without it. Those async issues have been resolved...Our boxes are running in our labs at 110 reqs/sec right now (we have a 100 reqs/sec run benchmark online...you can note that response of squid is still excellent at that load).
Anyway, given the proper tweaks, Squid can really scream on a low priced box. (Our $2139 unit is the one included in the bakeoff and our more recent benchmarks.)
The async io compile option allows what you've described. And it does wonders for performance. The Swell results from the bakeoff compared to our current results published on our page are a good example of this (we rated 77 req/sec at the bakeoff...we're testing 110 req/sec on less hardware in the lab right now). The difference was that we were experiencing problems (system lockups) with async io compiles at the bakeoff, so to simply get a solid run in we ran without async...performance was less than ideal because of it.
Those issues have been resolved, and the async io builds are significantly faster.
Beautiful output, works nicely with Sane, up to legal size paper, and the included SCSI card works fine. It's quite fast and affordable as well.
The driver is supposedly still alpha but seems pretty close to stable to me (the preview window has issues with some window managers).
The only unsupported bit is the 'lightlid' which provides backlighting for scanning negatives and slides. Super cool, but still needs work in sane.
It's clear that many aspects of a webpage that could be pregenerated every time the information is updated are not being done that way. Slashdot is a prime example. Presumably, thousands of people visit Slashdot anonymously everyday. Even though they see the same content, the page is regenerated with unsearchable/uncacheable content. Shouldn't it be a simple matter to have a script choose between a dynamic page for logged in users and a constantly up to date pregenerated page for anonymous users? Saving CPU cycles for the servers, allowing indexing by search engines, and speeding up accesses for users behind a cacheing proxy. Sounds like only good things to me.
Obviously this won't solve all of the problems, but many websites front pages are the same for every user. Wouldn't it make sense to pregenerate it as static content? This could be taken much further by news sites that provide the same story content to every user, but use a database frontend for simplicity anyway. This doesn't preclude use of a backend database for information storage and organization, but it does impose quite a lot of complexity in the implementation of a system to index all of the pages as they become available and make them into static, numbered pages.
I tend to fall into the category of folks who believe that site designers should be a little more aware of the outside world and making their content accessible via every possible means. I don't think it makes sense to prevent search engines from finding ones content. If you've put it up, you want people to find it. Why turn down that extra banner display simply because someone doesn't check your headlines and instead searches Google or Alatvista?
I'm sure there are other issues involved and I'm glad this was brought up...I've been trying to figure out solutions to these problems myself while implementing a company web page with our web designer. Being a cache server company, we've got to make sure our own pages are completely cacheable whenever humanly possible. Not to mention that when someone does a search on any engine we want our URL to come up if we've got something to say on the subject of the search. It just makes sense to be as openly accessible as possible.
So, is this a problem that should be addressed mainly by the search engines, or should web designers be thinking ahead to such concerns when they are building a site with dynamic content?
Joe, Swell Technology
My home at:
http://freeweb.pdq.net/jcooper
Has a bit of a howto on the subject. It also includes documentation that I recieved from a video card hacker on how to squeeze normal text modes out of some monitor/card combinations. It's a scary, very cool hardware hack that I haven't delved into yet. It involves flashing your video card a modified BIOS.
Last night I just setup the fb console device in the 2.2 kernels to allow all of my virtual consoles to display. I'll be documenting that as well, sometime in the future. A legible 154x98 text mode is a rather amusing sight to behold.
It's not all that difficult (X was working within 30 minutes for me) and it's well worth it, as these old workstation monitors were made like tanks.