Slashdot Mirror
WCArchive sets new Record
dcs
writes "The hardware upgrade for wcarchive came
not a single second too soon. In it's first full day of
operation with the new hardware, a new record was set...
969 gigabytes of traffic was generated, thanks
mainly to the recent release of RedHat 6.0. I'm
looking forward to the first terabyte in day mark, but it
seems an upgrade on network capacity is due before that
can happen. "
Hey, I find it kinda fascinating too to know what such large sites run. Someone mentioned in a previous thread though that download.com runs Netscape Communications/Solaris 1.12. I know that Yahoo! uses FreeBSD (an old copy of the FreeBSD newsletter had an article about Yahoo! and the different OSes they tried when they were starting up). Dejanews and Slashdot are linux based, and Hotmail uses FreeBSD and Solaris (from the kirch paper).
That's pretty much all I know about who uses what.
No wonder you're posting as an AC!
windows "nt" could never handle this load!
In microsoft's "best practices" documentation, they recommend a GROUP of machines more powerful than ftp.cdrom.com, just to server 6-8 GB/day.
Install windows "nt" on ftp.cdrom.com, and watch it crash, just like the debacle that occurred when microsoft tried to move hotmail from Unix to windows "nt"!
Are you taking into account that they only have to pay for what they download? i.e. tcp acks
Forget Mindcraft, this is where it really counts.
----
Every year during my review, I just pray the words "slashdot.org" aren't mentioned.
No, it shows that FreeBSD is capable, stable, powerful, and robust.
How does a FreeBSD machine's stability and power somehow prove something about Linux or NetBSD? It proves nothing more about Linux than an NT box doing the same thing would.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
As of 7:20 pm EST:
Welcome to wcarchive - home FTP site for Walnut Creek CDROM.
There are currently 3111 users out of 5000 possible.
Doesn't look slashdotted to me.
to find out what server and OS are being used by a given domain name. Try egg.microsoft.com
:)
I beleive those statistics are generated by a program called "Queso" (search for it on freshmeat) that does this at the command line.
You should check it out, it's hours of fun
-Erik-
Don't forget that Slashdot itself runs Linux and Apache and handles about half a million hits a day, much of that dynamically generated. By my calculations, at peak times, Slashdot tops 10 hits/sec.
--Phil (Way to go, Rob!)
355/113 -- Not the famous irrational number PI, but an incredible simulation!
It doesn't -prove- anything. It's merely an impressive feat. I have no doubt that another OS could achieve a similar accomplishment, however. Regardless, it is certainly a testament to FreeBSD's performance (not necessarily speed, also includes functionality) under extreme load.
"Bear in mind though, that all _independent_ testing has shown exactly the opposite to be true"
Proving that Linux is faster than NT on a desktop box with 64 megs of RAM doesn't really satisfy the statement "all testing has shown exactly the opposite."
And if you are referring to Smart Reseller's test, it was hardly independent. In speaking with the authors of the test, they readily admit to being biased against NT and went out of their way to cripple it.
That 17% was server OS sales for 1998. This is not 17% of total server installations.
Microsoft and Netscape dominate the Intranet web server market. Apache has only a small minority of this market.
But a 486 is hardly what people are running NT on.
The Oracle benchmarking that was posted to slashdot a couple weeks ago was also done in a biased manner.
By selecting hardware which is known to give good performance on Linux and poor performance on NT, the test is just as biased as the mindcraft study.
Which is fine, but don't pretend that they are unbiased an independent when they are not.
Oh, and BTW, all of the production servers at my company are running SMP. The intranet servers are quad processor Proliants, the Oracles are Sequents with 16 processors.
How many Linux servers do you see in production environments at Fortune 500?
The original poster was trying to make a point that seems to have gotten missed. You see, the original poster was saying that Free Unices are in fact capable of handling extremelly heavy load. FreeBSD just happens to be a free unix variant. Thus, this provides an example to saying, 'Free Unices can handle really heavy load.'
Everyone seems intent on picking them appart and treating them all as totally and completely seperate entities. However, that wasn't how the original post treated them.
If you put product "A" to a certain test, this proves absolutely nothing about product "B".
Perhaps not, but that isn't the comparison that was made. A much more apt comparison would be something like this:
Now, replace A with Linux, B with FreeBSD, and C with NetBSD. Now replace Foo with Free Unices. Perhaps this will clear up the intended point of the original poster?
Topher
Anyone who ever claims that the free Unices aren't up to handling heavy load ought to see this.
;-)
I think this proves very conclusively that the free Unices (Linux, NetBSD, FreeBSD, etc) are all very capable, stable, powerful, and robust. I'd love to see a box running a commericial OS try to match this.
Topher
It's now stable as a rock on Alpha's too :)
;-)
Is it? Neat.
I've heard lots of mixed reports on how far the Alpha port had progressed, though last I heard it was still fairly beta, but improving rapidly. The Sparc port though, last I heard, was pre-alpha still...
Topher
Why do we need to wait for more network upgrades for a terabyte in a day? Where is the bottleneck in this situation? Do we know? Or are we just spouting out that 'oh, it's the network thats slow' because it sounds good. OF course, that was the last second comment on the post, so that's probably the case.
I don't see that there would be much problem in boosting the total G services by 31G! Only 7% more! But heck, it made a great line to end the story on (yeah ok).
yacko
-- There is no sig line, only Zuul.
David Greenman, the Co-founder/Principal Architect of the FreeBSD Project just posted a new picture of the new wcarchive, it is now available here.
Updated hardware description is also available here.
It would be amazing if someone could pull some nice effects with The Gimp and make a cool looking "ftp.cdrom.com theme" for Windowmaker or something...
- Alfred Perlstein - Programmer and Administrator, Wintelcom.
How many Linux servers do you see in production environments at Fortune 500 companies?
Probably a lot more than you would believe.
Nobody takes an assay of "servers" if it's "just that box in the corner there". The only time people worry about their servers is when they're not doing what they're supposed to.
Chas - The one, the only.
THANK GOD!!!
Chas - The one, the only.
THANK GOD!!!
I've got 4 Linux machines with dual 300 MHz PIIs and half a gig of RAM each using round robin DNS to handle a very busy web site, and it doesn't serve anywhere near 1000 gigs a day, yet it needs hardware that is much more powerfull than cdrom.com, precisely because web serving is a much harder thing than FTP serving.
You are assuming FreeBSD and Linux have identical load handling patterns - they don't. It is not inherently harder to server static HTML pages than FTP files, and if used a special light-weight HTTP server (ftp.cdrom.com use a special light weight FTP server) then I do not think it would unfeasible to serve similar amounts of HTTP data.
In order to make ftp.cdrom.com capable of transferring that much data, however, sendfile() was needed. The FreeBSD sendfile API is, if I've understood correctly, different from the Linux one, in order to be able to support HTTP. If you'd want to serve web data competitively from a Linux machine, I think you would want to implement a similar API for Linux.
You'd probably also need to do a number of mods to the Linux VM system if you want similar performance to FreeBSD; however, I can't state that conclusively, as it is a long time since I've seen any benchmarks between the two.
Eivind.
Doubting the existence of evolution is like doubting the existence of China: It just shows that you're uninformed.
I've always gotten horribly slow connections from them, too many people always hitting it (mostly gamers I think). It was great back in '95 or so, but that place is too crowded now ...
... "No one goes to that restaurant anymore - it's too crowded."
what did Yogi Berra say
support gun control: take guns from cops
If I remember right, Microsoft had the record for a time, after releasing Windows 95. Then it was set by a big bunch of servers, not one single. This traffic lasted for several days. I don't remember how much it was.
Later, when cdrom.com moved their server, they copyed all the data over a 100Mbit connection and got the new record. I don't remember how much this was, either.
I haven't heard of anybody breaking this record before now.
"The assembler gave birth to the compiler. Now there are ten thousand languages." - Tao of Programming
Is the Smart Reseller test the same that was published on ZDNet?
If so, the configs were hardly "out of the box" - the Linux box in the ZD test was heavily tuned by a member of the Samba team. Furthermore, ZD didn't publish this information, where at least Mindcraft admitted that they tuned the hell out of the NT box.
--
Business. Numbers. Money. People. Computer World.
I'm not too surprised that it continues to break it's own records, CRL is a Tier-1 Backbone provider, so probably about 1/4 of the traffic is from within the same network, and the other 3/4 go across the NAP's on pipes dedicated to wcarchive.
--Jason Bell
--Jason Bell
Faster than the light of speed!
Mindcraft's test was a bunch of BS, they were very favorable to NT and not to Linux. Anyone here have an MSCE? We could do our own linx/NT test with the exact same machine and see which one does better. And how about a real world test, Linux can support 200+ users on a network, NT has trouble with more than 40. I've seen NT networks drop where a linux network wouldnt have even noticed the work load.
I'm a loner Dottie, a Rebel.
Huh? Did you read the article? Red Hat is the reason for the bandwith, not the reason the machine can handle the bandwidth.
Although I do agree that the article should have mentioned that wcarchive is a FreeBSD box. Since nobody else serves that sort of bandwidth, it's hard to see if Linux would be able to keep up (but it's certainly an experiment worth trying).
My Blog. Sela Ward can sell me long distanc
It's now stable as a rock on Alpha's too :)
Actually, their whole architecture seems strange. This seems like something much better handled by multiple machines with connections to different ISPs. Oh, but they're colocated in their ISP's machine room....
I'd love to see this kind of information (bandwidth, machine, OS) and more (time-of-day loading curve, ...) for all the big data providers, whoever they are... (download.com? yahoo? aol? conxion? ...) They don't seem to brag about it much. If they have little info pages like cdrom.com's, I haven't been able to find them.
I don't know why, but this kind of stuff just grabs me. Lifestyles of the bandwidth-rich and cache-famous? Packed-Tranfer Pr0n?
Umm, my 5.9 gcherokee does 0-60 in 7 secs, I don't think a hyundai can beat that :-)
(little sensitive sbout my car
My point was that it's a uniprocessor machine, and that benchmarking (and real-world deployment) has shown Linux to pretty conclusively outperform NT on uniprocessor machines. Quite likely FreeBSD would outperform Linux in some server benchmarks, but that's beside the point.
Well, Linux currently has 17% of the server market, and is estimated to be growing at 25% a year for the next few years...
If you check http://www.netcraft.net/survey/ you'll see that Apache massively dominates the web server market with around a 60% share. Being open source rather than commercially backed obviously hasn't stopped it from putting a huge dent in Microsoft's sales.
I'm sure Microsoft wishes that Linux _was_ a traditional single company commercial vendor, since that would give them a target to shoot at.
As Mindcraft's web site says (paraphrasing) "you identify your goals, we do the testing to satisfy them". Given that the paying customer was identified as Microsoft, it should come as no surprise that the goal was to show NT being faster then Linux. Bear in mind though, that all _independent_ testing has shown exactly the opposite to be true, certainly for uniprocessor machines such as the ftp.cdrom.com server.
... but as those two have indicated, this is a complete farce, and you can expect the "retest" results to be as information free as the first ones.
There have yet to be any standard SMP benchmarks (TPC-D, SPECWeb96 etc) published, although an unofficial Oracle benchmark indicated Linux to beat NT there also.
Also bear in mind that the "Mindcraft" testing has since been shown to have been performed in a Microsoft lab (the "Mindcraft" e-mails originated from a Microsoft domain)...
Ultimately, all the "Mindcraft" tests really proved is that Microsoft is starting to take Linux as a _very_ serious threat to NT - not surprising given the Linux server marketshare and growth numbers.
Microsoft is attempting to recover from the PR nightmare resulting from this testing by redoing the tests with "unimpeachable" Linux configuration expertise supplied by Linus and Alan Cox
How many net servers in the real world run off SMP boxes? Most ISPs use server farms of uniprocressor machines - much better bang for the buck. No-one's denying that Linux's SMP performance could be improved, but exactly how it compares to NT (which has it's own set of problems) is really unknown to this stage due to lack of fair testing.
The Oracle test I mentioned took one approach to fairness in testing both NT and Linux out of the box with no tuning on either side.
Given how artificial benchmarks are, the real world observations of NT vs Linux performance should probably be given more weight anyway. A quad zeon box is hardly what people are running Linux servers on - many are running on 486's! Try that with NT...
You can use this site:
http://www.netcraft.com/cgi-bin/Survey/whats
to find out what server and OS are being used by a given domain name. Try egg.microsoft.com !
This works by recognising the characteristic signatures of the different OS's TCP/IP stacks as they respond to a bunch of wierd packets.
Your reply is quite correct. Furthermore, I wrote "recent release of Red Hat 6.0", which should made it even more clear.
:-)
Alas, a second paragraph existed, which did mention FreeBSD in a as least offensive manner as I could manage. I guess CmdrTaco did not like my slamming of Windows instead...
(8-DCS)
What wcarchive needs now is a nice gigabit connection to an OC-12 or so. It's actually mentioned in /archive-info/slow.txt, too. At that point, wcarchive will truly be the best. (It already is, but its link is a bit slow for its popularity ;)
Didn't their "tests" "prove" NT was *faster* than Linux? A Hyundai is "faster" than a Jeep Cherokee, but guess which is more powerful. And after all, power counts more than speed in 90% of cases.
It depends on your definition. That server simply can't be overloaded with the relatively low bandwidth it has. It's simply a "stuffed pipe."
Why benchmarks? The claim made by the original poster was, the new record proved the stability and usefullness of the free unices. Now, ftp.cdrom.com runs FreeBSD. Obviously the new record can prove anything at all about FreeBSD only. It doesn't prove that Linux is _not_ just as capable either, but that's not the point made here.
If you put product "A" to a certain test, this proves absolutely nothing about product "B".
Sheesh.
Patrick