Slashdot Mirror
The root of all eBay's troubles
UncleRoger writes "A friend pointed me to this article would would appear to explain why eBay has had such troubles with downtime, including the outage since Wednesday evening. " It would appear that MS is tired of having the finger pointed at them - as they point out, it's an Oracle database that's running on Solaris that's causing the troubles.
You may suspect that it is false, but it seems to be ebay's story and they are sticking to it.
From the open letter from the founders now on Ebay's home page
To help ensure this, we are working diligently on a hot backup system that should automatically limit the length of potential outages to less than an hour or so. We have been working on this system for many months, and it is almost ready. Sadly, if we had this system in place a few days ago, we might have avoided this outage.
More interesting is that when I view Page Info with my Netscape browser, I get a June 9th date.
If you check Hotmail.com (now a product of M$) you will notice it is running Apache on a Unix box.
(Server: Apache/1.3.6 (Unix) mod_ssl/2.2.8 SSLeay/0.9.0b)
I guess even when M$ considers "The Importance of Reliability in an e-Commerce World" they choose _NOT_ to use NT, like everyone else.
How do they expect other people to use their products when they don't?
I don't get it.
My studio - www.graylands.ca
BTW, I really doubt that your server has been up since 1997, since you would either be a) not running the current service packs or b) not running the current software. You may be correct that the database software has not required a reboot, but I'd bet the OS requires it at least once a month.. That's the longest I've ever seen a production NT machine stay up.. especially if you are keeping up with the most recent hotfixes..
Also, If your UNIX admins are not able to get sleep because of problems with the desktops, you should get some new admins or maybe send them to LISA. They obviously don't know enough about the OS that they're using.. Has dell started to hire linux admins to run solaris machines now too?
No, as you noted, clearly 356.25 was the design goal.
---------
Bill Gates Is My Evil Twin.
My fault,.. I was reading another webpage while writing that,. hehh heh
My studio - www.graylands.ca
"One major point in the Sun machine's favour was that the code was on local file systems, the rest of the machines had to go through a 100 mb ethernet to get it."
Uh...doesn't that make the sun a completely useless data point in this benchmark? Builds are limited by the file access speed, not the processor or anything else.
"It was unbelieveable the difference in speed."
...until you calculate how much faster a disk access is than a network access.
That's right keep the faith. There's always a Microsatan behind everything that is bad in the world.
According to a lot of people, there's no difference.. when Windows dies, the PC is broken. And I have to step in and correct them.. no, it's not broken. It's running Windows. As far as they can understand, that damned wavy-window flag at startup must be burned right into the hardware..
Posted by d106ene5:
If your website is not high capacity, you should be okay. By high capacity I mean over 10 million hits a day. Hence Slashdot is not high capacity. If Slashdot got 10 million hits a day you would watch Rob's little world melt down into a puddle of pee - MySQL and mod_perl would meltdown big time.
If you are exceeding 10 million hits a day, avoid hitting a database. You're going to slam the machine in no time. Go for static pages or Apache SSI unless you truly need complex pages built out of a DB. Even then you can usually fudge it with static pages.
Your DBA who wants everything in Oracle should ask himself how much he wants Larry Ellison to control his destiny. Most old timers I know would like to minimize their debt to Oracle.
You're right. People care about money. They figure if they can convince the PHBs to use a system only they, and not the big service bureaus who employ MSCE's by the dozen, can administer (some obscure Unix whatzit, of course), that they'll be in the money forever.
So they hype the system they advocate, amplify anything bad they've ever heard about Microsoft,
and hope for the best.
Posted by d106ene5:
Sorry, those sites are small potatoes. You could run them with an abacus and smoke signals.
There are some good companies that are actually gonna try to put Win 2000 Servers into production use the day they come out.
Considering that Windows 2000 Beta 3 is out already and easily available for evaluation by these companies (for $60), they'll have ample time to evaluate and know what they are doing. I put Windows 2000 Professional on a machine tonight, and it looks nice so far.
You're spreading the FUD pretty thick there. And showing considerable ignorance on the topic.
Is Sun or somebody paying you, or are you just out on a religious pilgrimage?
Let's see, eBay has the head of an Ass, and the ass of a (flame-bait comment here). I think the core problem is lack of consistency. NT has already proven unable to handle high traffic (Hotmail) - if they had half a brain, they'd move the entire shabang over to Solaris/Oracle (or better yet - Linux/Oracle), rather than splitting things up like they're doing.
DeJa News (or Deja, as it's now called) used to be somewhat of a stream of data.
With the new Interface and focus, it's a pond. With a scum forming on the surface.
anybody green enough to use NT/IIS on an internet site probably isn't up to the task of running something as large as eBay.
Well, it wasn't exactly a rhetorical question. List the Linux sites with this kind of traffic. Or quiet down, kay?
I'm getting more than a little annoyed at the slander being casually thrown ebay's way. I know Mike Wilson, their VP of engineering. He is a solid guy and really knows his stuff. Anyone in the database business knows that things go wrong. ebay scaled up from nothing to huge in record time. Is that for nought because they have had problems recently? They moved the bar up a whole order of magnitude on what was considered possible for realtime e-commerce on the net.
So lay the fuck off Mike and his engineering operation until you understand the actual details of what was happening there. People seem to believe that "state of the art" comes about through random acts of kindness or something. No. It comes from learning from mistakes and accidents, and the bigger the flaw, the bigger the step forward that those on the leading edge make to put those things behind.
I agree that Microsoft's leap to take advantage of this in marketing and PR was uncalled for. On the other hand, McNealy's crew is famous for that kind of stuff too.
But the spinmeisters at Sun have nothing on Redmond, and so I still have to say
Bill Gates Is My Evil Twin.
Wow, who's your account rep? I wanna get together with them... Is that list or way deep discounted?
I'm not familiar with enterprise-class computing, or mainframes, or anything with that many digits following a dollar sign. But how on *earth* can *anyone* get away with charging a third of a million dollars for a computer *case*?
I never really understood why people were so excited about clusters and Beowulf and whatnot until now. Jeez. That's just insane.
I'm curious. Who *buys* these things unless they're simulating nuclear explosions or something? I can't think of very many things that use that much raw horsepower. Extremely high end simulations and models, okay. But what else? Besides, who can afford to put that much money into hardware every two years or so?
Is this where people like Sun make their real money?
He has NT Server experience. He read about it somewhere on an advocacy site.
A bird that doesn't flie does make an appropriate mascot. Or whatever.
Is it just me, or does MS tend to try and justify its problems by pointing out flaws in other systems? Does it ever occur to them that when problems are pointed out, it's generally a good idea to take the oppurtunity to fix them, rathar than saying "but everyone else is doing it!"?
--
"Insert witty quote here."
Clue:
It's not an ATX or an AT-style case.
geez.
After a quick pass through an optimizing compiler:
if (nt == unstable) { switchTo.linux() }
becomes
switchTo.linux();
Slashdot runs on Linux with mysql as the database, and it crashes too often. For a while mysql was crashing. Then the Linux kernel was crashing. All in all it's a big pile of unstable mess.
10 PRINT CHR$(205.5+RND(1)); : GOTO 10
I wouldn't mind one of those "unreliable" Ultra Sparc 5's :)
Of course the thing can crash its a computer.. Live w/ it Microsoft..
--Mark
ftp.cdrom.com is an FTP site. All it does, day in and day out, is deliver static files. That's not a very dramatic task. It's not a very demanding task.
You're in the wrong discussion. Nobody here is crazy enough to propose Linux be used for ebay.
I'm an oracle dba and I'd have to say that ebay's problems practically *have* to be human fsck-ups! I've never had big problems recovering oracle and even fewer problems keeping it going. What's this crap about failing hard disks? Where's the mirroring? I've had 3 disk mirror sets when it was very important to have 99.999% up time. One of my clients has oracle running on a 2 process ultra (unclustered) with NO disk mirroring and their system and database have been up for over 10 months now.
based on Microsoft's guaranteed 99.9% uptime, and $10,000 per hour downtime average cost, 24*7*365.25*.001*10,000=598,500 dollars per year in downtime costs.
- None can love freedom heartily, but good men; the rest love not freedom, but license. -- John Milton
I wonder how much of the blame can actually be placed on the Solaris and not on human error. The eBay announcements board says they've identified the problem, but doesn't detail what the error was. Does anyone have more detailed information?
Uh....wouldn't that be your browser's fault, not the server's? Unless it's using some sort of really wierd Java program or something that draws the entire page based on real-time data from the server, I don't see how the status of the server could mess with data that's already reached your computer.
Microsoft richly deserves blame for a lot of things, but not this, I think.
if (nt == unstable) { switchTo.linux() }
becomes
switchTo.linux();
that's after a -O. after a -O2 you get FreeBSD using SMP
-
ping -f 255.255.255.255 # if only
But I understand that MS is moving WebTV into the same building, so they may move to Magnavox and and Curtis Mathis any day now...
I still think the point stands. The majority of copies of these worms are getting run on 95/98 systems, not NT, AFAIK (and even if so, lots of people run on an NT account with admin privs...I do, but I'd never dream of running as run on a UNIX box). Unless you're running root on a UNIX/Linux/whatever system, you aren't going to take out your system with a single trojan (not that I'd be much inclined to run unknown binaries...), and if you had to you could create a "trojan" account to test a binary in isolation. On a Windows 95 machine....well, goodbye. Amazing that there isn't a version of Melissa yet that messes with .DLLs. Oh well. Give it a week for the Melissa Construction Kit, or something similar to come out.
A few days ago, they roll out their new website design, featuring more graphics and more dynamic content. What happens? They crash and burn during their high-load times...
I think they're just overloading their servers... Again.
If eBay uses a single Enterprise 10000 server for the back-end database, they should have had a standby server that they could have switched to in seconds. eBay could also have distributed database operations further.
One thing is clear: NT has no advantage in this area. Sun gives you the option of lots of little servers or one big server with a backup, and depending on the application, one sometimes has to make the latter choice. With NT, however, you are forced to go the lots-of-little-server approach.
On a side note, on the day on which Microsoft's poor security architecture in MS Office has been responsible for shutting down lots of corporate sites (including their own) and caused thousands of users to lose their data, their whining seems very ironic. eBay's problems are eBay's fault; the virus problems are Microsoft's fault.
I nternetWorld Article here.
There's an article from a couple of months ago over at InternetWorld that profiled the EBay server setup and its *two* Enterprise 10000's(Starfires).
Read it and you'll understand just how complex a setup EBay has. One of them performs the searching for the site.. "We had search vendors come in and tell us they had a great product, and we'd point a little of our load at it and it would melt into a puddle of metal on the floor."
Man, the balls on those guys at MS get bigger every day.
This seems to be a day of large scale computer disasters.. www.bbc.co.uk and all associated sites give me an empty directory with a subdirectory with a few bits and pieces and a bent paperclip?
Did someone, perhaps, have the whole bbc website mounted as a D: drive then blow it up with the outlook virus?
the world is going to pot! except I am sure the perl scripts will keep running.
Well, I'd certainly say that he relieved me of any fears of the worm now. Heh. Don't know what all this furor is about it, anyway.
:) ).
I rather suspect he isn't in marketing. I'm sure it was in good faith. But it was sort of pointless...people *will* run it, and if not, M$ *will* introduce ActiveMail or something stupid that will have a hole allowing auto-execution of mail at some point. Microsoft and security are like oil and water.
Don't get me wrong. I like Windows. Well, actually I like driver support for the 3d parts of my graphics card that Linux lacks so I can play Quake II more happily, but I'm sure that Windows is valuable to someone. I just would never dream of trusting it as a secure, stable system. And in the performance field...well, there isn't even opinion there...Windows, Server, 2000, whatever...falls flat. I don't know why you'd want to use one for databases *or* as a web server unless you were Microsoft trying to prove a point (which they did with Hotmail...
Well, I noticed they started crashing (repeatedly)
right after they changed the lay-out of their
pages. Maybe they should take a look at how
Slashdot orgainzes itself (or re-organizes itself)....
Anyhow I am mildly annoyed, I'm in the middle of
an auction. Rather Reminds me of the UO beta...
What do you mean Penguins don't fly, they fly perfectly well. Under water that is.
Actually, the "98%" figure is because there are relatively few non-macro virii that affect Macs. The last time I saw a new Mac virus (as opposed to a trojan) was probably 1996 or so.
Funny that you mention beowulf, clustering system for linux. MS tech couldn't get NT to run clusters so instead they decided to put out some FUD promoting technology they could handle--SMP. Where do you want us to tell you to go tommorow?
1) Std E10K are configured with 2 Ultra 5 boxes, each going though 2 seperate hubs to 2 E10K controllers.
2) E10K allows dynamic reconfiguration meaning hot swap of any component on the box while the system is up. To facilitate swapping out boards with expansion cards, they provide alternate pathing, which allows any component to have a secondary path to its desired location(seamless ethernet failover to another interface card.) EMC disks(used by ebay) provides powerpath, which provides up to 32 diffferent paths to the same component. There are enough power supplies and fans to survive double failures. There are also 4 distinct(correct me if I am wrong) paths, along with 2 address bus(?) on the backplane in which any one could fail w/o the box toasting.
This setup allows any single wire or piece of hardware to be short circuited w/o bringing down the system.
Sun is one of the best super duper high end dynamic made for the web serving thousands of happy users a day operating systems ever made. cdrom.com does jack to prove FreeBSD reliable for big tasks, it's a bunch of static FTP requests and sends, big whoop. If you want to point to FreeBSD reliability (I like FreeBSD dont get me wrong) then point out the special effects for Matrix were done with a big prallel (29 nodes I think) system running on FreeBSD.
I'm a loner Dottie, a Rebel.
That's because Microsoft's definition of "force" is generally, "We'll give it to you free. We think you should use it." It's how MS has gotten a serious stranglehold on education (where my PHBs and colleagues scoff at me/retreat fearfully from me whenever I mention Linux). While I can certainly understand the discomfort some people have when considering alternatives to MS, both FUD and the lack of cost make it very easy for managers to disregard any advantages to other platforms.
NeXT's WebObjects e-commerce offering to Dell was apparently designed and implemented in less than a week by one or two (granted quite gifted) developers. Microsoft's replacement, however, required a large team working for several months.
"Just fine" may suit your needs now, but you also had a system which likely suited your needs "just fine" which didn't require significant time and energy expenditure for replacement.
When was the last time you had to reboot the SQL Server for problems related to the operating system? When was the last time you had to restart the SQL Server Service? When was the last time you had to restart the server simply because you installed new software on the machine?
"My God...It's full of ads!" -Fry, about the Internet, Futurama
I have found Access to be a pretty good solution for what we do. We run the standard MS desktop app's, and Word, Excell and Access are company standards, so we can get some of our clients to do a lot of the inputing. . . it is compatible, and for lightly loaded, small db's it is really a good solution. The interface _is_ really nice. Not perfect, but for development, it allows for a quick turn-around and gives us a way to talk with our clients about the design. But for a "production quality enterprise app" I wouldn't fool myself for a second that it is suitable. Not even close. The ODBC drivers leak memory. It crashes far too easliy under any kind of real load. But for development, where you are going to have a few iterations of a design, it is a good lightweight solution.
My experience has shown Oracle to be really fast, really solid and _really_ expensive. As long as you are willing to put a lot of time into the set-up and initial tuning, run it on a really expensive machine, and put a little time into maintenance it is a rock-solid solution. But it is rather ugly if you need to make any changes to the schema or data types. If we want to make any changes, we have to go through the DBA, which is a real pain in the ass. He's a nice guy and all, but extremely overworked.
Actually, we just started using Oracle 8, Personal Edition on our desktops. That might be a good solution, too. If anyone is interested in how well it works, let me know and I'll keep you updated updated
Jeff
nrrd@earthlink.net
"Eye halve a spelling chequer, It came with my pea sea, It plainly marques four my revue, Miss steaks eye kin knot sea"
Posted by generic kewl tech reference:
Exactly. See #2.
I don't know about anyone else, but if a bad CD will bring down NT server I can't buy their line about HA servers. Maybe a bad CD would bring down Linux, too, but it's never happened to me. I can't think I've ever had a severe problem with a bad CD on any os other than MS's.
Just a thougt. . .
"Eye halve a spelling chequer, It came with my pea sea, It plainly marques four my revue, Miss steaks eye kin knot sea"
166 laptop. I had "connection reset by peer" on
cgi calling.
Guess what: CGI's run on NT+ISS
In my experience:
Oracle isn't that bad.
Sun's Enterprise 10000 are very good.
You are right about privileged access being an NT weakness due to most users being setup like this. However this has nothing to do with Worm.Explore, which simply deletes user files (documents and source code) and emails itself to other users - both possible with a non-privileged user.
The only thing that Linux would prevent is corrupting the equivalent of win.ini to launch the worm on other machines, or for other users of the same machine.
I agree witht he folks blaming the problem on poorly engineered backend on the part of eBay with no failover. Also, I'm no big fan of Solaris, nor of Microsoft systems. My personal favorite is linux, but if I needed to set up something that would be the best performance and most reliable, I would go with FreeBSD... I think ftp.cdrom.com alone should be enough to convince people of FreeBSD's perfomance/reliability. I nearly worked with a company who did all compiles on clusters of FreeBSD machines, uptime was over a year when I got there... I think more people should be singing FreeBSD's praise, but I'll still be running linux on my personal system for a while to come, maybe one day I'll install FreeBSD again..
XML is like violence. If it doesn't solve the problem, use more.
There is - it's called the "Disable Macros" button that pops up when you open a document that contains macros.
As I said before though, this most recent virus was not a macro virus.
Cool. It's good to see that MS is finally offering options of this caliber... which goes without saying that this is a new policy.. I'd love to figure out how to disable that stupid HTML replacement for a "host not found" error in IE4 (i hope this was fixed in IE5, it has to be the most annoying thing in the world).
Enough bickering though. Even though this button exists, and despite the fact I don't know crap about this "virus" (which 90% of media-labeled "viruses" are about as complex as a batch file -- a trojan), the admins at these places should not be allowing any form of automatically-executing or manually-executing attachments arriving in people's mailboxes.
The standardly traded mail format is just text - these guys should be processing this crap before it gets to the office. I'm thoroghly amazed it's not, especially in gov't offices where mission critical has just a little extra dash of critical sugar sprinkled onto it.
Caffiene is good.
-Erik-
All the recent wibble on E-Bay's site is about CGI servers and problems thereof - are these not running on En-Tee ?
Yep, they have two E10K's, plus numerous sun and PC boxes.
Question (I am putting together a web server myself): What are the specs of the machine? What server (Apache)? What dynamic content language (Perl? PHP?)
The US government is deserting UNIX and a bunch of long-dead systems in droves and going Microsoft.
Holy Christ, that's absolute bullshit. I can see that you don't work for the US government. MS just very recently got the bare minimum security classification to be even considered for anything that is senstitive and the government (being the effecient organization it is) isn't tearing apart their systems to make room. The NT deployments you see are mostly for secretaries and the occasional experimental system, nothing more.
Wow, over 40% of Win95 users reported that their PC quit working at least once a month... 15% on NT Workstation.
The SPARC Solaris machine I use at work has been running for 2 1/2 months straight (and that reboot was because of a power outage). On the other hand, if our NT server doesn't BSOD at least twice a day, it's a red-letter day.
Hemm. That's a new one - I could tell when B&N Online went from basically rock solid to an SQL cough-out error every other day I browsed that something had changed for the worst.
The TechNet article is written in a dull, prejudiced way that's more than a bit obvious in its selectiveness. The other points have been pretty well attacked, so I won't touch 'em.
I would just like to take this chance to plug ViaBid.com :)
ViaBid.com is a Linux 2.2.9 based auction site that is FREE,
and has many if not more features than eBay.
We have added a charities section to the site and have been
thinking about adding a 'Support Open Source Software' section
that people can post items in to sell and then donate the proceeds to
the OSS Project that they are supporting.
Please email me at nmourey@viabid.com with any comments
or questions you may have.
Some of this is just absurd. For example, the six points of failure with the Starfire. Hrmm:
"Applications running in Domains are only as reliable as the instance of the Solaris operating system. For applications to gain enhanced reliability from Domains, users must explicitly set up clustering, just as in standalone systems. Sun does not recommend clustering between Domains, suggesting instead that fail-over occur to either separate, standalone systems or Domains in other Enterprise 10000 systems."
Uhh, duh, isn't that the whole idea? Am I missing something here?
"Daemons that control domain operations and perform monitoring functions run on an unreliable device (Ultra 5 workstation), hardly a desirable situation in the context of a data center."
Excuse me? The Ultra 5 an "unreliable device"?? We have a farm of Ultra 5s that have been running for a year now. Total number of system failures or crashes of any kind: 0. Period. How is the Ultra 5 any less reliable than any other workstation-class system?
"When security is compromised on the System Service Processor, which runs on the Ultra 5 workstation controlling domain operations and performance monitoring, all running domains on the E 10000 can be brought down with a short command sequence."
No kidding. When (or rather, *if*) security is compromised, you could do a whole lot more than bringing down all running domains. Just the same as any other platform. How is this a weakness specific to Solaris or the Starfire?
And besides, these are supposed to *secured* (meaning, physically) control consoles. Meaning, locked in a cabinet in the datacenter.
"System boards that are hosting non-pageable kernel data structures cannot be removed from a domain without interrupting service. The Solaris operating system has to undertake a special "quiesce," or suspend, operation while the critical pages are migrated to another board."
Ummm, yeah. So? How is this any different from any other operating system? Again, I fail to see what the problem is. And besides, how often do you change system boards? Please.
Sure, go ahead... try and remove a CPU card from any NT-based system without first warning the OS. Not only will it hang horribly (ie; you can't do it!), you'll probably fry hardware as well!
The fact that the Starfire can even do this is pretty amazing.
"System boards that are hosting Token Ring adapters, ATM adapters, or non-Sun disk controllers cannot be present in a domain if board-remove operations involving kernel quiescence are to be performed on that domain."
Uh-huh. Sure. I know lots of people with Starfires running Token Ring off of non-Sun hardware that are removing boards with non-pageable data. Happens every day.
I'm not saying it doesn't happen per se, I just think that these arguments are rather ridiculous.
"If you remove a system board from a running domain without enough swap space, Solaris will hang. The administrative tools do not warn you if you do not have enough swap space available."
What kind of idiot doesn't leave enough swap space? What kind of admin would go ripping out system boards without really thinking it through first? What kind of person spends the incredible amount of money the E10000s cost without being informed as to the basics of running a Solaris-based system? Come on.
It's like saying "If you remove a CPU card from an NT-based system while running domains are active, the system will be brought down and all domains brought offline." Ummm, duh. If you remove your legs, you can't walk either. Apparently, M$ thinks that true Unix sysadmins are as stupid and lacking common sense like the server admins that they're used to dealing with.
"Reliable hardware is getting even more reliable. For example, customers can take advantage of 99.9% system-level uptime guarantees for Windows NT-based servers from major systems vendors, such as Compaq, Hewlett-Packard, IBM, and Data General."
These are guarantees on the hardware, not software. I'm sure this looks great for the PR, but hello? I'd love to know what the "major system vendors" think about Windows-based servers being equated with their hardware guarantees.
"Microsoft Windows® 2000 Server builds on these gains. For example, Windows 2000 Server supports COM+ load balancing, which eases customer development of highly available and scalable applications in a multi-tiered environment. On the back-end, Windows 2000 Advanced Server supports two-node fail-over clustering, whereas Windows 2000 Datacenter Server will support four-node clustering. IBM and other vendors will provide support for up to eight nodes."
WOW! I am truly impressed. Two or four-node fail-over. Please.
Finally, at the end:
"Which brings us back to eBay. For those keeping score, eBay relies on Windows NT-based servers running Internet Information Server to provide front-end web services, and a single Enterprise 10000 from Sun Microsystems to host an Oracle database on the back-end. According to published reports, the outages at eBay, which began in February, are due to problems at the back-end."
This is curious. Maybe I'm missing something, but a telnet to port 80 shows that www.ebay.com is using Apache 1.3.6 on Solaris. It doesn't get any more front-end than that, does it?
I did notice that pages.ebay.com and listings.ebay.com are running IIS 3.0, and cgi.ebay.com is running IIS 4.0.
Also notice that their web site is still up and running. Not that that means a whole lot, but hey.
I find a lot of what this article had to say utterly hilarious. The implications that the Starfire is an unreliable and dangerous system is the greatest work of FUD that I've seen in my life.
OK, enough said.
--globalnap.net, product of pure caffeine--
Just get a trained monkey to reboot your NT as soon as it goes down. A typical NT server will not go down more than once every 4 days. An NT email server fell over this often, because of a memory leak, usually NT is even more reliable than that.
:)
So let see 24 hours * 60 Minutes * 4 days
= 5760 minutes
At 99.9% uptime, this gives you over 5 minutes to reboot the system - should be OK if you have a fast trained monkey.
Lets face it, Linux cannot reach that level of reliability, unless you also hire a trained monkey to pull out the power cord at regular intervals.
We use GNU/SunOS.
What sucks is having to reboot when making the msot trivial of changes to the system, like modifying the DNS info or something.
If only "common" sense was actually that common...
Some of this is just absurd. For example, the six points of failure with the Starfire. Hrmm:
"Applications running in Domains are only as reliable as the instance of the Solaris operating system. For applications to gain enhanced reliability from Domains, users must explicitly set up clustering, just as in standalone systems. Sun does not recommend clustering between Domains, suggesting instead that fail-over occur to either separate, standalone systems or Domains in other Enterprise 10000 systems."
Uhh, duh, isn't that the whole idea? Am I missing something here?
"Daemons that control domain operations and perform monitoring functions run on an unreliable device (Ultra 5 workstation), hardly a desirable situation in the context of a data center."
Excuse me? The Ultra 5 an "unreliable device"?? We have a farm of Ultra 5s that have been running for a year now. Total number of system failures or crashes of any kind: 0. Period. How is the Ultra 5 any less reliable than any other workstation-class system?
"When security is compromised on the System Service Processor, which runs on the Ultra 5 workstation controlling domain operations and performance monitoring, all running domains on the E 10000 can be brought down with a short command sequence."
No kidding. When (or rather, *if*) security is compromised, you could do a whole lot more than bringing down all running domains. Just the same as any other platform. How is this a weakness specific to Solaris or the Starfire?
And besides, these are supposed to *secured* (meaning, physically) control consoles. Meaning, locked in a cabinet in the datacenter.
"System boards that are hosting non-pageable kernel data structures cannot be removed from a domain without interrupting service. The Solaris operating system has to undertake a special "quiesce," or suspend, operation while the critical pages are migrated to another board."
Ummm, yeah. So? How is this any different from any other operating system? Again, I fail to see what the problem is. And besides, how often do you change system boards? Please.
Sure, go ahead... try and remove a CPU card from any NT-based system without first warning the OS. Not only will it hang horribly (ie; you can't do it!), you'll probably fry hardware as well!
The fact that the Starfire can even do this is pretty amazing.
"System boards that are hosting Token Ring adapters, ATM adapters, or non-Sun disk controllers cannot be present in a domain if board-remove operations involving kernel quiescence are to be performed on that domain."
Uh-huh. Sure. I know lots of people with Starfires running Token Ring off of non-Sun hardware that are removing boards with non-pageable data. Happens every day.
I'm not saying it doesn't happen per se, I just think that these arguments are rather ridiculous.
"If you remove a system board from a running domain without enough swap space, Solaris will hang. The administrative tools do not warn you if you do not have enough swap space available."
What kind of idiot doesn't leave enough swap space? What kind of admin would go ripping out system boards without really thinking it through first? What kind of person spends the incredible amount of money the E10000s cost without being informed as to the basics of running a Solaris-based system? Come on.
It's like saying "If you remove a CPU card from an NT-based system while running domains are active, the system will be brought down and all domains brought offline." Ummm, duh. If you remove your legs, you can't walk either. Apparently, M$ thinks that true Unix sysadmins are as stupid and lacking common sense like the server admins that they're used to dealing with.
"Reliable hardware is getting even more reliable. For example, customers can take advantage of 99.9% system-level uptime guarantees for Windows NT-based servers from major systems vendors, such as Compaq, Hewlett-Packard, IBM, and Data General."
These are guarantees on the hardware, not software. I'm sure this looks great for the PR, but hello? I'd love to know what the "major system vendors" think about Windows-based servers being equated with their hardware guarantees.
"Microsoft Windows® 2000 Server builds on these gains. For example, Windows 2000 Server supports COM+ load balancing, which eases customer development of highly available and scalable applications in a multi-tiered environment. On the back-end, Windows 2000 Advanced Server supports two-node fail-over clustering, whereas Windows 2000 Datacenter Server will support four-node clustering. IBM and other vendors will provide support for up to eight nodes."
WOW! I am truly impressed. Two or four-node fail-over. Please.
Finally, at the end:
"Which brings us back to eBay. For those keeping score, eBay relies on Windows NT-based servers running Internet Information Server to provide front-end web services, and a single Enterprise 10000 from Sun Microsystems to host an Oracle database on the back-end. According to published reports, the outages at eBay, which began in February, are due to problems at the back-end."
This is curious. Maybe I'm missing something, but a telnet to port 80 shows that www.ebay.com is using Apache 1.3.6 on Solaris. It doesn't get any more front-end than that, does it?
I did notice that pages.ebay.com and listings.ebay.com are running IIS 3.0, and cgi.ebay.com is running IIS 4.0.
Also notice that their web site is still up and running. Not that that means a whole lot, but hey.
I find a lot of what this article had to say utterly hilarious. The implications that the Starfire is an unreliable and dangerous system is the greatest work of FUD that I've seen in my life.
OK, enough said.
--globalnap.net, product of pure caffeine--
...that microsoft's marketing department is like a midget boxer. They're harmless and easy to keep in check; however, if you let them get to close, they'll use your testicals for a punching bag.
All said and done, this is yet another good reason to not have ANY microsoft products in YOUR company's final solution.
Microsoft was giving ebay a firm kick in the teeth while they were down. Sun got splattered with blood, spittle and ebay's missing teeth. You know, if I were in EBay's position, I would really resent being used as market leverage. Yeah, fscking microsoft... that's the kind of people I want to do business with. It was about a brilliant a maneuver as Cabletron's bashing of cisco several years ago... cisco yanked their licensing agreement with Cabletron and someone in Cabletron's marketing department got fired. I suppose if you're stuck under some power hogging motherfscker in your company's marketing department, you have to use shock, stormtrooper tactics to get recognized, but jesus folks... there are much more creative ways to be fired or quit.
Still, we all know that nothing ventured is nothing gained. Sometimes it is better to not "venture" for fear of what you might "gain". If microsoft gains stupid customers from this venture, it will only make the final outcome darwinian.
MicroSquish lecturing Oracle on reliability?
That's like Bill Clinton lecturing the Dalia Llama on self-control.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
"Windows NT Workstation is the most reliable Windows operating system yet, resulting in significantly lower downtime for users. "
Talk about daming with faint praise!
Don't they have anyone to give this tripe the "giggle test" before they put it up on their web site?
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
I do programming on ad ("banner") servers, and most of them today rely on farms of these servers with localized, simple databases (codebase, ctree) which are refreshed during the night with new ads, etc. from an Oracle or Sybase database.
Of course, a "click-thru" (you really wanted to know what was behind an ad) is recorded in the main db. Since the click thru rate is miniscule, these servers are barely overloaded.
I've also worked with Access in recent years; It has probably the best user interface out there for manipulating databases, but is pig slow, and a very bad choice for multiuser or anything with over 100,000 records.
Oracle IMO is the fastest, and I have not had problems with it (but never used it to serve up web pages). They're consultants are the most expensive, however.
Hmpf, good think MS left some competitors to blame things on - I'd like to see Ellison and McNealy come out slamming NT/IIS3 - In fact, I've strong reason to think, from what little I've tried, that an old copy of IIS3 or ASP is what broke dragon-drop on my nt box.
Chuck
try { do() || do_not(); } catch (JediException err) { yoda(err); }
Okay, this is completely off topic, but it's too funny...
-
Yeah! I can see MS's marketing department gearing up right now....
90% of Yugo mechanics recommend MSActiveYugoYoke as the best Yugo yoking solution on the market*.
Studies have shown that the front passenger in the central Yugo in an MSActiveYugoYoke system sustains fewer neck injuries** than in competitors' systems.
Mechanics can take the MSActiveYugoSolutions Certification Test to demonstrate their compliance with a number of stringent guidelines set by Microsoft Customer Relations.
With an MSActiveYugoYoke system, you can go where you want to go today!
With MSActiveYugoYoke Enterprise Customer Satisfaction Enhancement Warranty, you can assure MSActiveYugoYoke functionality decades into the future!
Microsoft is firmly committed to enhancing MSActiveYugoYoke ease of use, particularly in high-speed interfaces with LargeSunTrucks.
Microsoft prides itself on its high degree of MSActiveYugoYoke-VehicleJava compatibility***.
MSActiveYugoYoke-the product line awarded the Gold Consumer Choice Vehicular Safety Award****.
Quotes from satisfied MSActiveYugoYoke:
"Well, it's a big improvement over MSActiveYugoYoke 95, I must say that." -- Dan Smith, Yugo mechanic
"Microsoft's Yugo yoking system is my system of choice for yoking Yugos." -- Steve Jobs, interim Apple CEO
"MSActiveYugoYoke is the supreme Yugo yoking solution." -- Jim Bob, a guy at Microsoft
"I guess if you like Yugos and use other Microsoft products, you might as well go with MSActiveYugoYoke to ensure product interoperability." -- an anonymous guy at C|net.
-----------------------------------------------
* Survey conducted by Mindcraft. Error margin +- 20 percentage points. [Note: only Yugo yoking system on market at the time was MSActiveYugoYoke]
** While traveling between 8.5 and 9.2 MPH on toll bridges during hurricaine conditions. Survey conducted by Mindcraft.
*** Certain procedures for unyoking in emergency conditions bear vague relation to VehicleJava emergency unyoking procedures in other vehicle yoking systems.
**** Award won was January 1996 award, won with MSActiveYugoYoke 95. Current product is the similar MSActiveYugoYoke 99.
I think B&N has always been on NT boxes (I've seen them, touched them, about a year and a half ago; Compaqs if I recall), though I'm uncertain what kind of back end they were running at the time. Presumably, SQL Server, but I never looked in those boxes.
And then where would they be, with a big investment in a top-notch marketing department?
I can second this. When I was at E10K training
last year, they had upgraded the SSP software to
a buggy level. As a result, the E10K hung, and
had to be phyiscally powered down. All during my
test.
----------------------------
Dammit Jim...It's "U-N-I-X",
Guys,
I'd hate to say it, but SQL server 7.0 is FAST! Last summer, into the fall, I got to run my own benchmarks, with my own data.
Systems Tested:
Sun E450 Quad Processor, 2 gig ram, dual raid arrays (10 drives) Solaris OS. tested with Oracle 8.x and Sybase 11.x
Dell 6300, quad 400 Mhz Xeons, 1 gig ram, 1 Raid array (3 drives), NT 4.0 SP4. We had 3 of these, and tested Oracle 8.x, Sybase 11.x, and SQL server 7.0
Dark Horse entry was a IBM Netfinity, 2 gig ram, Dual Pentum Pros, running DB2
Backup, non benchmark box was a Compaq Proliant Dual PPro 200s with 256 meg ram
All the vendors had full access to the boxes to tweak them (and we paid for the consulting time to do it!), and could optimize the the database design for what they needed
SQL server won, hands down, and during the benchmark period had much less problems than any of the other servers. Since we've gone live (a few months back), the only problems we have had were a problem with replication of a large table that has a text column, but this caused NO down time.
The only down time? An electrician pulled the lan cable on the router by accident.
Sometimes Microsoft does get it right!
Have YOU actually shelled out the kind of time it takes to do these benchmarks? It took us 16 man months, and an investment of 10,000s in software and consulting fees (we had the Dells in house, Sun loaned us the server).
When we went into the test, we expected Sun/Oracle to win on the perfomance, an Microsoft to win on price. The question was, how close was Microsoft going to be? Close enough that we could say yes? (The accountants like cheap ).
The order ended up like this:
1)SQL Server 7.0
2)Sun/Oracle
3)Dell/Oracle
4)Dell/Sybase
5)Sun/Sybase
6)IBM/DB2 did not even finish, as IBM never did get our database to work!
5)
Maybe investors think crashes are indicative of the market growing faster than their technical planners expected....
Buy an Ultra 10 instead
If you're a US citizen, it probably is. The US government is deserting UNIX and a bunch of long-dead systems in droves and going Microsoft. Lots of boring statistical bureaus and some higher profile defense-related thingies. That scares me. Someone mentioned somewhere recently that some of the missile-targeting systems were supposed to be NT-driven. Boy, would Wargames ever have to be rewritten..."He's got the final digit! The missiles are launching! Oh, wait...no, the remote computer just blue-screened. It's OK! We're saved!"
Of course, Microsoft would take credit for saving the world or something...
I think it would be quite possible to write a Worm.Explorer type worm for Linux - all you need is to be able to find Samba or NFS mounted drives (mount(1)), find the user's email address book (mail UI dependent), and (optionally) find a real message in their Inbox to fake a reply to (not too hard if they are using mail(1) format mailboxes, or Netscape Messenger.) Then pipe a message into sendmail or mail(1).
I'm not advocating anyone writing a worm like this, but Linux is going to be quite susceptible to this sort of problem. If sending mail requires authentication beyond just being logged in can you prevent this, but that's not very realistic.
In fact, all this could easily have been done on the mid-80s UNIX systems that I used to run.
I agree that macro viruses are based on the absence of any security controls for Office macros, but this sort of worm is not dependent on that - in fact you could write it to attack Netscape on Windows or Linux.
>As far as they can understand, that damned wavy-window flag at
>startup must be burned right into the hardware..
What! You mean I can get rid of it? You have to show me.
Oh, wait mine doesn't have a wavy-window. It has a penguin sitting. Never mind I don't want to get rid of the cute penguin.
I have used windows2000 since beta2 and it never crashed once on me. I am actually scared. If the reliability is this good in the final release, I can kiss my linux and solaris servers goodbye at work. Manangement actually believes only windows has tco support built inside it and since there is no hype about it in other platforms then its not there.
I can tell you the bsod bug in which you overload ther server and it crashes that is present in NT4 is not there in windows 2000. I can picure NT beowulf supercomputers everywhere in 5 years using dcom rather then one large risc based unix server. AHHHHH
Uh...wouldn't most of the potential concurrency/multiple processing problems come from entirely within the back-end, rather than from communication between the front and back ends? I got the impression that the front end didn't do all that much, from what others were writing.
Ok, i'll give you a factor of 10, the rest of the machines had local drives for compilers and object files. I think that this is also offset by the fact that the Sun is a heavily used multi-user machine and the others were entirely stand-alone dedictated to the build.
The difference between Canada and the USA is that in Canada healthcare is a right and gun ownership is a privilege.
I'm an Oracle DBA and just looked through the ebay announcements. It looks more like a systems problem than anything else, not an explicit DB issue (well, it looks like the disks that house the DB got fried, and the DBAs have to restore from tape).
So, they're running without mirrored storage arrays? Or, if they are, someone kicked over the two arrays in a martial arts demonstration?
Yeah - but who th f&*#k are you? The original poster said quite clearly that there are a dozen high profile, stable sites running NT - but only 2 that he knows of running Linux, and neither is that stable.
If you wan't to disagree - fine give a URL so we all have an example of a large, stable Linux site. Else its just hogwash and chest thumping.
The MS article implies that Barnes & Noble is an all M$ system. It was true, but it isn't anymore. They dumped SQL Server for Oracle running on Sun.
The multitude of front-end web servers are still IIS running on HP/Intel boxes.
It just ain't so. Microsoft is lying again. Nothing new, eh?
--
Get your fresh, hot kernels right here!
The Truth is that the Oracle Database got corrupted but this was because of an NT Server fronting the pages. The ThunderStone (Texis) peice was still running..and you can test this by doing a search on Ebay. Texis RDBMS Runs on the same Solaris Box that Oracle runs on. Solaris is running fine and still keeps running flawless. The Oracle/NT RPC via HTTP calls to the database is what caused the problem..not SOLARIS
Nothing. Or it'd be at Hotmail. With lots of PR releases plastered all over the site.
That leaves them with one reasonably uncompelling point about dependence on a service processor when they start plugging their own high-availability "story" - and what a piece of fiction that is. I've worked professionally in the HA field, and Wolfpack is the laughingstock of the industry. It's the most unbearably pathetic HA package I've ever seen, only surviving because of its parentage, and even then it amazes me somewhat that anyone uses it.
I know marketing material is not intended to be objective, but this piece is the most blatantly offensive piece of misinformation I've seen in a long time. While no individual claim in it is untrue, the overall result is incredibly misleading. Whoever wrote it is a master of their craft, but some forms of mastery don't deserve to be acknowledged.
Slashdot - News for Herds. Stuff that Splatters.
CNet implies that it's Sun, but doesn't come out and say it.
By the way, their IIS is version 3, which very few people are still running. Hard to see why they stick with it.
Of course you can do a search on eBay now, it's back up. And are you seriously saying the web server can cause the back-end database to crash? Explain this please.
I wonder who helped spread the word. I don't know if I'd slam Microsoft so far as to assume it's them, but i don't know who else...
All respect to Hemos, but Slashdot is a much simpler proposition than Ebay, even is it were getting 10 million hits a day. The forums could be divided among different servers on /., but on Ebay the need for cross-communication between different parts of the database is much greater.
Whoever is in charge of maintaining Ebay has a herculean task, and I respect them.
However:
I don't like the way Ebay handled the Ebayla fiasco, I didn't like the way they dealt (er, didn't deal with) the earlier allegations of security problems, and I'm just plain concerned about Ebay's growth outstripping their ability to keep up with it.
But we have been having power problems, the jokers who wired our new server room screwed up big time. We've also lost 4 power modules in an APC Symmetra that was powering it. Can't fault Sun._ ___________________________
___________________________________________
Auditing and dentistry are excellent career choices for people who don't like other people but aren't coordinated enough
That's right keep the faith. There's always a Microsatan behind everything that is bad in the world.
No need for any faith; their true colors are plain enough, especially since the DOJ has thrown light upon the sleaze.
If what I heard was correct, the part that croaked on Hotmail was Exchange, not IIS.
(Which shouldn't be a suprise - Exchange only started supporting > 16 GB in it's database last year or so. For those not used to dealing with corporate mail enviornments, 16 GB is not very much.)
--
Business. Numbers. Money. People. Computer World.
Irrelevant.
Even if this PARTICULAR virus isn't an Office Macro it sure does depend on hooks into the email system to spread. Not to mention that a quick look at any virus def file these days shows that 90% of recent virii are based on attacking Office security holes. Mcrosoft's Office is a nice agar plate Just Waiting to be innoculated.
Only Microsoft is bozo enough to develop a system enabling the rapid spread of portable, CROSS PLATFORM virii. If you are a Mac user, something like 98% of your virus def file is made up of Office macro virii.
Office macro virii are a huge cost for IT organizations. If they are not careful somebody is going to realize this and start publishing a cost analysis....
Well, I dono. He probably had it and just tossed it in. Besides, if the Sun was acting as a big fileserver at the same time...
But how on *earth* can *anyone* get away with charging a third of a million dollars for a computer *case*?
I think it's the chassis, powersupply, and (most importantly) the backplanes for the system cards.
Can you see Gates in Tibetian monk garb?
Posted by FascDot Killed My Previous Use:
I see that eBay's share price has dropped sharply today. Related?
--
"Please remember that how you say something is often more important than what you say." - Rob Malda
MySQL isn't really a database. Until it understands "commit work;" and "rollback;" it's just an incomplete SQL interpreter. Since it also has no concurrency features except for table locking, I should imagine that taking a valid backup means shutting down the system. No transactions and concurrency level 0 are no way to go through life, son.
cenobite, who still can't find his password
Posted by generic kewl tech reference:
I guess it is possible that another OS can, under certain conditions, fail. Of course, given my experiences with NT Server, I think it would be insane to put an NT server under those types of load conditions.
Any of y'all out there running Linux with the types of loads eBay has been experiencing?
And two other observations:
1) One wishes that Microsoft was as perceptive about their OS' flaws as they are about Solaris.
2) I'm just a wannabe anyway, what the hell do I know?
I only see critiques of Sun at these URL's
"The percentage of users running Windows NT Workstation 4.0 whose PCs stopped working more than once a month was less than half that of Windows 95 users."
More here
Yeah, yeah, it's workstation and not server, totally different operating systems. Not.
I use Linux at home.
I use NT at work.
From those two facts, guess which one I have found more reliable, more useful, and enjoy the most?
Thad
The Bolachek Journals
Yeh, I've done it w/ Tandem too. I used to work in the Hardware labs of the Austin, Tx. site. Worked on the PUMA project.
Do you have any links for MS cooperating w/ Tandem on this stuff?
Even with Tandem's help, I don't see it doing them much good. The reason you were able to hot-swap the CRU's with the Motherboards and stuff in them, was because the hardware switched things over to one of the many backup boards. Unless MS gets someone to develop this kind of hardware for x86, I can't see it doing them much good.
Your right that eBay should have switched over to a Tandem. Even though this is a software and not a hardware or OS problem, Non-stop UX does a hell of a lot to help you when something goes wrong. They probably would have been able to detect the problems with their software by now.
Well, before I blame eBay to much, I'm off to go see if Oracle has recent ports to tandem hardware.
Just a couple of choice pieces of FUD from the M$ web page.
--------quote on----------
3.When security is compromised on the System Service Processor, which runs on the Ultra 5 workstation controlling domain operations and performance monitoring, all running domains on the E 10000 can be brought down with a short command sequence.
--------quote off----------
So, let me get this straight. The workstation which is responsible for "controlling" operations can be used to stop operations? What was Sun thinking, including a command that would turn things off! Of course, we all know that one of the main features of NT Server over NT Workstation is that the "Shutdown" command has been removed from the "Start" menu.
--------quote on------------
4.System boards that are hosting non-pageable kernel data structures cannot be removed from a domain without interrupting service. The Solaris operating system has to undertake a special "quiesce," or suspend, operation while the critical pages are migrated to another board.
--------quote off------------
This is supposed to be a problem? Now, I think it's pretty neat that you can migrate kernel memory off of a certain piece of hardware and swap it out at all. We're supposed to believe that under NT you can do this at all? Much less without telling the kernel first? The only conceivable way to allow this to happen without telling the kernel to clear the board out first would be to make sure that all the kernel memory has had a copy paged out to disk. Or perhaps keep multiple copies of all kernel data structures (and hope they don't get out of sync.) Maybe NT does do the last one. That would certainly explain why it's a memory hog.
Pretty amazing if you ask me. This web page is clearly meant for the PHBs of the world, as anybody with any knowledge at all of how computers work is simply going to laugh at this.
We also from time to time hear the anecdotes about
SunOS 4 mail servers in closets that everybody used but nobody remembered about, which, when found, had been up for 2 years.
Seems like M$ knows a lot about Sun's "weaknesses". Maybe they should spend less time finding faults with other companies and more time fixing their own....crazy idea right?
Did anybody consider that it's EBay's software to blame, and neither Sun nor Microsoft nor Oracle?
Their database is corrupted. Does this mean that Oracle went kablooie and left bits everywhere? I suspect not. Or does it mean that EBay's software (which they just installed an update to) for running their site crapped all over some critical database table and made the entire database unusable?
Designing software systems to not have these kinds of problems is possible but difficult and I think it's apparent that EBay hasn't done this -- even if it was a hardware failure or whatever, they didn't design a system that could handle the failure, and their business is suffering as a result. EBay hasn't tried to pass this off on any particular vendor. Sun's silence might be due to the fact that they don't want to embarrass a rather large customer of theirs.
--jss (not speaking for anyone).
STING RAID: If you remove a system board from a running domain without enough swap space, Solaris will hang. The administrative tools do not warn you if you do not have enough swap space available.
This is pretty low. Yeah, it can happen - what else is an OS supposed to do when it has more processes than now remains as memory?
Come on now, NT's "You are running low on virtual memory" error messages is one of the most beautiful parts of the OS. It is perhaps the single most profound statements bestowed upon us. If Solaris (or Linux, what the hey) cannot provide the most highly trained administrators (I'm talkin top notch MCSErs here) with this sort of insight, well you get what you deserve then.
My understanding was that Microsoft paid Dell for the conversion costs. (And that the WebObjects setup was breaking under the load, but who knows if that was hardware of software.)
--
Business. Numbers. Money. People. Computer World.
This seems to me to be an operations problem.
1) No hot swap for the back end database
2) According to ABC news they are restoring from tape.
I mean, even if your back-end database bursts into flames, you should have a path to recovery that doesn't take 12+ hours, right?
How is that sung to "Going the Distance" I just can't find a way.
Perhaps the problem is data that didn't reach that AC's computer. I had the same thing happen apprx. 3.5 hours later, clicked reload, still just the part of the page that shows initially, scroll down and there's nothing else there. I finally clicked Help, About Netscape cause I couldn't remember which 3.x I'm running. This repainted the screen with the info instead of popping up a dialog box. When I clicked Back the full MS page loaded with the middle of the page showing in my browser window so maybe all those scroll bar clicks got caought in a loop somewhere and didn't execute until the MS page reloaded. Maybe it's my browser or hardware. But how come it's MS pages that always seem to have or cause the most trouble?
I see even classic Slashdot is now pretty much unusable on dial up anymore.
...two. At least according to an InternetWorld article a month or so ago.
Advice: on VPS providers
Actually, all of those sites have had their fair share of problems; just less obvious than ebay's.
Ebay's problems impact a larger cross section of folks and affect sellers as well as buyers. The closest thing to ebay in the above list would be the on-line stock trading houses; however, since there's REAL money involved there, those systems have human/telco backends the kick in when the computers stop doing what they're supposed to do. Even with that failsafe, I think there are some pending class actions against some of the on-line stock trading houses.
Besides that, a single transaction at some place like amazon.com or dell.com may requires less than 100 pages to be served up/queries. A single transaction on ebay? Probably larger by a factor of 10 or more. Complex things break more sooner and more frequently than simple things.
I think everyone squabbling over front end/back end details is missing that point.
We have a couple of these at work. It was the only thing that could handle the level of data warehousing we wanted to throw at it. You don't buy a new one of these every two years.. And individuals certainly don't buy them.
maybe try redbrick? we use this where i work (sorry, I'd like to say where, but i can't) and me like it a lot. me like a whole lot.
It didn't bring down the server. The companies made the operating decision to bring down the server because that is far easier than telling 10Kpeople not to open attachments to their e-mail. Have you ever tried to tell everyone in a corporation the same thing at the same time while using terminology that most do not understand? ("What's an attachment? What's he talking about?")
I'm sorry, that's a very poor excuse. Either MS's software should have a option to be able to keep these things from happening, or the users of these systems should be educated in these things.
It's no different than the idiot who wires up a chain of servers to a light switch, forgets to label it, and the janitor comes in and takes the servers down for a few hours.
Either there should be a method to disable these macros, or the admins at these systems should be processing all incoming/outgoing anything with something to remove the attachments (procmail does this I believe).
Ignorance is only an excuse for children and alzheimers patients.
-Erik-
Wow, your Microsoft sponsored brain cells must have worked overtime to think up that post.
You'd have to be pretty damn clueless to suggest that someone would come into a forum wtih this topic, announce by their own free will they work for microsoft, and not be sincere.
Perhaps you are looking for the poster who was talking about that beachfront property in nevada....
-Erik "Chronic Caffiene Deficiency" Hollensbe-
"6.If you remove a system board from a running
domain without enough swap space, Solaris will
hang. The administrative tools do not warn you if
you do not have enough swap space
available. "
I may be off but is system-board == motherboard? In that case I
wanna see a normal PC remain functional while you
remove the motherboard while its running.
I know not what course others may take; but as for me, give me liberty or give me death!
Their stock dropped 9.2% after they suffered 21 hours of downtime.
Estimated cost of downtime: $87,000,000 per hour.
Jamie McCarthy
Jamie McCarthy
jamie.mccarthy.vg
24 x 7 x 356.24, yeah, right. E-bay obviously did not test the disaster recovery drill. Short 'em.
A dramatic (less than 24 hours) reversal has happened in the OS ratings at http://www.deja.com/rate/item.xp?CID=11460&PDID=53 36
Somebody has used a bot for no good.
From looking at their announcements about the ongoing problem, looks like they've got database corruption issues. I'm not an Oracle guy (Sybase..), but I have to wonder how good, or bad, their disaster recovery planning is - what they do for failovers, that sort of thing. You really shouldn't be able to knock an entire "e-business" out like that, if things are set up properly. I wonder if this is more a case of too fast growth, not enough real hardcore planning and work on the robustness of their back-end.
Any server is only as reliable as the people who
run it, whether it's your own Linux box, or an
E10000 running the Oracle backend at Ebay.
I would stack up the reliability of any Sun Sparc running Solaris 7 against a Wintel box running NT any day.
Either there should be a method to disable these macros, or the admins at these systems should be processing all incoming/outgoing anything with something to remove the attachments (procmail does this I believe).
There is - it's called the "Disable Macros" button that pops up when you open a document that contains macros.
As I said before though, this most recent virus was not a macro virus.
Coming soon - pyrogyra
your dba is "overworked" not because oracle databases are inherently difficult to modify; there are plenty of 3rd party applications for graphically manipulating tables and other objects for Oracle and every other major database out there. The reason the changes go through him/her is because on a highly loaded system, MS Access-style datatypes like "Text" and "Number" just don't cut it; data structures have to be highly tuned to fit the typcial usage as closely as possible without adding any unnecessary overhead to storage volumes or lookup times. You might hand him a table to store name and address info, and his job would be to add a primary key to it, some indexes, possibly normalizing it with more than one table and some foreign key constraints, stuff like that. If these things are not done, with any substantial load your database will grind to a snail's pace if not deadlock or completely crash, no matter *what* database vendor you are using...Oracle in fact will show the symptoms of these problems much more slowly than MS-SQL in my experience. There is no tool out there that can do these things without a capable database programmer - just like every other "wizard" tool MS gives you that allows unqualified people to throw up unstable code, the Access tools are helpful for design-time "fooling around" but have no place in a production environment.
Is it that hard to see? They changed their page layout and tweeked the software, and now they are getting data corruption. It doesn't have anything to do with Solaris or Oracle. If you design a database that can't correctly handle concurrency, doesn't have good constraints, no triggers, etc. ad nauseum, then you are going to get data corruption. Also, the whole 'high availability' arguement is laughable. eBay's buggy software is still has high availability. ;) Just because their software crashes doesn't mean Oracle and Solaris are crashing.
It's just more FUD from the Empire.
--- A Jesus Fish eating a Darwin Fish only proves Darwin's point.
The eBay crash made the front page of today's Orange County Register:
EBay dark for 2nd day
ONLINE: A software problem leaves the site's auctions blocked.
The only techinical detail, FWIW, is the quote ``Company officials said Friday that the outage apparently stemmed from a problem with software provided by Sun Microsystems Inc.''
The article mostly deals with the missed selling opportunities, including a quote from someone who had quit her job to sell stuff on eBay and was losing a fortune.
IIS4 is better than IIS3 for dynamic content. The introduction of MTS, and improved data access objects and such is a great boon for programmers.
However, the move from 3 to 4 is not uneventful, especially with dynamic content. It would require considerable testing and several programmers to fix the problems as they arise in testing.
The old adage likely applies... if it ain't broke, don't fix it.
At work right now we're upgraing our intranet web servers using IIS4 by applying SP4, and beginning to use VStudio6.0 stuff. We've encountered a number of issues. Although mostly it's been that code which was poorly written will no longer run as it now throws an error.
I recall a certain Exchange server at my last job which lost a disk out of a RAID and we lost 67% of the store. Granted the data loss would've been pretty much cancelled if we had a backup and HARDWARE RAID, but still... 15hr straight and 33% recovery with the stuff I had to work with. . .
--
--
Me spell chucker work grate. Need grandma chicken.
I suppose Microsoft would like to claim that EBay's backend databae system would never crash if it was running on a NT w/ SQL Server.
I'ld like to see them try to pull that off!
Not only is Microsoft's website often returning database error when I try to utilize it (mostly the msdn sections), but it's frequently rearranged so that there are many broken links, and I can't easily find the same page there twice.
The latest insult is that a few days ago when I tried to reach http://www.microsoft.com/Data/ in order to download the latest MDAC, I kept getting what appeared to be very wordy 404 errors that indicated that that page did not exist. This was very discouraging, since I needed MDAC 2.1 for a consulting project. Figuring that they'd rearranged the site yet again, I did a search of Microsoft's site for "MDAC" only to find that all the hits were under the previously attempted link, and all of these pages returned the same error. All of this occurred on a Win98 box with all the latest updates (except IE 5) running IE 4.01SP2.
Unwilling to believe that they'd taken down the data access site, I tried to reach http://www.microsoft.com/Data/ from a WinNT 4.0SP5 box running IE 5. Guess what? It worked. It appears that Microsoft has effectively "IE 5 only"ed all of the developer's content on their site. Considering that IE 5 exhibits big memory/handle leaks on the one machine I have it installed on, I'm not eager to promote its use (however, I do like the updates to Outlook Express, and have had no troubles with that part of the product.)
Anyone care to check if http://www.microsoft.com/Data/ is accessible from Netscape, Opera, or other browser?
everything, perhaps they should start writing decent software ... just a thought
-
if knowledge is power, the internet is god - me again
Microsoft has it wrong on the E10k. It sounds like they've been talking to people here and there and haven't actually played with the hardware. The major SPOF is not the SSP workstation, it is the control board. If the control board dies, all your domains will go down. The control board is what, among other things, gives the clock to the entire system. But most E10ks are equipped with two system boards so that you can swap and get up-and-running again quick.
If the SSP (Ultra 5) dies... well, wait. It really doesn't happen. Something like a hard drive crash might do the trick. When you are without and SSP, the domains (virtual hardware systems) on the E10k continue to operate. But you're not going to catch things like record stop dumps (hardware error and warnings... such as persistant ECC memory errors). However, most sites that have purchased E10ks have also purchased two SSPs. They're so cheap in comparison, it makes sense. We have YET to fail over onto the secondary SSP on any of our 10 E10Ks. Since when is an Ultra 5 an "unreliable device"?
Sun complaining that the OS needs to be temporarily quiesced in order to move the kernel from one bank of ram to another? Heck, it's a miracle that it can even happen at all. I'd like to see microsoft write the code to move the kernel on the fly. Not a project I would want to be on.
Poo-poo on the adaptors that don't do DR? Hardly even an issue. Look at them... token ring, ATM, third party. I wouldn't even run a third party SBUS card on my E10k. The translation is that "a minority of SBUS cards are not a good choice for the E10k." Big deal, Bill.
About the swap space issue... they might actually have an issue there. I'm sure Sun is working on a warning now, if it is a problem. BTW... at that point you haven't actually REMOVED the system board. You are doing an operation called a "DR Drain" which moves all the pages of memory from the RAM in that system board to another. Once successful, you are able to remove the system board from the configuration, or abort the change.
So a Sun machine with Oracle goes down and it's obviously not their fault. But if it were a Windows machine you guys would be tearing Microsoft a new asshole. I'm just as much a Linux fan as most of you, but at least I can recognize complete hypocrisy when I see it. You need to realize that non-Microsoft products CAN fail, no matter how much you don't want them to.
Kind of reminds me of that Wierd Al song...
`Good Enough For Now'.
I don't remember the lyrics, but basically he was telling his troll-like girlfriend (Windows NT) that since nothing better seemed to be in view, she was good enough for the moment. I wonder how many Windows 95 users just look at NT when they need to make a server choice and say `Hell, it's good enough for now.'
Random Thoughts of a Happy Q3Test player...
I could just see it "You computer has just been infected with a virus. You must reboot your machine for the virus to work"
Maybe windows does have some security
Let me translate Microsoft's position:
1. Sun Enterprise 10000 systems have single points of failure. You can't hot-swap CPU boards arbitrarily, and the Ultra-5 front-end is a critical component.
2. Sun recommends that for high availability you cluster between multiple 10000 systems. This is bad.
3. Microsoft's commodity hardware platforms do not offer any of the scalability or reliability features of the Enterprise 10000, so clustering is the only option. This is good.
4. Microsoft's current clustering offering is primitive. In a survey, a majority of people said it was adequate.
5. Microsoft promises that Windows 2000 will have better clustering than NT.
6. eBay is not following Sun's recomendations that high-availability requires multiple systems. They have experienced outages.
BTW, it is shocking to me that eBay could have only a single server. This is at best incredibly naive; at worst blatant incompetence. Therefore I suspect it is false.
Your plan still wouldn't cause the massive damage that it is causing for one main reason: priviliged accounts. MS is an OS where a standard user (level program) can take down the whole OS. I'd never run unknown attachments as root. As a matter of fact, most Linux distros set it up to automatically foreward all mail to root to another user. So, _maybe_ one users data is trashed. The system is running and backup tapes are at the ready. This virus didn't affect most NT systems, but there were still people out there who felt like reading email while logged on as an administrator. It's stupid users, not stupid OSes that cause the most damage. So at least use an OS that protects a user from himself in production environments.
Reid G. Ormseth, Esq.
Thanks for pointing out the abcnews article. Restoring from tape can only mean one thing, invasion. No wait, that's not it. Anyways, it could simply be a good old fashion hardware failure in a poorly designed system.
..."
BTW, when I went over the the abcnews site, it had a late breaking story; DeForest Kelley has passed away.:-( Sorry for the tacking subject line, but this is how I will fondly remember my favorite doctor. That phrase and, "I'm a doctor not a
Perhaps they are just trying to drive down the stock price so they can get back in tomorrow and make a few million more.
The stock is down 10% just this afternoon...
Incendiary asked:
Very interesting.
I checked it out on Netscape 3.04 running on WinNT4. I should add that I have Java and JavaScript disabled, and go through 2 proxies (Junkbuster and LPWA). So the first proxy lies about my browser and OS, and the second proxy strips out the reference to OS. So as far as Micros~1 knows, I'm using Netscape 3.something Gold, but they don't have an OS mentioned.
The first time I went there, I got about 19K of the page, but blank page below all the images at the top. I thought this was FrontPlague's infamous omission of the closing table tag. But when I viewed source, I found that transmission of data had been cut off mid-sentence. Hmm. So I hit "reload", and promptly got a (bogus) 404 message.
I'm an Oracle DBA for a living, and to me it smells like a *major* hardware fault (i.e. a pipe fell on the EMC array), or some idiot SysAdmin (or DBA with too much access) did something amazingly bad, like rm -R * from /, and the machine let them.
I've had Oracle instances on Solaris running in Production, handling millions of transactions a day, with uptimes in the 6-month to one-year range (depending on what the maint. schedule is. Generally, Oracle/Sun is a good combination.
There is another possibility, though...
I've found Oracle (and Sun for that matter) to be extremely reliable -- as long as you are about one version behind the "latest and greatest", when all the (usually minor) bugs are resolved. If they tried to run 8i in production, for instance, Lord help them. Again, that would be an Admin fault...
Amazon uses Oracle...they don't go down. What's the problem at Ebay? Hmmm....
When people can't design a reliable system with budgets that allow the purchase of Sun 10000's!
I run a Sun 10000 with two SSP's. 10000's are connected to their SSP's via private ethernets. I have three private networks; two to allow redundant interconnects between the SSP's and the two 10K control boards and a third for general use, NFS mounting CDROM's and the like. Most people will have no reason to put the SSP's on a public network at all -- I certainly don't. In order to hack the SSP, one must first hack the 10000. Once they've done that, the ability to reach the SSP's by network is irrelevant. The point about the "problem" of the SSP having control is as silly as claiming that EMC Symmetrix disk arrays (heavily used in IBM mainframe shops) can be crashed by the single laptop each array contains.
I would love to know the details of their failures -- I suspect the article is hinting at issues that have nothing to do with their real problems. Further, I'd bet that the main vulnerability that people cluster Sun's against is hardware failure -- and I'd also bet that the main reason people cluster NT boxes is software unreliability!
Geeky modern art T-shirts
An excellent article from Performance Computing on this very subject and the cluelessness of /all/ the vendors in the market as these beasts become more common:
9 05f1.shtml
http://www.performance-computing.com/features/9
Also check out http://www.wintercorp.com. An interesting note from that site is that, even though MicroSoft participated in and helped sponsor the survey, they did not place in the top 10 in any of the important, published categories. Maybe some (all?) of the interested slashdot readers could send the Redmondites a missive, asking for comment on this survey.
BTW, the Performance Computing site is maddog's publishing home and an excellent source of well-thought, sane information for both the erudite Linux/Open Source professional and his PHB.
It's funny that MS holds up Dell as an example of a reliable, scalable NT-based site. At least their WebBoard support area is frequently inaccessible, and always incredibly slow.
MS also touts 99.9% uptime guarantees from Compaq, etc., but fails to mention that Sun claims 99.95% for the Enterprise 10000.
Nonetheless, my intuition (totally unsupported by any concrete info, other than their poor response to the eBayla exploit) is that eBay is a mickey mouse operation that got really lucky and rich, but does not have the technical expertise commensurate with a multi-billion dollar company. I wouldn't blame any of their vendors, MS or otherwise, for their troubles.
I've spent the better part of the day trying to get Windows NT4.0 (Build 1381: Service Pack 3) to properly work with my mouse. Do you have any idea how difficult it is to work with your system when your mouse randomly locks up, or how hard it is to change drivers with out a MOUSE?
I do.
Maybe EBay should go to an Access database. The last bug that was found was only in there for 4 or 5 years.
If Microsoft is going to bring quality software to the world, I expect that McDonald's will be leading the revolution in health food. I can rest better knowing that a company that couldn't write a program to properly subtract 1.99999 from 2.0 and get the right answer will soon be setting everything straight. Thank you Microsoft, I know my data is safe with you.
Much as I'd like to read the linked-to article, Microsoft's web site seems incapable of serving up a version of it that I can read in Netscape 3.01.
I get one window of text (along with the usual decorations) which is empty if I scroll down, and has vanished if I scroll back up. Fascinating. "View source" shows more JavaScript than actual document text...
No doubt it works just peachy in Internet Exploiter. But MS misses the first point of communication, which is to convey the message.
No wonder MS is losing.
-- Alastair
hey, don't forget the trailing / :
s/Suing/Using/
:)
So what you're saying is that its the machine's fault its complex, and leave the untrained sysadmins as sob cases? B-O-O H-O-O. Take the admin courses offered by Sun. E10k's are capable of multiple domains on a box and dynamic reconfiguration. An ENTIRE SITE, how practical that is, I'm not sure; can be run on it and the domains can be safe and sound from each other. Sun's support on these boxen are a minimum of Gold Spectrum contract. With so much flexibility, it will take some to configure right, but, they WILL WORK WITH YOU.
sixl6
sixl6@agfo.org
According to netcraft (http://www.netcraft.com/cgi-bin/Survey/whats), they are running on MS-IIS/3.0.
------------------------------------------
www.ebay.com is running Microsoft-IIS/3.0
Microsoft-IIS is also being used by Walt Disney, Compaq, Nasdaq, and The National Football League.
www.ebay.com is hosted by ebay.com.
------------------------------------------
Mario.
The above is not a rhetorical question--I really would like to know! What product from MS and its hardware partners is supposed to be equivalent to a Sun E10000? I guess I haven't been following MS well enough, because the last I heard, NT scaled to no more than eight processors and loses any sign of linear scalability after four.
A Sun E10000 maxes out at 64 UltraSPARC processors. NT may (try to) compete with Solaris on low end machines but are people really using NT servers for tasks that, in the Sun world, would typically be assigned to a fully loaded E4500 or higher?
Even on Alpha, NT isn't a 64-bit OS. That matters at the level we're talking about. I could understand Microsoft taking on Sun's workstations (where NT is eating everyone alive except Linux and the BSDs) and low end servers, but unless the game has changed dramatically there isn't anything Microsoft that competes in this market.
Microsoft's claims of the E10000's faults sound ridiculous considering that, to my knowledge, no MS-based system has any of the functionality they discuss AT ALL, much less better. I find it hard to believe even MS could make such statements, so what product do that have on that level? Clearly I missed something!
As for eBay, I used it for a few months a year or so ago and it was slow and extremely unreliable even then. I have to believe that poor configuration and administration are more to blame than any OS, even NT.
I have a lot of experience with the E10ks. I manage 4 of them (fully loaded). As far as the SSPs being a security risk. That is true *IF* you do not secure them. But then again if you login as root you can do just about anything on that machine. MS is trying to point out that the SSP is a different machine than the E10k. Physically it is, but logically it is not. I have secured my SSPs and I have not had any troubles. Period. MS talks about how you can not dynamicaly pull out the system board that the kernel resides on. Well, when was the last time you saw a NT box that you could pull a system board out of? Never comes to mind. For pure processing power, like a huge Oracle dbase that is a backend for a website, the E10k is very hard to beat. Sure it has it's flaws but doesn't all hardware platforms? Pointing out the flaws to diverte the attention from your flaws is just plain bogus.
Can we all just get along?
Scott
Scott
C{E,F,O,T}O
sboss dot net
email: scott@sboss.net
Scott
janitor
sdn website family
email: scott at sboss dot net
A single Starfire is rated as being able to deliver 99.95% availability with one - ie no clusters, and without all those caveats above - though it does need to be setup with reliability in mind for this - there's plenty of options.... Starfires aren't simple either - up to 64 CPUs, many more PCI and similar slots, memory slots, etc, etc. So, plenty of things to go wrong. Similar sized computers (from everyone) are really hard to transport without something going wrong. The only people more nutso on reliability on 'big iron' computers are IBM (from the companies I know a fair bit about anyway). Not only do they have backup CPUs, in their CPUs they do the same operation twice (in parallel, with checking at the end) to trap the ultra-freak chance of cosmic radiation or something casuing a flipped bit, or worse. (yes, they do seriously actually worry about such things... I remember an IBM proposal about how to design memory that can handle a once-a-month chance, for when you have a huge about of RAM, for some particular kind of radiation....)
The only complaint I've ever heard about Starfires in general is that if a PCI card (though not SBus card) breaks down it can hang the entire system until an operator manually flicks a switch to say that that particular card is defunct. Though this is really because of how PCI works - Sun's 99.999% reliable Netra 1800 has some highly specialised custom hardware to get around this problem with PCI cards, as well as backup CPUs and plenty of other stuff... ridiculously expensive too, they are... though apparantly more cost effective than anything in the same class. The Netra 1800s are a few months old, while the Starfire design is over 2 years old, btw.
I dunno about all of MS's claims, but I'm pretty damn sure that you can have hardware redundancy for just about everything, if you want, including the Ultra-5 controller. Most of the other claims seem to be related to the fact that you can hot-swap PCI cards, memory, CPUs and even mother-boards in a Starfire...
EBay do seem to have had more than their fair share of problems though... quite a few hardware problems it seems - I vaguely remember a problem earlier in the year was due to some controller card or something. As far as I know, nobody has had anything close to the problems EBay are having with their Starfire(s)...
Another little point... MS's idea of expensive downtime is $10,000/hour. I remember reading something on Sun's site a while back about high end availability systems. Sun's idea of expensive downtime is $10,000,000/hour - ie stockbrokers. They also had a list of most common causes for 'unplanned' downtime on their HA systems - first was 'operator error' (or lack of training, etc). I can't remember was second was, but third was 'fire'! (I'm pretty sure Sun's computers don't have a reputation for spontaneous combustion!)
Was ebay really running it's backend on one system, with no fail-over? That's pretty stupid...
For those of you who failed to read the entire article, here is the end paragraph:
"Which brings us back to eBay. For those keeping score, eBay relies on Windows NT-based servers running Internet Information Server to provide front-end web services, and a single Enterprise 10000 from Sun Microsystems to host an Oracle database on the back-end. According to published reports, the outages at eBay, which began in February, are due to problems at the back-end."
So no, the problem is NOT with Microsoft. This time. However, do not lose sight of the fact that Solaris is still closed source, same as Microsoft. This should not be taken as an opportunity to bash MS, as they raise a very good point:
"Planning for high availability deployments takes time and discipline. Successful deployments invariably depend as much on strict adherence to administrative best practices, as they do to the specific technologies being deployed. Ultimately, customers need to strike the right balance between high availability requirements, system cost, ongoing support and service costs, and skill level of technical support staff."
The problem lies with Sun's closed system, not with Microsoft's closed system. Don't get me wrong, MS has their problems. Its just that this time, those problems didn't come into the picture.
NightStriker
Home Page
But, to make use of this ability, you've pretty much gotta be a Sun outfit. I'm an old mac-head, so I know how much it sucks to be beholden to a single company hardware-wise. My PCI powermac represents a breakthrough for Apple just to use an industry-standard expansion bus. Then it took forever to get to the point where I really can use commodity goods, because the Mac third party manufacturers really didn't like competing with PeeCee suppliers and their down-to-earth prices.
In sum, the article is a piece of marketing crapola, spinning the problem away from Seattle like a top. It also makes its point in response to a pretty serious problem that reflects quite poorly on M$ products (not that I *knew* eBay was down; is there a slashbox I can set up to keep me appraised?). After reading this, I'm eagerly anticipate what they can come up with in response to ExploreZip/ZippedFiles! :P
Again, thanks to Hemos for posting this.
jaz
Death to Argument by Slogan!! (This post twice-encrypted with ROT-13. Replies not using same will be ignored)
The problem with your assertion is that Microsoft is using eBay's problems, which may or may not have anything to do with the reliability of Sun hardware or software (more likely it's human error) to further their outrageous FUD campaign.
It has nothing to do with a Microsoft system crashing, but rather trying to turn the eBay problem in into a FUD event that has people here upset.
This web page is the most outrageous piece of crap I have ever seen. Advising customers to rely of some piece of untested software still in beta to handle a massive mission-critical load. If I was eBay's CTO I would be looking to go upscale into some really heavy iron like the Himalayas or mainframes that operations that need REAL reliability use. The idea of going to a MS operating system for this sort of application is purely ludicrous.
The same day this is going on we have another round of word macro viruses terrorizing MS users everywhere. Why don't you see Corel and Lotus touting the fact that Word Macro virii don't trash their systems? Because they aren't low-life like Microsoft. Do you see slashdot trashing MS over this? No, even though they richly deserve it.
Microsoft deserves to be roundly excoriated on this one.
As we can see from this example, hiring administrators who only know how to point and drool may be less expensive in the short run but you'll get burned and burned badly in the long run.
Anyone with half a brain would have a failover (or two) for such an important system. Again, it's cheaper in the long run. Hardware does fail. Systems go down. It happens. If you got your administrators out of a cereal box, your company goes down because of it. If you can afford that $10,000 an hour down time (Or that $1,000,000 an hour down time) go ahead and get the point and drool admins at $40K to $60K a year. If you can't, get the $120K+ ones. It'll be less expensive in the long run.
Posted by d106ene5:
Everyone who has tried tying Oracle to a live, high-capacity system on the web ultimately has learned this lesson.
You can squack all you want about what you think you know about Oracle, but the proof is in the pudding. Remember when Excite's home page used to give you an Oracle error??
Lycos is going down this road now too - trust me, they'll be sorry.
I am not talking out my ass, I work at a company that has learned this lesson on a high capacity site.
I'm seeing this story everywhere. If the systems at your bank go down, and all the ATMs are out of service for 6 or 12 hours.. nobody even really notices... and those are considered "mission critical applications."
:-)
What I think we might be seeing here.. is the creation of an even *higher* category of important systems. Some sort of world-stopping problems that result from a downed server/cluster. Maybe the PHBs will start talking about ultra-critical systems.. or some buzzword like that.
Previously, a system goes down..and you can expect to catch hell from everybody in your company, and maybe even a few outsiders... but now.. you catch hell from CNN and the likes..
"We know what the probem is and we're fixing it," said Doug VanAman, director of public relations. "We're investigating it further."
He said the glitch appears to be unique to EBay and not with any other Sun customers.
At the bottom it says "The site outage was unrelated to the Worm.Explore.Zip computer bug, which began infecting systems across the country this week." Excuse me why I die laughing...
Well, you go to the story, which is on M$'s site, and you get a paragraph, cut off abruptly. When you scroll down and then scroll back up, the rest of it DISAPPEARS. Along with titles on buttons, etc. M$ can't even keep a Web site up, this Ebay Purple Star/Power Seller/Ambassador says, Ebay, get rid of EVERYTHING M$ and stick to Sun, SGI, or PREFERABLY ***IBM*** stuff, and stay away from toy OS's I wouldn't trust to run a toy tank.
Did anyone notice that they say eBay has problems in the back end? They say in another sentence that they use Oracle on the backend. What they don't say is that the two sentences are related! Perhaps they are having malfunctioning switch (which now can run NT!) on the backend.
The story is that they only implied that Oracle on Solaris was the problem. Perhaps a programmer forgot to unlock a row in Oracle. Does that mean Solaris' reliability is bad? They never said.
Microsoft and its enterprise associates believe that customers benefit most when they combine value priced, high performance Windows NT® server-based systems with the high availability advantages that distributed computing offers.
If you believe this sentence coming from Microsoft, then let me sell you some beach property in Nevada please. One rule in reading articles from any corporation, is to question the origin. Was the article written by an independent observer, or someone paid to write that line. In this case, I think someone was paid to write that line, whether or not he/she believed it.
-Ben
>>given my experiences with NT Server, I think it would be insane to put an NT server under those types of load conditions.
/.
Any of y'all out there running Linux with the types of loads eBay has been experiencing?
There's a long list of sites on the web at least as big as eBay running NT (Dell, Barnes&Noble, 1800flowers, ESPN, etc.), and they don't have eBay's persistant reliability problems. I suspect eBay programmers are at fault.
What's the biggest Linux site on the net you can think of? Dejanews? Hardly a bastion of reliability, and neither is
Posted by FascDot Killed My Previous Use:
MSSQL "rox" at what? Not providing basic features like large page sizes and query-embeddable user-defined functions?
I'm researching what to use for a midsize datawarehouse-esque DB and MS got the axe immediately in a totally objective feature comparison. That's before we talk about uptimes and speed.
--
"Please remember that how you say something is often more important than what you say." - Rob Malda
Sun Microsystems Inc. SunOS 5.5.1 [...]
# uptime
4:26pm up 278 day(s) [...]
I wish I could say THAT about even ONE of our
NT servers.
--
Chris
Where I work we have been Suing Oracle for a lot of internal apps, but nothing realy high-load. In fact 80% of out apps are simple enough and low-use enough that they work on M$ Access. Of course it's a really big bummer if there is a lot of demand on an Access DB, as it'll take down the server. . .
We have this Oracle DBA who is pushing for _everything_ to be done on Oracle. I would love to hear more about the problems you've had with Oracle and web apps. What is the "best" DB you've used for the web? What kind of problems have you had with Oracle?
"Eye halve a spelling chequer, It came with my pea sea, It plainly marques four my revue, Miss steaks eye kin knot sea"
The one, let's call it Site A, uses a $20,000 Dell NT Server 4.0 SP3 (dual PII-300) with 50 win95 clients; also runs MS Proxy Server 2.0.
Site B uses a Sun Ultra 2 Model 2300, dual sparc 300mhz. It supports tin, pine, lynx, gcc, filesharing, and ftp with 300 concurrent users across a 2 mile radius WAN with .2 cpu usage.
Is it fair to note that the last "restart" of the Sun was 67 days ago--the last "restart" of the NT...well, with all seriousness, it's been at an average 2 crashes per day (that's an 8 hour day).
eBay may have DB problems; let's not forget Oracle has all of its products availible for Linux and Oracle products are sold OEM thru Dell and their Online Store. I'm afraid Microsoft is trying to bolster their image. Don't believe NT needs PR help? See http://www.ntsecurity.net
Adios.
You're comparing Slashdot to this? I'm sorry, but are you just desperate? That's the best second example you can give?
Cripes, it's run by a handful of guys having fun, providing info, and getting their bills paid. While they sound damn proficient at programming and sysadmining, I hardly think that equates with proprietary tech support that those running NT or Sun on their e-commerce sties deal with.
If this is the best you can do, geez.
Just exploring the oracle site after looking over this article and saw this great message after clicking on the links on the left menubar:
"Can not service this request, please try again later"
It should be noted that Oracle is also used by these top sites.... and they don't have problems....
www.amazon.com
www.1800flowers.com
www.cdnow.com
www.charlesschwab.com
www.cisco.com
www.dell.com
www.etrade.com
www.onsale.com
www.rei.com
I fear that it is the ebay ppl who are at fault....
Really? Can you provide any documention on these claims? I was always under the impression that Hotmail ran on some Solaris servers. 1600 servers for the MS site. Now I won't believe THAT until I see some documentation.
power up the IBM Mainframe with DB2.
You have to wonder about the relative competence of an IT department that chooses to run IIS and ASP in lieu of the more reliable alternatives.
The worst thing is the PHBs and advocates will now use a dumn article like that to avoid acknowledging the alternatives.
"My God...It's full of ads!" -Fry, about the Internet, Futurama
Sometimes you have to wonder how things ever get approved to be on their website. Let's look at a few of the more imflammatory claims, which is really quite a kettle o' fish:
RED HERRING: Daemons that control domain operations and perform monitoring functions run on an unreliable device (Ultra 5 workstation), hardly a desirable situation in the context of a data center.
So what? The E10000 will continue to truck on as before without it. This is a complete red herring. The SSP is a really just a console station, nothing more. If it dies, you reboot it, or in the worst case, replace it with another one from the closet, which with Sun's AutoClient technology, can take on the entire identity of the failed box in a couple of minutes. (AutoClient allows Wall St. traders to replace their workstations and be working again with NO IMPACT in 5 minutes. Let's see NT do that.)
FUD SHARK: When security is compromised on the System Service Processor, which runs on the Ultra 5 workstation controlling domain operations and performance monitoring, all running domains on the E 10000 can be brought down with a short command sequence.
No one in their right mind would put the SSP on a network that extends beyond the glass house!! It's a *console*, designed to be locked up securely, like all other mission-critical control consoles. MS still doesn't get the data center, do they?
DUH WHALE: System boards that are hosting non-pageable kernel data structures cannot be removed from a domain without interrupting service. The Solaris operating system has to undertake a special "quiesce," or suspend, operation while the critical pages are migrated to another board.
This is incredible. They're knocking the E10K because you can't walk up to it and pull a CPU card at random without telling the machine first that you plan to do this. These cards contain memory, too, folks, which is why it's pretty reasonable to let the system move things to a safe place before the card goes bye-bye. Pretty much only Tandems can accept this sort of things (because they've got at least two of everything all the time, and they cost like they have even more), and if you're after real fault tolerance, you won't be running NT on them, even though you could...
STING RAID: If you remove a system board from a running domain without enough swap space, Solaris will hang. The administrative tools do not warn you if you do not have enough swap space available.
This is pretty low. Yeah, it can happen - what else is an OS supposed to do when it has more processes than now remains as memory? Although a warning would be nice, E10K admins aren't stupid (we hope), and they understand that there are easy workarounds to this - the E10K makes it very easy to move enough resources into the OS domain in quesiton on a temporary basis. If you don't have enough hardware to do that, you misconfigured the machine in the first place. This is hardly a weakness.
On the whole, the incredible thing about this is that MS is throwing rocks at a really good system with availability features far in excess of that for any practical NT box. You've gotta admire their guts, though - some people will read this and think the E10K is a really expensive, dangerous computer. Funny how they neglect to mention that there's not an NT box on the planet that can provide the performance of an E10K, regardless of how much you spend. This may change eventually, but it's pretty cheeky now.
If you need real fault-tolerance, get a Tandem/Compaq - but after you've paid all that money, I bet the Compaq folks would be the first to advise against using NT on it if you really want fault tolerance.
"The future's good and the present is nothing to sneeze at." - Roblimo's last
Well, for *that* price I can get an Alpha (533MHz 21164, 2MB cache (UX2 board) *with* UW-SCSI), a big cool G3 or a very good x86 PC (or 3 better cheap ones).
I'll send you my K6 2-300/128MB (should be good enough as terminal as well) and you give me one of your Ultra5's, OK?
More like...
It's about time that there's some media coverage.
eBay has had outages all over the place, and every time they do, it seems that their stock just goes UP another dozen points or so. Maybe somebody might actually figure out that even though the company is worth $billions on paper, their infrastructure still seems as unreliable as a garage operation. (Apologies to all the well-run garage operations out there.)
...is the amount of time that sales dept dickheads spend at this fucking site and not making calls. if they have a dba that can't manage that monster then i hope they keep him.
.
MrCreosote Meow!Thump!Meow!Thump!Meow!Thump! "You're right! There isn't enough room to swing a cat in here!"
I wrote hostview* (although I don't know if it's the same version which currently ships w/the E10K). Everything you say rings true to me. I detect a fair amount of penis envy in the MS statement. Face it, scaling is everything and bigger is better!
*hostview is the program which resides on the "unreliable" SS5 and monitors the E10K.
gsc@acm.org
yay! im sure the MS guys who *dont* use MS products on hotmail.com would love to see your justification for that one.
A couple of years ago, Dell used to run their ecommerce site on NeXT's WebObjects, but Microsoft basically forced them to switch to MS products. The old WebObjects site ran great and it took the engineers many many months to build an equivalent version using MS's tools. It's pretty sad to see Microsoft trumpeting that Dell uses the MS platform when it clearly wasn't the first choice and after they had so much pain in switching.
Yea, yea. We? We who? Having seen Microsoft's server (server?) software at work, I certainly have a hard time believing that. And I don't think "a few" NT servers are gonna be able to beat a Starfire on database serving. A Starfire is just plain huge.
Sam: "That was needlessly cryptic."
Max: "I'd be peeing my pants if I wore any!"
UNIX totally blows. I know ebay's database server is UNIX/Solaris and handles about the same amount as Compaq's servers do and they are totally MS.
They need to get a few Compaq servers and stuck MS SQL 7.0 on it and thier problems will be over.
We did and it solved all of our problems. Our MS SQL server gets over 1000 queries per second and it runs very smooth and it's on a 2Ppro200Mhz box with 1024MB RAM and 64 RAID disk array.
"We are very sorry, but our database server is still down. Everything that can be done is being done now to bring it back up as quickly as possible. We thank you for your patience and continued support and look forward to being back and to extending your auctions as explained in our last post."
That's all I have to say except that eBays Admins are top of the line people.
I work for a software company and did time as build captain for a while on our product. I was amazed at the difference in speed between various systems. The product is a fair size with a mix of C and Java and here are the relative speeds of a build on the various machines. (this was over a year ago and from memory :)
- nt on a 200 mh intel 128MB standalone - 140 minutes
- nt on a 300 mh alpha 256MB standalone - 130 minutes
- Unix on some sort of AIX ~40 minutes
- Sun Ultra with 2 160 mh CPUs 500 MB - 50 some odd users developers pounding on it, over 400 processes - 6 minutes
One major point in the Sun machine's favour was that the code was on local file systems, the rest of the machines had to go through a 100 mb ethernet to get it.
It was unbelieveable the difference in speed. Next look at the absolute pain it is trying to set up automated builds on an nt machine, not to mention the fact that both the nt machines would be dead half the time when you try to launch the builds. Microsoft has nothing to brag about.
The difference between Canada and the USA is that in Canada healthcare is a right and gun ownership is a privilege.
I don't know about EBay, but I know that E-10000's are extremely tricky to configure correctly.
Sun markets them as ultra-reliable and hardware-level redundant, but the truth is that configuring them is so complicated that even a team of experienced sysadmins is bound to screw something up sooner or later. If you bet the store on a single E-10000, then sooner or later the machine will crash hard and your store is hosed.
Given their size, expense and complexity, they are not appropriate for use as the main server in an internet commerce company. Sun should not sell these machines to companies like EBay.
In defense of Sun, I should point out that their "smaller" systems, namely the E-4000, E-6000, E-3000 etc., are rock solid and just as easy to configure as a small server. But no one would dream of running a whole store on a single one of them -- for reliability, you need to run several of them redundantly.
And Windows NT is far less reliable than any Sun machine. NT is the opposite of reliability. Production Solaris machines routinely stay up and running for months or years at a time. Show me an NT server which can do that.
UNIX totally blows.
Uh huh. And ftp.cdrom.com's daily transfer records mean squat.
btw, who is "we"? "we" nameless person that promotes MS but can't give an example.
You know what, I'm TOTALLY sick of seeing /. commenters correct the lies that Micro$oft proports on their pages. You need not look at 5 comments to this article alone to get the idea. Instead, let's do something CONSTRUCTIVE with all this criticism! :)
/. did this, they'd get hundreds upon hundreds of emails.
I'm declaring tomorrow Constructive M$ Bashing Day!
(Why do I have the power to do this? Because Barney says everyone's special in his or her own special way, and I'm invoking my privledge as a Special Person. That'll teach you to ask why. Feh!)
The next time you feel like correcting something that M$ claims and that's blatantly false, do so. THEN, email it directly to M$! If we had one day where EVERYONE from
Then, everything would start changing. The wheels would be in motion. M$ would realize the error of their ways and become Tibetian monks to pray for forgiveness!
We can make a difference, dammit! Can't we??
-
Excuse me but I have worked for Dell for 8 years now as a Network engineer and nobody forced us to use MS products. We use them because NT and related MS server products suits our needs just fine. Hell I can't remember when the last time I had to reboot our MS SQL server for any problems related to software..... was it 1997 I can't remember. Now as far as the MS Client/desktop software goes, that's another story all together. I love getting enough sleep. Not many UNIX admins ( the ones that used to work here ) could say such a thing. :)
I'm not really sure, but I think this was ment as a joke.
NT can't even run smooth without an MS SQL server on it, so this must be a joke.
Just out of curiosity, anyone know if hotmail is still running on Solaris? I seem to remember Micro$oft trying to switch the site over to NT when they bought it, then changing it back to Solaris because NT simply didn't work. Now that I think about it, didn't microsoft.com used to be mostly Solaris machines?
s/Suing/Using :)
john
I have been a ebay user for a couple of years now. They have been crashing so often I would swear that they were using a string of windoze 95 boxes. When they weren't crashing they were really Sslooooooooowww.
I got to be honest here. I do not understand ebay's high stock price. Shareholders must never use the site.
Maybe if they did, they would bitch slap ebay with a stock sell off. Maybe then ebay would deploy a load balancing MOSIX cluster and get some serious uptime for a change.
BTW, I really don't understand Microshaft's take when they can't stop that file loader called windows from crashing my box after two hours.
"ebay.com" return 'Solaris' and
"www.ebay.com" returns 'Microsoft-IIS/3.0'
Note: 'ebay.com' gets redirected to 'www.ebay.com'. So, they are probably using MS products for the internet part of their software and Sun/Solaris software for the back processing. Maybe the problem lies in the communication between the 2 systems... A buffer might have the probability of getting corrupted during the transfer (bad use of semaphores resulting in overwriting part of the buffer or such). That might be what caused bad data in the database. Their software may have problem to handle these corrupted data properly.
Mario.
I refuse to let 95 or 98 in my office. Not even a CD-ROM. NT 3.51 is a bit more stable than 4.0 in my experience, but doesn't handle big hard drives, so I moved my data crunching system up to 4.0 and it has basically been fine.
What really sucks is having to reboot whenever you install a new app with one of Micro~1's crappy shared DLLs. The latest offender was Foxpro 6, which I only need for some light file management, and which had me reboot three times (twice for the mandatory install of evilbad IE which I never use). Rebooting for network changes is bad enough, but for mere mortal applications??
This reminds me yet again why
Bill Gates Is My Evil Twin.
The real tragedy is that this won't take Ebay out of business.
I am sick of people spamming 'for sale' newsgroups (i.e. sci.electronics.equipment) with 'advertisements' telling me to go to ebay to pay lots of extra money for their equipment.
We need a few companies to go down hard, and a few hundred thousand investors to loose a LOT of money, to bring the 'net back into perspective.
Interesting that this is an almost-identical repeat of a previous post. Trying to get as much attention as possible, eh?
I can't decide if this is just flamebait or genuine astroturf. (When an MS person writes something like this, posing as an ordinary person, he is trying to create the appearance of grass-roots support for MS. Astroturf is fake grass. ESR seems to thinks it's a verb; I think it's a noun.)
I suspect the latter, but do they really think anyone here is dumb enough to take this seriously? Maybe they're just doing it so they can elsewhere claim to have the support of some
It's pathetically transparent. Why won't they just go away?
David Gould
David Gould
main(i){putchar(340056100>>(i-1)*5&31|!!(i<6)<< 6)&&main(++i);}
"Of course, given my experiences with NT Server, I think it would be insane to put an NT server under those types of load conditions."
Very limited experiences I presume.
Ye gods. This same person (Microsoft is king! Compaq is queen!) is gonna go out and get Windows 2000 Server and will probably still argue in favor of their system's stability.
People don't care about computers. They care about money. They hear that lots of money can be had in the computer industry. They go out and take their stupid little M$ certification courses (whoever it was that thought those courses up at M$ should be promoted...the long term benefit for M$ is incredible). They never learn anything but M$. They trumpet how great M$ is for the rest of their career, because they simply have nothing to compare it to.
Were the "problems" MS SQL solved caused because you had no idea what you were doing with the the previous (probably UNIX) boxes? Or perhaps your predecessor...did you come in the same time the MS stuff did?
Used as an SSP for a 10k it can't run solaris 7. The SSP software only runs on 2.5.1 (as of last month anyway). In fact the patches for the SSP software that controls the 10k are somewhat shaky also. I would urge any and all to be vary wary of SSP patches. Of course this doesn't really apply to this discussion so I'll go do something really geeky ...