Slashdot Mirror
Sun Gagging Customers Damaged By Memory Problems?
cchuter writes "Apparently Sun has been getting it's customers to be 'mum' about a certain memory problem for as long as 18 months. The problem is assumed to be the cause for many website outages (most visible, ebay). "
quite a number of times. 6 errors that I have handled in my Sun shop of 3 admins and 50 machines. Only seem em on the 400 mhz/8 mb cache processors. Looks something like: panic[cpu28]/thread=0x307dbe80: CPU28 Writeback Data Parity Error: AFSR 0x00000000 00800002 AFAR 0x00000001 8104dfe0 We've found that attaching a grounding strap to all of the servers that were effected has cleaned the problem up. Haven't seen one since the straps, been around 3 months where before it was once a month.
Darn Jedi mind tricks...
Libertarianism is rich wolves and poor sheep playing gambler's ruin for dinner.
Uh - No!
SRAMS are built with several transistors in a latching arrangement (there are 6T and 8T designs running around now-adays.) SRAMS are not nearly as likely to be taken out by radiation as DRAMS which are really just capacitors storing charge. Radiation was an issue 15-20 years ago and turned out to be MOSTLY due to radiation from the plastic the rams were cased in! Modern methods include coatings that make this a non-issue.
Caches being built out of SRAMs mostly have OTHER failure modes that are in some way design errors - like maybe electron migration which is essentially a way IC's age.
So - summary - it AIN'T radiation that is taking these chips out.
Have you compiled your kernel today??
>With Microsoft losing its Imperial hold, Sun is >beginning to look like a pretty shifty company, >casting doubt on its commitment to its customer >base.
Can we have some facts to back that up? I'm suspicious of the notion that any successful company or product can't be good. Last time I looked, IT was the most competitive business around - I doubt Sun can afford to neglect its customers.
If Linux gets more popular, will we start seeing everyone criticising it and vaunting FreeBSD instead?
I'm posting anonymously to protect my job and my employer. We have 3 E10000 and 7 E6000 systems in our production environment. All of them have had the problem. All of them have gone through multiple exchanges of CPUs, memory and system boards. We are seriously looking at switching hardware vendors at this time. We are looking at RS6000s as a possible replacement, but it would take a considerable amount of work to recompile all of our custom apps for their environment. We also have about a dozen S-390 mainframes and have been looking at Linux under OS390 too.. Either way, I think we are going to be getting rid of Sun
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
Also, do understand that these sort of NDA's are somewhat common when dealing with potentially explosive matters like this. Certainly Sun is interested in keeping tight lips, but they also would prefer to announce a solution along with the problem. It is an engineering problem where the "more eyes on the problem" approach doesn't necessarily bring about the greatest good.
-- Solaris Central - http://w
Wasn't it Sun that was complaining about Microsoft forcing clients to sign all those agreements forbiddding them to talk about some of Microsoft's practices? Granted, these are two separate issues, but now that Sun is having the issues, it's suddenly a different matter. Shut everyone up, and hope that no one finds out, before we can rememdy the problem or ship a new product.
And why would you have to bribe people that you'll fix something quicker, if they sign an NDA? That's an automatic red flag. I find it hard to believe that CEO and other top brass fall for such nonsense. There must be more to the story that was has been disclosed.
--
The nondisclosure agreements were apparently offered with a claim that signing them would bolster Sun's commitment to resolving the problem quickly, Henkel said. Sun customers began reporting the problem as long as 18 months ago, he said.
Wow does that sound like a bad idea! Talk about giving up your leverage. Sun must have offered some serious concessions in order to get them to sign this. I wouldn't even consider something like this without an expiration date. That would at least give the hardware vendor some incentive to focus their resources on resolving the problem.
This has been on USENET, since February, at least.
Suddenly, it's news to the world!
--Jeremiah
"Flyin' in just a sweet place,
Never been known to fail..."
Yes, but if everyone takes the red pill, they never have to piss anyone off, so they get to keep all their customers happy.
If you are a manager, and the choice is take the NDA, and get good service, or risk going bust - there is no decision to make here.
Let someone else make a stand: principles are nice, working webservers/databases are much, much nicer when your business is on the line.
So long as everyone plays nicely for Sun, they get all their customers to sign NDAs, and their customers all get good service.
G
I certainly can't speak to why ALL the customers who signed the NDA did so. What I can speak to is why my company (a decently large telecom company which shall remain nameless) did so: Sun had a software patch that they felt might help alleviate the problem (I still can't reveal the details of what it does or why). In order to receive the software, we had to sign a non-disclosure agreement. It's that simple: we have a problem. If you want us to solve it, sign the paper. Otherwise, shut up and wait along with everyone else. As we had business-critical systems that were affected, it's not hard to understand why management did not hesitate to sign the NDA.
Ebay's problem was that it was running new hardware (E10000) with a very old OS (Solaris 2.5, not even 2.5.1) and a version of Oracle that had documented problems with that version of Solaris.
It had nothing to do with RAM, although I'm sure their former IT director would love to claim that.
I, for one, welcome our new Antichrist overlord.
This story ?
--
Trolling using another account since 2005.
I did convince my boss not to buy Microsoft anymore. The problem is there is no way to convince all of the other employees not to buy Microsoft anymore, thus leaving us having to learn the newest mistake^H^H^H^H^H^H^Hoperating system from those lovely people in Redmond while all of the servers are quietly switched to Linux.
I couldn't however convince him not to buy Intel anymore and no one in the company wants Sun... oh well 2 out of 3 at least.
P.S. The servers' best uptimes:
NT: over a year (got shut down for Y2K, never was stable again)
Linux: just over a week (still being tweaked)
Devil Ducky
Devil Ducky
MY peers would get out of jury duty.
Sun is way too powerful for its own good. Some people like Sun better than Microsoft because Sun has come out with good technology like Java and Solaris, but in fact, I assert that Sun is not any better as a corporation. It is just a dangerous as Microsoft, if not more because it's more trusted.
:(
I really hope that GNOME is in good hands here but I can't help but shudder at Sun's involvement.
-
'That sounds like a good deal, but I have a better one. I give you the finger, and you give me my phone call.'
Trouble is, you would get about the same reaction Neo did.Think about it. There is nothing legally requiring Sun to deal with problems in the order that they are informed about them. There is nothing wrong with Sun implementing a high priority queue, of people who sign NDAs, and a low priority queue, of people who don't.
So you face a decision take the red pill, and you get your website back up and running. Take the blue pill, and Sun gets a bit of bad press, and you go bust.
If you are someone like Ebay, it really comes down to that. You are your website, and you must sell your soul to keep it up 24/7 (or the best you can).
Here's a little story:
I know of a UK company who had a problem with Win95. It crashed every 49.7 (I think) days. So they went to M$ UK. They were told it would cost tens or hundreds of thousands of £ for M$ to look into the problem. M$ knew the company had no clout, and could not afford this, so they decided to fuck them.
The company had some form of relationship to a larger US company, so they got them to take it to M$ in the US. This time, M$ insisted on the company signing a NDA. When they did so M$ admitted that this was a known flaw in '95. The clock didn't wrap nicely, so when you reach 2^32 milliseconds - 49.7 days (as I remember) Windows 95 (at least version A) crashes.
M$ has since admitted publicly.
People like micros~1 and Sun have reputations to keep, and a great deal of power. When you are dependant on them for your businesses survival, they can make you their bitches.
Chalk it down on the 'List of Good Reasons to Use Opensource'.
G
Taken from the Sun website:
This is from the Platinum Warrenty, which is standard with a E10K (what EBay runs). They have a contractual agreement with everybody that they sell such a standard configured E10K to have an average response time on urgent calls, and even on the most minor problems, within an average of one day, if no other time is convienient.
In addition, if your web site is that important to your business, you can have a separate system availability guarantee. If Sun has agreed to provide five 9's, then they get 5 minutes 15 seconds of downtime a year. Even if they only have to provide three 9's, that's still only ~ 8 hours downtime a year.
Sun makes their money by providing very reliable hardware, guaranteeing obscene quantities of uptime, charging an arm and a leg, and then delivering on all of their promises. If they don't deliver, then they will get their asses handed to them in a breach of contract lawsuit. If people agreed to an NDA, it was either Sun doing a very good job of talking fast, or promising better service than what they had contracted for. Any business which had to sign that NDA in order to stay afloat should have invested the extra money in a better warranty agreement, because if your web site is that important to you, you should spend the extra cash to get your uptime guaranteed and contracted.
Business types don't really mind really expensive hardware/service agreements. Those are nice, fixed, predictable costs, especially if you have contracted with a reliable vendor (Sun). What they hate is having to lay out a bunch of money that they didn't plan for, because something unpredictable went wrong, and they didn't have their risks hedged. Hedging other people's risks is Sun's bread and butter.
Or check http://www.netcraft.com/whats/?hos t=www.ebay.com.
Fh
Firs they have a reputation for stability because they earned it.
Now they want to keep it through legal agreements preventing people from reporting failures in the system.
It shouldn't matter if Sun knows what the problem is or not.. if it's a commercial product (not beta) and it fails, especially at that price, the consumers have a RIGHT to know.
I would almost think that forcing customers to not reveal flaws in your system should be illegal.. it's very anti-consumer.
Let me tell you my sun story.
First, the background.
I've used Sun for years in various projects and jobs. I like Sun. I know what it's like, what it's capable of.
So.. my company needed a couple workstations. I already knew what I wanted. So. I called my local Sun office and asked for a quote.
Then.. this sales guy *insisited* on coming to have coffee. Okay.. sure. no sweat.
He brings his 'engineer' with him. While sipping our fresh coffee, I show them around our place, tell them about what we do... and explain to them why I need the two workstations. I show them my *already* new network room freshly populated with servers.
What do they do? They sit down with me to give me their 'presentation' about how great Sun is and how crappy everyone else is, and keep trying to convince met o buy Sun workstations. HUNH? I think? WHAT? I alreadyh TOLD them I was going to buy them. WHy are they still trying to sell them to me?
Oh. And THEN they got on about servers. I had to cut the meeting off, saying 'Look fellows.. I do know about your servers... I just bought servers, and there is no way it's changing for the moment.'
Of course, then they invited me to their demo center to see how their little 450 acting as a 'file server' was so much cooler than the NetApp filer that I was about to buy... okay, I thought.. I'll go see that.
The entire meeting consisted of some guy from Sun showing me their 'NT' integration package, how it does CIFS and how it does domain control, and explaining how it was derived from actual MS source code. Whoppie, I said. I *have* NT servers to do this stuff. Does it do dynamic NIS to NT domain mappings? Oh.. no. Does it let me edit NT ACL's with vi? No.. sorry, they didn't know. Oh, and in order to make it *just* like NT, it has the same bugs in the file sharing code.
Great, I said. Guys... I want the benefits of unix here... not just another NT box. You are offering me a file server solution that is a) more expensive b) only has software raid and c) although i'll grant it has Solaris on it, and is flexible, it still doesn[t have some of the basic snapshot and backup features of the NetApp. And it's far slower at file serving.
I pointed out (politely) that here they were, demonstrating me a product that was designed to get NT admins to get into Sun (and NOT designed to let unix admins do anything cool), even though I already explained to them that I *LOVE* solaris, and already know this crap.
Then, they phoned and phoned and phoned.
Now.. this is *NOT* the behavior I expect from a professional company when I am nice enough to call THEM already ready to buy something.
Perhaps it's the joker with the HERF gun next door..
Fair point, but there was a reason for '95:
This was a software company. The software was a piece of server software, so customers ought to be running it on NT, but they wanted to be able to say it would run on all M$ OSes, so they were testing it on all OSes they could lay their hands on ('95, NT, and some betas, I think - this was a few years back now).
I was not working for the company, so I cannot say any better than this. Ultimately, I think that the product remained in development for a long time, so I think it was released after '98 came out. I *think* the problem was fixed by then (not sure) so I don't think it was ever a problem in the real world, I was just describing M$'s behaviour.
G
That hasn't happened in a while... ;)
I had a hardware freeze in a brand-new E450 today (4x400 UltraSparc II's with 4 meg cache) and was trying to figure out what might've caused it when I saw the article.
That makes sun 0/1 so far...
Yeah, point taken, and good information.
Nevertheless, I would still submit that whatever you say, when the Sun engineers arrive at Ebay, the Ebay suits would cheerfully suck the engineers' dicks if they thought it would get the server back up a minute sooner.
Okay, I'm sorry, maybe there is no need to get offensive here, but my point is, would you, honestly, not sign that piece of paper?
G
As a sr solaris sysadmin, who has worked on Sun boxes for years, I have /nothing/ but praises for Sun service and support. Sun QA is top-notch, in comparison to the rest of the tech industry. I got my start in Linux, and still use it a great deal. At home, all but three of my boxes run Linux, including several PCs and a Sun 670MP. I also use various BSDs. Pretty much, so long as it's Unix, it's ok by me. /never/ seen it outside of 2x400 configuration in an Ultra II).. It wasn't. Ebay was a victim of bad sysadmins. Perhaps they were very good sysadmins, who had no idea of what to do with an E10k. Perhaps management made the decision for them. (This happens with eerie regularity)
/. headline for today.
Bearing this in mind, realize that I am capable of obejctive, honest review.
Sun has done more for the free software community than anyone therein seems to want to acknowledge, even though they are threatened by Linux. They are a large company, and do have their share of corporatism, but they also get an unfairly bad rap in the Linux community, for reasons I do not comprehend. Sun hardware has always been the industry standard for rock-solid reliability, and IO bandwidth. They never have been the blazing speed machines.
Going back to Ebay, where people were asking whether this was a problem with the cache (It is not a RAM issue, but an issue with the cache on the 400Mhz UltraSparc II processors, and I have
The fact of the matter is, the E10k is not a 'super-processing-power' box. It's a 'IO pumping, high-availibility' box. The sysadmins at Ebay had the E10k running flat out, not partitioned (As they're meant to be run) in quadrants. They grew so fast that they put the other E10k into production in the same fashion, instead of using it as a hot standby. Each E10k was a single point of failure, with the ability to be multiply redundant internally removed. A single problem with an OS that wasn't even officially supported on the E10k running at an invalid patchlevel caused a very highly publicised downtime. Instead of blaming bad setup (Which would be disasterous for investor relations), Ebay blamed Sun.
As to the latter part of this article, I know nothing about Sun covering up that problem, (Which I have seen before), but don't deny that Sun, being a big corporation, might do such things, as all corporations are wont to do, even the ones very popular in the Linux community. Usually that problem manifests itself in the system log long before any problem is ever seen. This problem is also listed on Sunsolve.
Sunsolve is one of the most open policies I've ever seen to system-related issues. The only group of people that even come close to that level of support is Debian.
While I know this was rather long-winded and might generate lots of flames, I do mean it. Don't bash Sun summarily, and don't bash Sun on QA. It's like talking about raising "Serious questions about Honda QA" if Honda issued a recall for defective OEM tires (A year after the vehicles with those tires were issued). Almost nobody would think to bash Honda QA over a single issue. Sun may have had a few quality issues from time to time, but so does everyone. And at least Sun is actually saying something, unlike companies that deny forever.
Why bash Sun, and not Intel - Another
-Kysh
--=:: Wings and tail and snout and scales of blackest night
That's an entirely different issue. I can almost *guarantee* you that you had poor airflow on those Sparc 5's. Either they were on their side (not the way to set a Sparc 5) or they had so much dust and lint on the intake side that no air got in. Either of these causes heat issues and that leads to premature hard drive and memory failures (moreso than motherboard failures.) I supported a site with 2500+ Sparc 5's and that was the problem 98% of the time.
Please. As much as I think Linux is a great operating system, x86 hardware and Linux still can't touch the performance and scalability of Solaris on Sun SPARC hardware. I maintain an application which runs on dual Sun E6500s each with 10GB of memory and 30 UltraSPARC II processors. A Linux box couldn't touch this right now. The backplane is MUCH faster than any multi-CPU x86 machine and the fact that it can juggle 30 processors is something that Linux on x86 simply does not have going for it right now. Maybe someday, but certainly not now.
--
Never hit your grandmother with a shovel, for it leaves a bad impression on her mind...
I'm not representing Sun in this post-- just the facts. Get the facts straight:
It affected very few customers. (Sun has bent-over-backwards for customers to fix this. This included free consulting services to some sites and Sun did not use NDAs to gag customers on this. The NDAs were for disclosure of strategic planning regarding future systems/products and service offerings. How can a company keep customers that have had problems without being friendly to them?
It's an issue that only affects 8GB cached 400Mhz processors.
It's an issue w/ CACHE on a CPU NOT system memory.
The problems usually crop up in systems in poorly maintained data-centers. This includes centers with large temperature fluctuations, poor voltage regulation, poor humidity controls, and improper grounding. "User-error" and "misuse" exasserbate the problem.
The problems are limited to a particular production run of CPUs. New CPUs don't have this problem.
Sun hasn't denied a problem. Sun hasn't bragged about the problem, either. Would you?
While Sun could have manufactured the part to be more environment-tolerant, the users can also virtually eliminate the problem by operating their datacenters well within spec.
Therefore, I don't completely blame Sun for this. Even if there was no environmental influence, and it was just pure manufacturing flaw... the rates that this flaw happens at are fairly low.
11*43+456^2
Sun, on the other hand, has laid quiet, touting its successes where and whenever possible, covering up its failures, helping demonize Microsoft (unite the people behind a common enemy), and not really living up to its promise as the superior technological company.
With Microsoft losing its Imperial hold, Sun is beginning to look like a pretty shifty company, casting doubt on its commitment to its customer base.
This is not the way to build a lasting empire.
I think everyone's missing the obvious solution for this problem. Just take the red pill, sign the NDA, then post away all you want to Slashdot as Anonymous Coward. Simple, really.
I'd sign the NDA in a second - If they promised to build me a Lego Office.
Hey, boss. The bad news is we're still crashing. But just look at the pencil sharpner I built using #2371 and #1726.
--
IANAL, but I have picked up tidbits about contract law. Basically, if you sign something away, the contract is only binding if you receive something of value in return. Be it information, cash, services, property, or whatever.
A half-assed lawyer could convince a judge that the customers got nothing for signing the NDA, since they were (presumably) already entitled to timely fixes by warranties or service agreements.
My mom is not a Karma whore!
Show me a PC that still costs that and has:
:)
1. Built in ability to boot off the net or _any_ other device you want, and can be set to default to that.
2. Serial console ability out of the box.
3. Massive online support center (sunsolve) from the vendor.
4. If hardware is made for the system, it _will_ work, period (I've run into plenty of PC hardware that doesn't play nice on some mobos or in combination with some other cards).
Sun hardware is expensive, and support is costly, but you damn well get what you pay for. We can get Sun here within an hour for critical issues, and the next day at the latest for non-critical. How's your local vendor and/or manufacturer on similar issues with PC hardware? I've had PC stuff fail and it takes forever to get replacements (unless the store's open, they have it in stock, and will let you return it). It's usually faster to just buy a new part.
BTW to your #3: Which OS is native for PCs? DOS? Windows? Linux? Personally I prefer Linux, but PCs weren't designed for any specific OS. Sun hardware was. Linux runs nicely on it though.
Oh yeah, can you run 64 bit on that PC? Didn't think so.
- My favorite error message: xscreensaver, running on an old Sparc 5 w/ 8bit color: bsod: Couldn't allocate color Blue
What about something like this? And if that's not big enough, try the RS/6000 SP...
Or you might try going with Compaq/Alpha, who also have some pretty decent machines that can scale up to 32 CPUs in a box and even more for the SC series.
Okay... I'll do the stupid things first, then you shy people follow.
Okay... I'll do the stupid things first, then you shy people follow.
[Zappa]
Are the companies getting anything out of this at all? I know that if I was in their position, and some corporate goon (tm) from Sun came along and said:I'd enter into my Matrix-esque quote, 'That sounds like a good deal, but I have a better one. I give you the finger, and you give me my phone call.'
--
CitizenC
Oh come on. All it took to get these people to sign nondisclosure agreements was a promise that Sun would work harder to fix the problem? Give me a break!
Nobody decided to say, "yo, how abouts, you fix it now, and I won't find another server and tell all my friends about this issue."
There must've been something else in the offering to keep these clients quiet... I wonder what it could be...
I worked as a contractor for Sun in the first tier hardware support center in Burlington, MA from Nov 99 - Feb 01. This was a commonly known problem and was by no means treated as secret. There were a few syslog errors on the 400Mhz 4MB and 8MB cache processors that were tip offs. One, if I remember correctly, was a red state exception error and there were a few other cpu panics that were tip offs. It really was a low percentage of CPU related calls taken, but it was a known "mystery" problem. The procedure was to replace the CPUs immediately, and collect a core dump for the kernel team to analyze if save core was enabled and they actually had a dump. As techs we were not under any gag order on the phone. We could acknowledge there was a problem of which no cause or resolution was known and all we could do was replace the offending CPUs. Side note: my experience with Sun, as a contractor, was nothing but positive and they had a really great working environment in the support teams. We were always under instructions to side with customers and give service as best we could. Almost all of the reps would bend over backwards to help and give service on expired contracts, etc. Of course there were bad apples, but most people really wanted to help.
I'll tell you what really bolster's a company's commitment in fixing a problem, and that is realizing they are going to lose a lot of business if they don't fix it. An NDA simply means they get to take their time, fix it on their schedule, and not suffer severe ramifications for their actions.
A more proactive company would have jumped right out there, admitted their problem, and outlined exactly what they were doing to fix it. That prevents a leak from making them look like slimeballs.
This is not the way to build a lasting empire.
THIS problem isn't RAM related, either. It's the ecache on the CPU module that's causing the problem. It isn't DIMMs or DIMMs that are causing this.
We are the 'dot' in 'dotdotdot Why doesn't this f*cking .com site load?'
I've logged a kernel bug with sun for more than a year now. We opened tickets and got no real fix. After heavy threatening we got the vxfs licence for free which enabled us to work around the bug.
As of now, I still can kill every cgi-enabled webhoster running Solaris/UFS.
If those lame-assed, self-satisfied, over-confident bastards piss me off one more time, the exploit will be on bugtrack.
/ol
My group runs a mid-size to large server farm at a (very) major ISP and we're constantly replacing processors that crash due to "Ecache Writeback Data Parity Error"s. We were told that even replacing the processor isn't a remedy because the new one is just as likely to eat itself as the old one if you left it in place (As far as they know.)
..... (something bad that you wouldn't wanna be called).
In addition, the latest fix is a software patch that is supposed to massage the Ecache so that it never finds itself in the condition that they believe causes the error. Remember, they're still guessing at this point. 18 months later. How many of those 400Mhz are now used up with self-checks and Ecache scrubbing?
Ever babysit a Sun E-anything on bootup? Not only does it cost the company tons of $$ in downtime (made more extreme by the long boot process), it also costs them $$$$/hour for their engineers to sit there and watch these things POST forever.
I think the most aggravating part is how for all intents and purposes, Sun is now using the worlds largest enterprise sites as beta testers for it's product just like M$ uses the world to test it's software except that Sun expects us all to sign our voices away with the NDA so they don't look like a bunch of
(Non?)sequitor question: has anyone been able to get Sun ftpd to log to syslog like the man page says it can?
*** My opinions are my own and not necessarily the same as my employer.
Well, the money certianly was not thrown away on the "Loose Cannon" commercial. That one rocks! Had me completely fooled.
However, I can do without the "Power of the Dot" space movie ripoff commercial.
Visit DC2600
Eve Fairbanks says I drive a hybrid!LOL
---
This sig has been temporarily disconnected or is no longer in service
then again, each year ram density increases so I'm not sure which density is considered safe and won't need ECC.
for boxes that will stay up a week or more, I tend to buy ECC 'just because'. its not much more expensive and doesn't slow things down enough to care about (even though gamers and o/c'ers will disagree).
--
--
"It is now safe to switch off your computer."
The existence of a problem with the 400MHz CPU with a 4 or 8MB cache has been well known on Usenet groups and a couple of online magazines. Sun engineers posting to the discussions say, "Yes, there's a problem. We think it might be foo, bar, or baz. Try the following steps..."
I don't think I've ever heard or read anything about Sun denying a funky problem in those chips. They may still be looking for the precise cause, but every time the issue comes up, somebody from Sun generally admits to it.
Dunno where the Gartner Group gets its figures from.
You cannot apply a technological solution to a sociological problem. (Edwards' Law)
Something doesn't add up here - One company in this article mentioned that Sun helped them rearrange their data center to get the boxes cooler, then they stopped having problems. The other company essentially said that Sun hadn't helped at all, and had not forwarded any information.
Why would Sun not tell customers to keep the machines cooler until a fix could be found? Really, how many bigish Sun machines are running in uncontrolled climates? I would think turning down the thermostat a few degrees wouldn't be a big deal compared to "frequent" crashes.
Also, why would a customer sign an NDA with a vendor? Support levels should be stated in the support contract; any less and the lawyers get involved. This sounds like a device to prevent distribution of news, protecting stock price and corporate image. I always thought that should be done in testing/QA.
Don't pick up the pho*(@)$*@&@!@ NO CARRIER
From my experience in the past few months with sun's equipment, I've had a number of failures with the cpu modules.
I have about 15 of the systems at a customer site and another 15 an exodus. There have been no problems with the systems at exodus, only the ones sitting in the customers poorly vented and overcrowded equipment room. Of course the ambient temperature at exodus is such that a jacket is often required if you intend to stay for long...
From the line about also checking the customers installation environment, my guess would be that the majority of the issues are that they chips are failing when put under environmental conditions that are near their posted maximums.
People tend to forget that when you rack mount a server, you have to pay close attention to the airflow to that server. Sun's 4x00 series of systems are rather strange in that the airflow is right to left, not front to back, so you cannot put a 4x00 system into a 19" rack and expect it to work correctly simply due to the airflow issue. (This is why Sun's 5x00/6x00 come is such a strange looking rack with all of that empty space on the left... )