Slashdot Mirror
Sun Gagging Customers Damaged By Memory Problems?
cchuter writes "Apparently Sun has been getting it's customers to be 'mum' about a certain memory problem for as long as 18 months. The problem is assumed to be the cause for many website outages (most visible, ebay). "
quite a number of times. 6 errors that I have handled in my Sun shop of 3 admins and 50 machines. Only seem em on the 400 mhz/8 mb cache processors. Looks something like: panic[cpu28]/thread=0x307dbe80: CPU28 Writeback Data Parity Error: AFSR 0x00000000 00800002 AFAR 0x00000001 8104dfe0 We've found that attaching a grounding strap to all of the servers that were effected has cleaned the problem up. Haven't seen one since the straps, been around 3 months where before it was once a month.
Darn Jedi mind tricks...
Libertarianism is rich wolves and poor sheep playing gambler's ruin for dinner.
Tom Swiss | the infamous tms | my blog
You cannot wash away blood with blood
Also, do understand that these sort of NDA's are somewhat common when dealing with potentially explosive matters like this. Certainly Sun is interested in keeping tight lips, but they also would prefer to announce a solution along with the problem. It is an engineering problem where the "more eyes on the problem" approach doesn't necessarily bring about the greatest good.
-- Solaris Central - http://w
Wasn't it Sun that was complaining about Microsoft forcing clients to sign all those agreements forbiddding them to talk about some of Microsoft's practices? Granted, these are two separate issues, but now that Sun is having the issues, it's suddenly a different matter. Shut everyone up, and hope that no one finds out, before we can rememdy the problem or ship a new product.
And why would you have to bribe people that you'll fix something quicker, if they sign an NDA? That's an automatic red flag. I find it hard to believe that CEO and other top brass fall for such nonsense. There must be more to the story that was has been disclosed.
--
I certainly can't speak to why ALL the customers who signed the NDA did so. What I can speak to is why my company (a decently large telecom company which shall remain nameless) did so: Sun had a software patch that they felt might help alleviate the problem (I still can't reveal the details of what it does or why). In order to receive the software, we had to sign a non-disclosure agreement. It's that simple: we have a problem. If you want us to solve it, sign the paper. Otherwise, shut up and wait along with everyone else. As we had business-critical systems that were affected, it's not hard to understand why management did not hesitate to sign the NDA.
Ebay's problem was that it was running new hardware (E10000) with a very old OS (Solaris 2.5, not even 2.5.1) and a version of Oracle that had documented problems with that version of Solaris.
It had nothing to do with RAM, although I'm sure their former IT director would love to claim that.
I, for one, welcome our new Antichrist overlord.
-
'That sounds like a good deal, but I have a better one. I give you the finger, and you give me my phone call.'
Trouble is, you would get about the same reaction Neo did.Think about it. There is nothing legally requiring Sun to deal with problems in the order that they are informed about them. There is nothing wrong with Sun implementing a high priority queue, of people who sign NDAs, and a low priority queue, of people who don't.
So you face a decision take the red pill, and you get your website back up and running. Take the blue pill, and Sun gets a bit of bad press, and you go bust.
If you are someone like Ebay, it really comes down to that. You are your website, and you must sell your soul to keep it up 24/7 (or the best you can).
Here's a little story:
I know of a UK company who had a problem with Win95. It crashed every 49.7 (I think) days. So they went to M$ UK. They were told it would cost tens or hundreds of thousands of £ for M$ to look into the problem. M$ knew the company had no clout, and could not afford this, so they decided to fuck them.
The company had some form of relationship to a larger US company, so they got them to take it to M$ in the US. This time, M$ insisted on the company signing a NDA. When they did so M$ admitted that this was a known flaw in '95. The clock didn't wrap nicely, so when you reach 2^32 milliseconds - 49.7 days (as I remember) Windows 95 (at least version A) crashes.
M$ has since admitted publicly.
People like micros~1 and Sun have reputations to keep, and a great deal of power. When you are dependant on them for your businesses survival, they can make you their bitches.
Chalk it down on the 'List of Good Reasons to Use Opensource'.
G
Taken from the Sun website:
This is from the Platinum Warrenty, which is standard with a E10K (what EBay runs). They have a contractual agreement with everybody that they sell such a standard configured E10K to have an average response time on urgent calls, and even on the most minor problems, within an average of one day, if no other time is convienient.
In addition, if your web site is that important to your business, you can have a separate system availability guarantee. If Sun has agreed to provide five 9's, then they get 5 minutes 15 seconds of downtime a year. Even if they only have to provide three 9's, that's still only ~ 8 hours downtime a year.
Sun makes their money by providing very reliable hardware, guaranteeing obscene quantities of uptime, charging an arm and a leg, and then delivering on all of their promises. If they don't deliver, then they will get their asses handed to them in a breach of contract lawsuit. If people agreed to an NDA, it was either Sun doing a very good job of talking fast, or promising better service than what they had contracted for. Any business which had to sign that NDA in order to stay afloat should have invested the extra money in a better warranty agreement, because if your web site is that important to you, you should spend the extra cash to get your uptime guaranteed and contracted.
Business types don't really mind really expensive hardware/service agreements. Those are nice, fixed, predictable costs, especially if you have contracted with a reliable vendor (Sun). What they hate is having to lay out a bunch of money that they didn't plan for, because something unpredictable went wrong, and they didn't have their risks hedged. Hedging other people's risks is Sun's bread and butter.
As a sr solaris sysadmin, who has worked on Sun boxes for years, I have /nothing/ but praises for Sun service and support. Sun QA is top-notch, in comparison to the rest of the tech industry. I got my start in Linux, and still use it a great deal. At home, all but three of my boxes run Linux, including several PCs and a Sun 670MP. I also use various BSDs. Pretty much, so long as it's Unix, it's ok by me. /never/ seen it outside of 2x400 configuration in an Ultra II).. It wasn't. Ebay was a victim of bad sysadmins. Perhaps they were very good sysadmins, who had no idea of what to do with an E10k. Perhaps management made the decision for them. (This happens with eerie regularity)
/. headline for today.
Bearing this in mind, realize that I am capable of obejctive, honest review.
Sun has done more for the free software community than anyone therein seems to want to acknowledge, even though they are threatened by Linux. They are a large company, and do have their share of corporatism, but they also get an unfairly bad rap in the Linux community, for reasons I do not comprehend. Sun hardware has always been the industry standard for rock-solid reliability, and IO bandwidth. They never have been the blazing speed machines.
Going back to Ebay, where people were asking whether this was a problem with the cache (It is not a RAM issue, but an issue with the cache on the 400Mhz UltraSparc II processors, and I have
The fact of the matter is, the E10k is not a 'super-processing-power' box. It's a 'IO pumping, high-availibility' box. The sysadmins at Ebay had the E10k running flat out, not partitioned (As they're meant to be run) in quadrants. They grew so fast that they put the other E10k into production in the same fashion, instead of using it as a hot standby. Each E10k was a single point of failure, with the ability to be multiply redundant internally removed. A single problem with an OS that wasn't even officially supported on the E10k running at an invalid patchlevel caused a very highly publicised downtime. Instead of blaming bad setup (Which would be disasterous for investor relations), Ebay blamed Sun.
As to the latter part of this article, I know nothing about Sun covering up that problem, (Which I have seen before), but don't deny that Sun, being a big corporation, might do such things, as all corporations are wont to do, even the ones very popular in the Linux community. Usually that problem manifests itself in the system log long before any problem is ever seen. This problem is also listed on Sunsolve.
Sunsolve is one of the most open policies I've ever seen to system-related issues. The only group of people that even come close to that level of support is Debian.
While I know this was rather long-winded and might generate lots of flames, I do mean it. Don't bash Sun summarily, and don't bash Sun on QA. It's like talking about raising "Serious questions about Honda QA" if Honda issued a recall for defective OEM tires (A year after the vehicles with those tires were issued). Almost nobody would think to bash Honda QA over a single issue. Sun may have had a few quality issues from time to time, but so does everyone. And at least Sun is actually saying something, unlike companies that deny forever.
Why bash Sun, and not Intel - Another
-Kysh
--=:: Wings and tail and snout and scales of blackest night
I'm not representing Sun in this post-- just the facts. Get the facts straight:
It affected very few customers. (Sun has bent-over-backwards for customers to fix this. This included free consulting services to some sites and Sun did not use NDAs to gag customers on this. The NDAs were for disclosure of strategic planning regarding future systems/products and service offerings. How can a company keep customers that have had problems without being friendly to them?
It's an issue that only affects 8GB cached 400Mhz processors.
It's an issue w/ CACHE on a CPU NOT system memory.
The problems usually crop up in systems in poorly maintained data-centers. This includes centers with large temperature fluctuations, poor voltage regulation, poor humidity controls, and improper grounding. "User-error" and "misuse" exasserbate the problem.
The problems are limited to a particular production run of CPUs. New CPUs don't have this problem.
Sun hasn't denied a problem. Sun hasn't bragged about the problem, either. Would you?
Show me a PC that still costs that and has:
:)
1. Built in ability to boot off the net or _any_ other device you want, and can be set to default to that.
2. Serial console ability out of the box.
3. Massive online support center (sunsolve) from the vendor.
4. If hardware is made for the system, it _will_ work, period (I've run into plenty of PC hardware that doesn't play nice on some mobos or in combination with some other cards).
Sun hardware is expensive, and support is costly, but you damn well get what you pay for. We can get Sun here within an hour for critical issues, and the next day at the latest for non-critical. How's your local vendor and/or manufacturer on similar issues with PC hardware? I've had PC stuff fail and it takes forever to get replacements (unless the store's open, they have it in stock, and will let you return it). It's usually faster to just buy a new part.
BTW to your #3: Which OS is native for PCs? DOS? Windows? Linux? Personally I prefer Linux, but PCs weren't designed for any specific OS. Sun hardware was. Linux runs nicely on it though.
Oh yeah, can you run 64 bit on that PC? Didn't think so.
- My favorite error message: xscreensaver, running on an old Sparc 5 w/ 8bit color: bsod: Couldn't allocate color Blue
---
This sig has been temporarily disconnected or is no longer in service
then again, each year ram density increases so I'm not sure which density is considered safe and won't need ECC.
for boxes that will stay up a week or more, I tend to buy ECC 'just because'. its not much more expensive and doesn't slow things down enough to care about (even though gamers and o/c'ers will disagree).
--
--
"It is now safe to switch off your computer."
The existence of a problem with the 400MHz CPU with a 4 or 8MB cache has been well known on Usenet groups and a couple of online magazines. Sun engineers posting to the discussions say, "Yes, there's a problem. We think it might be foo, bar, or baz. Try the following steps..."
I don't think I've ever heard or read anything about Sun denying a funky problem in those chips. They may still be looking for the precise cause, but every time the issue comes up, somebody from Sun generally admits to it.
Dunno where the Gartner Group gets its figures from.
You cannot apply a technological solution to a sociological problem. (Edwards' Law)