Bitten By the Red Hat Perl Bug

snydeq writes "Smart coders always optimize the slowest thing. But what if 'the slowest thing' is the code supplied by your vendor? That was exactly the situation Vipul Ved Prakash discovered when he tinkered with a company Linux box on which Perl code was running at least 100 times slower than expected. The code, he found, was running on CentOS Linux, using Perl packages built by Red Hat. So Prakash got rid of the Perl executable that came with CentOS, compiled a new one from stock, and the bug disappeared. 'What's more disturbing,' McAllister writes, 'is that this Red Hat Perl performance issue is a known bug,' first documented in 2006 on Red Hat's own Bugzilla database. Folks affected by the current bug have two options: sit tight, or compile the Perl interpreter from source — effectively waiving your support contract. If a Linux vendor can't provide comprehensive maintenance and support for the open source software projects you depend on, McAllister asks, who ever will?"

Comments

waiving your support contract?

[dougmc]

Installing your own perl under /usr/local, leaving the system one alone under /usr, that waives your support contract?

Seems unlikely, and if actually true, remarkably stupid.

(However, messing with the perl under /usr, that would be a mistake. It could easily break other things that depended on that specific version ...)

Re:waiving your support contract?

[Richard_at_work]

No, it doesn't waive your support contract, but it does mean you will be relying on a subsystem that is not supported by the vendor - which validates the 'effectively' modifier in the original statement.

Re:waiving your support contract?

[corbettw]

It does on any issues that crop up in the applications using the locally compiled Perl. What's so stupid about a vendor not supporting something you compiled yourself?

Re:waiving your support contract?

[Dolda2000]

Even if it is true, the nice thing with a free operating system is that one can at least fix the bug oneself, support contracts voided or not. Try doing the same if there's a problem with Exchange or IIS.

Re:waiving your support contract?

[KaZen]

You would have no support contract for the /usr/local/* version of perl. And if that is what runs your multimilliondollar app, running with no support is typically not an option.

Red Hat would still support you if came across a bug in /usr/bin/perl. But their fix might not be useful for your /usr/local/bin/perl, even if it were the same bug.

Re:waiving your support contract?

[Anonymous Coward]

But Redhat has apparently known about this bug for 2 years so that support contract doesn't seem so great if they won't fix the problem.

Re:waiving your support contract?

[CautionaryX]

Woosh

Re:waiving your support contract?

[Bill_the_Engineer]

Perl is not an operating system and is open sourced no matter the operating system you choose to run it on.

Open source zealotry does not apply to the issue at hand.

A better question is are you getting your money's worth from Red Hat's support? The distribution being open sourced doesn't relieve Red Hat from the responsibility of fixing an issue contained within their product that an end user is paying for.

Re:waiving your support contract?

[no1home]

Very true. And this has been an ongoing issue with Linux adoption... I have a friend who runs mega-million-dollar, mission-critical systems and they've had to move off of Linux in favor of (Sun? Don't remember right now). It isn't about functionality. It isn't about open source. It's about support. Red Hat, et. al. want to be enterprise systems, and claim to offer enterprise support. But they don't perform enterprise support. As indicated here, change something to fix a bug, and you don't get support for that piece anymore. More, they won't support a system that doesn't have the latest updates, which is a problem on mission-critical systems. We don't update needlessly, and we certainly don't update to 'today's' patch. We have to wait and be sure the patch is stable and provides an improvement without risking our mission.

Until the players selling support realize all of this, Linux will be a difficult sell for such key systems (and the PHBs all think ALL their systems are mission-critical).

Keep in mind, I say this lovingly.... I want Linux to succeed and prefer it over the popular alternative.

Re:waiving your support contract?

[tkinnun0]

In this case, the moral of the story is that this wouldn't have happened with Exchange or IIS, because there is no middle-man that could mess with their code.

Re:waiving your support contract?

[segfaultcoredump]

If you were stuck with a closed OS like AIX, HPUX or Solaris you would still have the option of compiling perl from source.

The nice thing is that the _app_, not OS, is free and thus he has the option of picking another way of working around the issue.

That said, if this was a kernel issue, he would have no choice but to compile a new one in and totally void his support contract. Yes, he would have the option of doing it but then he loses the "support" that he paid for.

Re:waiving your support contract?

[/ASCII]

The company I work for does support for any Linux distribution, custom compiled packages, whatever. If the customer uses non-standard packages and oddball solutions, it often takes more time to solve their problems, but since we work by the hours, that's their problem.

I find it hard to believe that businesses such as ours are unusual.

Re:waiving your support contract?

[KaZen]

This is nearly opposite my experience. I'm working at a very large Wall Street firm.

Red Hat does *not* tell us: "Oh, I'm sorry you're not running the latest support pack, no support for you."

We've had to run a modified GCC for a while and Red Hat, *again* didn't say "You've changed it, so support for you." What they *did* say was, "Can you reproduce this on *our* gcc?" Which again is better than We've gotten from some other vendors.

We're still running AS4U4 in some places and RH has worked with us to track down bugs. Sometimes it ends with: "This was fixed in 4.5 please update." Sometimes it ends with "This is a bug, and here is the HF, please update to the released version when it becomes available."

In fact I have a hard time sometimes of getting our Admins to open tickes with the *right* vendor, because they'd rather open a ticket with RH, because it gets solved sooner. (Course that is more a dig on HP, Veritas, EMC and some other "Enterprise Software" companies.)

Unfortunately for Both us and RH, we don't like to update either, and even when RH has proven an update solves the problem, it's hard to get the Admins to actually update the boxen.

Re:waiving your support contract?

[/ASCII]

Since perl is open source, so open source zealotry still applies.

Re:waiving your support contract?

[DittoBox]

Interesting. The entire reason we like Open Source is because we can change the code and fix bugs and make our lives better without being explicitly tied down to a vendor. But when I sign up with a vendor to provide support with things like this, I'm not able to fix those problems and worse off neither is the company that I signed up with.

What's the point then?

Re:waiving your support contract?

[Grimwiz]

My experiences are the opposite to that listed above. RedHat have been more forgiving and sensible supporting live production systems than my experiences from SUN or Veritas. Both experiences were for mega billion-dollar companies.

Re:waiving your support contract?

[Jonboy X]

The stupid part is a vendor only supporting a broken version of a package.

Re:waiving your support contract?

[Anonymous Coward]

... As indicated here, change something to fix a bug, and you don't get support for that piece anymore. More, they won't support a system that doesn't have the latest updates, ...

These issues are not specific to Linux, or open source support vendors.

    None of our closed source vendors will provide support if non vendor supplied "fixes" are applied to their products either. And in the majority of my experience the first thing they will make you try is applying the latest updates.

    YYMV but most of the issues I've run into like your friends are driven by 3rd party support of their products running under certified operating system configurations. I.E. install OS Z, then install product X, then recompile the OS kernel or even apply OS vendor supplied updates to fix something. Now all of a sudden your product X support agreement is void because the platform isn't certified any longer.

Re:waiving your support contract?

[Kjella]

Well, what do you expect? The whole thing is about a CentOS user complaining about how Red Hat is doing a poor job supporting RHEL. What do you know, if he and others actually paid Red Hat for support instead maybe they'd have money to actually do support. Everybody wants something for free, but then deal with what you're getting. Under the GPL he and CentOS is free to leech off what Red Hat is doing but they're basicly bitching their free support isn't good enough. You think I should go on the LKML and bitch to Linus about fixing my bug, because Ubuntu uses his code and I'm using Ubuntu? It would make about as much sense.

As for your other points, I do work with some other enterprise software and if you've ever tried getting support on anything you've tweaked yourself, he's either never tried it, not mentioned the tweaks or been extremely lucky. Anything outside the supported environment, any custom modifications you've done and they'll tell you to revert to the supported version and reproduce, and they won't even touch it until you do. I can't exactly say I got a broad experience with support contracts to speak of but my impression is that it's general practise and that if you want them to support your custom modifications you'll be paying through the nose for real enterprise support. My guess is they had something like regular business support? If so, I'm not surprised...

As for updates, expect that they will ask you to reproduce on the latest version, they almost always do. One solution that may be possible is to upgrade a test system and show the error is still there, I have done that sometimes in the past. One of the issues I imagine Red Hat has is the broad number of packages - in my case I'm talking about a single product with a few major service pack versions, if you got continous patches like a distro I imagine it's difficult to support every combination. But again, I'm sure they'd do it if they got paid enough. That said, if you are running an enterprise distro with only security hotfixes they're usually the kind I'd rather apply than not...

Re:waiving your support contract?

[Anonymous Coward]

This is how business works. If you get a little company with a problem (or a group of non paying users), they'll (maybe) get around to fixing it sometime...and only if you follow their rules to the letter. If you get a big company with some clout that could do damage to a bottom line if they left, I guarantee you developers will be working weekends to debug/fix the issue. All customers are created equal, but some are more equal that others...

Re:waiving your support contract?

[prisoner-of-enigma]

No, it doesn't. The user isn't paying Perl for support, he's paying RedHat. The value of the support he's receiving is not (in his opinion) equaling the cost of said support. This would apply whether it was OSS, FOSS, CSS, or any other variation. The fact that Perl is a FOSS project is irrelevant to a discussion about paid support from a third party no involved in creating Perl.

Re:waiving your support contract?

[Yahma]

The fact he's using CentOS already implies he doesn't have a support contract from RedHat. So what's the big deal?

Re:waiving your support contract?

[Znork]

Red Hat does *not* tell us: "Oh, I'm sorry you're not running the latest support pack, no support for you."

I've found it's more likely that _I'm_ telling customers that we should patch to fix problems they have with Redhat systems than Redhat telling me.

We're still running AS4U4

Heh, we have some AS4U2 systems, with only selected patches applied. I suspect some customers have used software from 'enterprise' vendors who don't have quite as strict don't-break-stuff policies with their updates. I can sympathize; I've had vendors change actual major versions of software in 'patches', rather than backport bugfixes.

So I have to agree with you, Redhat is a rare pleasure to deal with in the enterprise software support space.

Re:waiving your support contract?

[FishWithAHammer]

They're not that unusual (I do some of it on the side for local businesses, myself), but they are generally not enterprise-level support systems.

Re:waiving your support contract?

[FishWithAHammer]

CentOS on the dev's workstation, RH on the test/production servers?

Re:waiving your support contract?

[belmolis]

Yes, and on what kind of boxes do you run your Cobol programs, grandpa?

Re:waiving your support contract?

[Anonymous Coward]

they won't support a system that doesn't have the latest updates

I am a Tech Support Engineer at Red Hat. Your statement is incorrect.

We do provide support for older packages (assuming you are running a RHEL version that isn't over 7 years old and even then we usually provide best effort support). However if you hit a known bug when running package version X and we have released a fix in version Y we'll ask you to upgrade to version Y and won't provide a version X.y just for you. Well, you can still apply patch from Bugzilla and compile your own X.y but we can't provide real support for that. Of course, if you get into troubles when upgrading to version Y we'll support you (and provide version Z if you hit another bug).

Now, when we need to quickly find the cause of a problem (and a workaround and a proper fix) it is not unusual to ask if you can reproduce it with latest version, especially with complex packages that change quickly (like the kernel) but we understand this is not always practical and we surely don't systematically ask you to upgrade to latest version.

Re:waiving your support contract?

[PinkFreud]

Even if it is true, the nice thing with a free operating system is that one can at least fix the bug oneself, support contracts voided or not. Try doing the same if there's a problem with Exchange or IIS.

Which begs the question - why bother paying for a commercial distribution in the first place?

Re:waiving your support contract?

[PitaBred]

You're still thinking the proprietary world. Why not install RedHat on both machines? Why would you want a dev environment to be different from the production one? The only limitations would be on any Non-GPL software that may be included with the Redhat install, but I don't think they do that.

Re:waiving your support contract?

[FishWithAHammer]

Not necessarily; they may just not have enough support licenses and would rather go with something community-supported for the workstations.

Re:waiving your support contract?

[jimmyharris]

If you read the blog post, he explicitly says: "We use OS X boxes for development and Centos 5.2 for production", so no RHEL anywhere it seems.

Re:waiving your support contract?

[the_B0fh]

I find that *VERY* hard to believe, having worked with them when I was with a F100 company, and again, when I was with a F1000 company (which later got bought out by a F50 company, but that didn't count, since RedCrap didn't know about that).

It was always "have you rebooted" and "have you updated", and when RHELL 4 -> RHELL 4 U1 came about, their AMD64 bit support/packages sucked donkeys balls, and again when U2 came out. I had to spend 3 weeks debugging some crap for them. Oh, yeah, especially wonderful was when up2date crashed and killed the rpm database, and the recommendation was reinstall. Luckily my boxes were designed to be reinstalled quickly, but damnit, this is *NOT* enterprise support.

Re:waiving your support contract?

[the_B0fh]

Ummm... other than shiny pictures, what is the difference between redhat and centos?

Oh, support. Wait, you mean, a bug in centos isn't a bug in redhat until someone magically buys a redhat license? Now I've learnt something new.

Re:waiving your support contract?

[the_B0fh]

Why not? What if you used a package that used the Perl that came with Unix Services that Microsoft provides? And the issue got fixed by installing Perl from source, or from ActivePerl?

Re:waiving your support contract?

[rcw-home]

More, they won't support a system that doesn't have the latest updates

In my career I've gotten that from Cisco and Dell too. At least with the Cisco incident it was a software bug, not a part replacement.

If you want to pay for enterprise support. go ahead. You can definitely pay for it.

If you want to get the support you need, you need to find people with the knowledge, skill, and means to care about and fix your problem, even if it's otherwise a waste of time for the vendor.

And that's what makes the whole damn industry such a crapshoot.

Re:waiving your support contract?

[Krunch]

I believe that the problem with such a scheme is that it doesn't scale very well. I can't really imagine a company becoming very big selling only customized, paid by the hour support. Unless it becomes a consulting company.

Re:waiving your support contract?

[no1home]

Probably bad form to reply to myself, but...

There have been plenty of good comments here, but there are several that follow a certain theme: "What do you expect for free support?"

I am (as are most of us, I believe) talking about enterprise level support, as in pay up front for a guaranteed level of service. This has nothing to do with the free support sometimes offered for those who use the OS for free. Two very different beasts, so I wanted to clear that up.

Re:waiving your support contract?

[belmolis]

Flamebait? Not at all. Some moderators evidently have no sense of humor.

Re:waiving your support contract?

[kelnos]

The user is using CentOS -- apparently he isn't even paying RedHat for support, and is still bitching about it.

Re:waiving your support contract?

[FishWithAHammer]

In which case, he is retarded. (His blog seems to be blocked at work for me.)

Re:waiving your support contract?

[Pharmboy]

I think the actual bitch is the fact that RH has known about the bug for most of a year and hasn't fixed it. RH==CentOS for the binaries, and he doesn't expect CentOS to fix the bug, but is instead surprised that RH, who does get paid for this, hasn't fix the problem.

I just swapped over my first server from OLD RH to CentOS, that that would explain the perl performance. I am shocked that RH hasn't fixed it (my reason for leaving RH goes back many years). They don't owe ME (or the original author) anything, but they DO owe their own customers. Me, I will just compile Perl from source and install it in a different path, no biggie.

Re:waiving your support contract?

[Anonymous Coward]

To be fair to Prakash, he's not directly complaining about Red Hat support. The article written about the blog post does though.

Re:waiving your support contract?

[hughesjr]

you CAN NOT install RHEL on machines that you do not have support for. You wave that right when you have any RHEL licenses. So, they can install RHEL if they have a valid license and they can not if they don't. CentOS is installed on an estimated 2 million machines world wide because of this. This issue is in RHEL and reproduced in CentOS, so it would not matter which one was installed.

Re:waiving your support contract?

[Sun]

I find it hard to believe that businesses such as ours are unusual.

Owning such a business myself, I agree with this statement. I also think it is irrelevant.

While you and I know that your company can give a service which is as good, and probably is better, than what RedHat can give (for simple people/client ratio reasons), the PHBs have not had that lesson learned yet. They are looking for "big names" support, and in the case of Linux, the obvious place to search is RedHat. The second RedHat do not provide the service one would expect them to, the general atmosphere in the business world becomes "you cannot get enterprise support for Linux".

Which is, really, the only reason Debian isn't doing better in the corporate market. Its QA level is at least on the same level as RedHat's, the long upgrade cycles are not, generally, a problem for corporations, and it certainly offers many more packages out of the box. The only problem is that the corporations, and even more specifically, the 3rd party ISVs, don't think of it as a "company supported Linux distribution", and don't account for it. This is despite the fact that it is not, generally, difficult to get paid support for it.

Shachar
Full disclosure: I am a Debian Developer.

Re:waiving your support contract?

[sjames]

They weren't comparing apples to apples. If you want someone to support your one-off Linux installation, you need to pay someone for that custom support. That will be much more expensive than support for a bog standard distribution. It's perfectly understandable since there's little economy of scale when each client's system has custom versions of critical packages (possibly even with local patches).

Some companies will support that as part of a standard support contract, but that's because they have built the dedicated hours into the price up front.

Enterprise is not exactly a well defined term. It's most common usage is as a marketing bullet point meant to imply "really good" but always implying "it'll cost you!".

RedHat is targeting their Enterprise support directly at the PHBs that think every system is mission critical and all complex and enterprisey even though it's just a standard install and yet refuse to pay for the kind of support that will happily deal with a custom patched system.

At the same time, they really should consider re-compiling the perl packages if it's making that much performance difference. At the same time, it's working OK as is for many who will be reluctant to make an unnecessary (for them) change.

The real challenge is finding the right balance between high quality support and sticker shock.

Re:waiving your support contract?

[the_B0fh]

The issue is that there are two kinds of "ENTERPRISE SUPPORT".

One is the one they sell to ordinary people, and where they don't really give a shit to what you need. Someone with RHELL support recently called in a NFS bug, and RHELL people agreed it's a bug. Refused to fix it. Told him to wait for upstream to get the fix.

The second kind of ENTERPRISE SUPPORT is the kind they sell to extremely large companies, the ones where they are afraid that the PHBs will one day wake up saying - wait a minute, you guys fix all our issues[1], why are we paying RHELL shit loads of support money? It's not the money leaving that's the problem, it's the publicity from it.

[1] And if this doesn't sound like you, that just means you're a PFY no matter what you think. These companies always have bofhs in high levels, usually with names like "FELLOW" or some such, making more money than some VPs. The F100 company I referred to upthread has a few of these, and treat them very very well.

** Not saying other large vendors don't do the same shit - they do. RHELL just does it a lot more, probably because they're trying to grow the company.

Re:waiving your support contract?

[the_B0fh]

In this case, the moral of the story is that this wouldn't have happened with Exchange or IIS, because there is no middle-man that could mess with their code.

Do you realize that Microsoft licenses code from 3rd parties?

Do you realize that Microsoft has BSD/non-GPL opensource code in their software (for example the ASN.1 code - which is why *THEIR* stuff were vulnerable too from that ASN.1 vulnerability a few years back)

Do you realize that Microsoft sells/gives you GPLed code?! So, all their talk about viral stuff is bullshit. Go look up "Windows Services for UNIX" and see that it's a free download, and also comes builtin with Server.

See also: http://www.unix.com/whats-your-mind/33991-windows-korn-shell.html

Re:waiving your support contract?

[NateTech]

Incredibly smart posting. Linux at our company has had similar problems in deployment (and at previous employers) on servers because of "packaging wackiness" by [insert favorite distro of the day here].

It's like this... you either buy a commercial OS and live with its problems, or you "buy" something open-source and live with its problems.

Those that say *most* companies have the time to crack open the source on open-source things and fix them, are smoking something. We don't. We have a a business to run.

The real hard-core underlying issue here is that software written to work right the first time is exceedingly rare. It's a quality thing. Problem is, if you write it that way to start with, people buy it ONCE and it's over.

Software as a business will always have this inherent problem. Software as part of a bigger solution, doesn't. But it's hard to move to the latter model.

Re:waiving your support contract?

[Pharmboy]

The problem is that the bug is in Redhats OWN compilation. If you compile from source, you don't get the problem. RH themselves is the problem, which they refuse to fix. Regardless of what support you get, you would expect a properly compiled Perl that doesn't run at 1% of speed.

Re:waiving your support contract?

[Anonymous Coward]

Red Hat, et. al. want to be enterprise systems, and claim to offer enterprise support. But they don't perform enterprise support. As indicated here, change something to fix a bug, and you don't get support for that piece anymore

So please tell us, exactly *how* did you manage to change the code yourself with proprietary software and still maintain a support contract agreement with it?

I would really love to know.

That's what you get.

[SatanicPuppy]

Who uses vendor Perl? It's like GCJ; if you don't really need it, it's good enough, but if you really need it, you download the real thing. And like java, it's easy to have multiple versions of Perl on your system.

I guess that's snarky, but seriously. These guys were running a fancy production package on the crap perl install that comes with Fedora? They needed performance (and chose perl?) and they didn't look first at compiling perl from source? It doesn't take long at all, and the benefits are substantial, even aside from not having this bug.

Re:That's what you get.

[timster]

Well, I'm anything but a hardcore Perl hacker -- just use it to pragmatically list some rubbish now and then -- and I've never even heard of compiling your own Perl.

In truth, it's NOT like GCJ in the least. GCJ is a relatively immature JVM built from an entirely different codebase than the Sun JVM. "Vendor" Perl and "real" Perl ought to be substantially the same thing.

Just like all the foundation-level vendor tools, I would expect Perl to be built correctly on any official distro release. I shouldn't need to build my own GCC, my own Python, my own X, or my own Perl.

Re:That's what you get.

[moderatorrater]

I don't care what language you choose, 100x slower is going to make you want to fix the problem.

However, I agree with your analysis that with something that's mission critical, you need to have the flexibility to compile and maintain the program yourself instead of relying on the vendor. In my view, you get the support contract for the things that aren't central to the business and you don't have 15+ people hired for.

Re:That's what you get.

[foobat]

could you explain what's crap about perl in fedora?

according to those perl guys, the fedora guys are quite good

from this blog
http://use.perl.org/~nicholas/journal/37274

"â It has been a different matter for 5.10.0 in Fedora. For that, the maintainer has been very communicative, and so we were able to help him fix problems and get Perl 5.10.0 into Fedora Core 9."

Re:That's what you get.

[castle]

You can have third party applications (GASP Enterprise ones even) that are based on Perl, or are capable of using Perl to access their API. If following this, you install a non OS Vendor version of Perl, and have other problems requiring support your support headache could forseeably increase.
In this kind of scenario (though you are likely someone who trashes Perls performance and think it beneath your favored language(s)) 100x slowdown affects you greatly, and will negatively impact business.

Re:That's what you get.

[SatanicPuppy]

Well, I am harder core than the average schmo where Perl is concerned, so for me it's a requirement...The vendor version is always inferior. Most forums will tell you the same thing.

But like I said, if you don't really need it, it's fine. I doubt the average user would ever run into this problem.

Re:That's what you get.

[jc42]

Well, I'm anything but a hardcore Perl hacker -- just use it to pragmatically list some rubbish now and then -- and I've never even heard of compiling your own Perl.

You certainly don't qualify as a perl hacker! NTTAWWT. ;-)

From the very beginning, the primary (and recommended) way to get perl has been to download the source and compile it yourself. It's true that most unix/linux distros have included perl for a decade or so, and of course it's usually not the current version. But this is only a minor time saver during installation. I've often upgraded to the current perl while installing systems, and I've always found it easy.

The idea that a perl user wouldn't even have heard about the easy availability of the source is sorta surprising. That's the way you're supposed to get it, after all. The standard textbooks start off telling you where to get it (or at least they did the last time I looked at one, which has been a while ;-).

OTOH, perl does have a bit of a reputation for being solid and without any problems, so it's easy to see how someone might be lazy and just use whatever the vendor supplies. I wonder what went wrong with the RH release?

Re:That's what you get.

[Anonymous Coward]

Perl can run plenty fast! See Radiator (open.com.au)

Re:That's what you get.

[Ed Avis]

I always use the vendor perl on Fedora. Why should I pretend that I am cleverer than the people who did the work to package perl for my distribution? And this particular bug does not change my view of the our relative intelligence.

There's also the issue of using rpm to maintain your system, which like many good habits is a bit of a pain at first but soon becomes indispensible. Even when I wanted to upgrade perl to 5.10 (with Fedora 8, since 9 wasn't out yet), I preferred to build 5.10 as an RPM based on the Fedora source package for 5.8 and a new upstream tarball. That way I got a smooth upgrade when the official perl 5.10 package was included with Fedora 9 - the same smooth upgrade as all the other packages in the distro.

Multiple versions of perl on the system? No thanks. This isn't Java where you have a dozen crappy third-party apps all insisting on their own exact version of the interpreter or they won't be 'certified'. Just take one good version - preferably the package that comes with your distro, which you know thousands of others are using and testing - and stick with it.

Re:That's what you get.

[Anonymous Coward]

The reason to run vendor code is support. Suppose your CTO gave you a few million dollars to spend on purchasing a computer cluster and writing some software to parse logs or convert files or whatever these guys were doing. This is a large investment for the company, needed to get some critical analysis done fast every day. Now you've set it all up and you go download some recent version of an interpreter from a group of developers ranging from semi-paid to hobbyists that was released a couple of months ago to run it. If this interpreter breaks, runs slowly, leaks memory, or otherwise fails you, your expensive cluster is sitting idle, and your deadlines aren't being met, who is to blame? Only you. Even worse, for you the only problem is that you might get fired, but for the company, there will be serious trouble explaining to investors and customers why some core part of the infrastructure isn't working and the company can't meet the consumer demands it said it would. This is why businesses buy software with support. If you're a hobbyist running a personal project or even some admin at a real business doing something non-critical, and you want to use random recent software you downloaded off the net, go ahead. But if your business depends on it, it's worth having a team of experts ready to help with any issues as soon as you call, as well as the knowledge that your software has been built and tested by a proven profitable company and the potential for getting compensation if things go wrong.

Re:That's what you get.

[Christian Smith]

... so it's easy to see how someone might be lazy and just use whatever the vendor supplies.

Wow, I wouldn't want you managing my servers.

Home compiled software can easily be a source of security holes, as tracking what you have compiled versus what is patched using vendor updates adds significant management overhead and another point of failure. As an example, a popular open source project had its website compromised because the maintainer used a self compiled version of CVS, and forgot about it. Had the maintainer used a vendor CVS, the security hole would have been patched by the vendor update.

Lazy is good. If you can do the job and be lazy, that is a win-win.

I wonder what went wrong with the RH release?

The RH bugzilla ticket indicates this as the initial issue.
http://rt.perl.org/rt3/Public/Bug/Display.html?id=34925

So it appears RH have not applied this fix, perhaps because the patch is included with a more cutting edge perl than is considered safe for inclusion with RHEL. Certainly, it looks like it was fixed in perl 5.9, but that may be an experimental branch more akin to the old 2.[135] linux kernels (and no vendor would have touched them in an enterprise targeted distribution.)

Re:That's what you get.

[amorsen]

The vendor version is always inferior.

The vendor version in this case has a bug fixed. The bug caused incorrect behaviour. In this case the vendor version is only inferior if you prefer fast but incorrect results. There isn't anything wrong with preferring fast incorrect results over slow correct results, but most people probably want slow and correct to be the default if given the choice.

Fast and correct always wins, and the real Perl hackers are working on that. In the meantime we take what we can get.

Re:That's what you get.

[jd]

In general, that could be said of (almost) every single software package of any substance. If you want it to run well, you have got to roll your own. Well, almost. In theory, there's nothing to stop a vendor with a decent server farm gathering your system info then compiling the RPMs for you for that specific system, and/or having a set of stock ISOs rolled for, say, the five most-popular systems.

A central build has the advantage over something like Gentoo in that vendors can usually afford better horsepower and can auto-tune the options per application better than the average Joe could ever hand-tune them. It's also more supportable, as the vendor then has the exact information for each package, as they would have had they rolled the RPMs from their own default configurations, which a totally user-defined setup would deprive them of.

There's also the dependency hell and the namespace clash that "standard" distros suffer from all the time. I've never come across a distribution YET that supplies stock binaries that is capable of supplying ones that actually work together. First rule, guys, is that if package A is rebuilt, ALL packages in which A is directly or indirectly a dependency should automatically be rebuilt. It should not be possible, using JUST the stable packages (DEB or RPM), to get into a situation where packages barf or cannot resolve their own dependency requirements.

(I won't even get into the issue of broken packages in the stable updates, where you cannot complete an install or a deinstall because the flippin' scripts don't work and don't cleanly handle the case where something breaks, effectively barfing over the disk and the package database.)

The more I use Linux, the more I am convinced that the distros out there are operating on a flawed assumption. They work great, but only when that assumption holds true, but will catastrophically fail outside of those bounds. This assumption is that people will use the distro with a relatively narrow aim on relatively generic systems.

Re:That's what you get.

[Curien]

It's not a Fedora vs Red Hat thing; it's a 5.8 vs 5.10 thing. Apparently, there are different package maintainers for the different versions, and the 5.10 guys are much better about working with the upstream.

Re:That's what you get.

[Bill_the_Engineer]

In my view, you get the support contract for the things that aren't central to the business and you don't have 15+ people hired for.

I disagree. You pay Red Hat to provide a baseline server with all of the provided applications and languages. You pay Red Hat to provide timely updates for their distribution.

You pay your 15+ staff to work on a custom application that may depend on something Red Hat provides.

Now you may need to custom compile something as a short-term solution, but if Red Hat can't do something as simple as keeping their Perl interpreter working correctly I would seriously reconsider paying them any more money.

Think about it. If your paid staff has to make a custom compilation of a vendor supplied application (and consequently keep it updated) then what are you paying Red Hat for? The days of paying Red Hat out of the goodness of our hearts are over. It's time for Red Hat to act like one of the big boys and earn their money.

Re:That's what you get.

[Anonymous Coward]

Home compiled software can easily be a source of security holes

It can also be a significant improvement, as you can build it with exactly the options you need, and no more.

Re:That's what you get.

[chromatic]

Fast and correct always wins, and the real Perl hackers are working on that.

No released version of Perl ever had this bug. Red Hat pulled a patch from a development version of Perl and maintained it over released versions of Perl which did not need it. That's the source of this bug. The Perl developers fixed this bug before releasing the next stable version of Perl.

Re:That's what you get.

[ppanon]

A bunch of administrative scripts in Redhat are written in perl and python. Don't know if it's still the case, but a few years ago, it used to be that if your upgraded to the version from the next major release, some of them would break. So you could live with the current state of things, or you could rev the O/S. It's called stability through configuration management. Sometimes it's a bit frustrating, but in 99% of cases, it's a win.

Re:That's what you get.

[jc42]

Wow, I wouldn't want you managing my servers.

Home compiled software can easily be a source of security holes, as tracking what you have compiled versus what is patched using vendor updates adds significant management overhead and another point of failure.

Funny; pretty much all my past employers have had the opposite policy: Anything of importance, especially customer-facing servers, are rarely or never automatically upgraded from vendors' releases. They all seem to have learned this lesson the hard way, from having their servers down (or compromised) after installing a vendor's upgrade. So they all insist that upgrades be first tested in isolation in a lab setting. Usually this includes compiling from the sources when sources are available. I always recommend this so that we can enable and/or install logging/debugging features that vendors often suppress for the slight performance improvement this usually gives. But they usually seem to have lots of experience telling them to not trust vendors' upgrades without testing.

And, of course, the security folks have long been telling us all that if you want a secure system, one of the primary rules is that you never run binaries that you haven't compiled yourself. That's just inviting vendors to slip backdoors and spyware into your system. But, of course, this is irrelevant if you don't have employees who are able (and have the time) to study the code. In that case, you might as well just install your vendors' upgrades, and hope that they're honest. And you will get bitten by this sooner or later.

After all, just a year or so back, we had the entertaining example of Sony sneaking rootkits into customers' computers via an audio CD. Who ever thought that you'd have to watch out for a case like that?

Re:That's what you get.

[rgviza]

> and I've never even heard of compiling your own Perl.

I've been doing this for a very long time, in addition to compiling my own php, apache etc.

If it's exposed, I compile it. If a 0day hits, I don't have the luxury of leaving the fate of my production security in some vendor's hands.

At the negligence trial, where I'm being prosecuted because my box got pwned, the plaintiff's attorney is going to ask why I didn't fix it if I could. I can, so I do.

If you are moving from packages to compiled stuff, it takes a lot longer than it does if you already operate this way.

The last SSL worm beat me over the head with the importance of this.

While compiling doesn't make you more secure, it sure as hell allows you to become secure faster *when* something happens and the vendors drag their feet. It beats unplugging your servers until a fix is available, which was my other option when this happened.

-Viz

Re:That's what you get.

[Anonymous Coward]

Nope, I think it's a safe bet that the vendor version is inferior.

Sure the stock Perl may be fine for quick and dirty. But for major projects- you're going to have to build Perl the way you want it to be built. When I used to work for a Perl dev team, not once did we find that the vendor version adequately met our needs. Granted, it wasn't Red Hat we were using, but I'd say it's not a bad guess as to what other platforms are like.

Plus, building your own Perl is usually not that big of a deal. Why not build Perl the way you want and sidestep all of the potential problems?

Re:That's what you get.

[mislam]

We compile and install our own perl and the needed modules. We do not depend on the vendor.

Re:That's what you get.

[jc42]

There isn't anything wrong with preferring fast incorrect results over slow correct results, but most people probably want slow and correct to be the default if given the choice.

Well, I'd be a bit careful about making such general statements. There is evidence that people aren't generally that intelligent.

I remember back in the 1970s, when I was at a large university that shall remain unnamed, and a bunch of CS people did a detailed study of the Fortran that accounted for fully half the runs on the campus's central mainframe (which shall also remain unnamed). They found that fully half the runs produced at least some incorrect output due to undetected integer overflows. The hardware gave interrupts for floating-point overflows, but for integers, it just set a flag bit, and you needed to test that flag to catch overflows. The compiler had an option to generate such tests, but it was off by default. The vendor said they did this because they had found that most customers preferred faster code.

The local gang didn't believe this, so they did a bit of a survey. They asked lots of users of the Fortran code whether they would prefer their programs to catch all arithmetic errors if this meant that the code ran slower, or if they would prefer faster code that sometimes didn't catch errors. Roughly 90% of the people they asked this said that they'd want the faster code. Later on, I ran across references to similar tests at other schools, with similar results.

Personally, I was shocked by this. This mainframe was used to do the computing for most of the scientific work on campus, and scientific computing was almost entirely done in Fortran. So half their data runs had undetected incorrect output. They now knew this, and they still preferred the faster speed to correct output.

Somehow, I suspect that this situation hasn't changed. I've dug into various programming languages since then, to learn how they handle this and other potential sources of erroneous results. Most current languages still ignore things like overflow flags by default. Some have no way of enabling the tests of such flags.

Yes, I know lots of ways of explicitly testing for such errors myself. I've done it a lot, because I know I can't rely on others to enable the builtin tests (when they exist) when they recompile the code. But when looking at other people's code, I almost never see anything that will detect overflows. When you're N levels deep in function calls, you usually have no way of verifying the possible range of the current function's args, so there's no way of proving that an overflow can't happen.

Sometimes I'm amazed that our systems run as well as they do, given this sort of nonchalant attitude towards known sources of hardware errors. And I do a lot of paranoid, defensive programming, even though I know that my employers probably don't want it because it slows down the software.

Re:That's what you get.

[Anonymous Coward]

Vendor versions of everything are inferior. I always build my own copies of servers and tools that are heavily used.

Re:That's what you get.

[Anonymous Coward]

you are completely *wrong* about this. the vendor added a bug to an otherwise working perl release. They took untested code from the source tree and merged it in before the developer who checked it in was done. Just look around in the comments to this article and you can see that.

Re:That's what you get.

[amorsen]

When I used to work for a Perl dev team, not once did we find that the vendor version adequately met our needs.

This is certainly making me reconsider whether Perl is an appropriate language to write anything in. Giving up on vendor-provided updates and going back to compiling my own isn't going to happen. This is 2008, not 1995.

Re:That's what you get.

[Anonymuous Coward]

Are they still using a threaded perl on RH/Fedora/Centos ?

I used to know some funny stuff about how threading works in perl.

Briefly, it's a joke. If you want your perl to be at least 2 times slower and 4 times as bloated for a typical workload, you compile it with ithreads support. And nobody wants threads support - you should be plain /crazy/ to use a perl threaded application in production.

Re:That's what you get.

[div_2n]

That's what I did when they were using old snapshots of Samba that had bugs that couldn't be fixed by tweaking settings. I went with Ubuntu instead.

The beauty of Ubuntu is that you don't have to pay to play and yet still get security updates. You can buy support for servers that need it and buy support for those that don't should a need arise.

Re:That's what you get.

[lemox]

and then you realize you just broke half of your vendor's maintenance scripts and administrative tools.

Re:That's what you get.

[timeOday]

Yes this is bad, and yes it's RedHat's fault. But I will say, it's much harder to regression test for performance than for correctness. Actually it would be very nice if VMWare could make a common platform for performance testing, that runs equally fast/slow on any hardware exceeding some minimum requirements.

Re:That's what you get.

[Karellen]

Reminds me something I heard a wise hacker say once, when someone tried to convince him that their new version of some code was better that his, because it ran in 10% of the time his did but produced (slightly) wrong results in a few cases...

"If it doesn't have to produce correct results, I can make my version use no memory and run in zero time."

Re:That's what you get.

[chromatic]

p5p uses a tool called perlbench to check performance periodically. It doesn't cover every combination of language features, but it's a good baseline.

Re:That's what you get.

[Anonymous Coward]

I agree with you about Red Hat's support being lacking in this case but it seems more than a little odd that the blog poster is running CentOS and isn't paying Red Hat at all. He's hardly got a right to be complaining about their support!

Re:That's what you get.

[dag]

Think about it. If your paid staff has to make a custom compilation of a vendor supplied application (and consequently keep it updated) then what are you paying Red Hat for? The days of paying Red Hat out of the goodness of our hearts are over. It's time for Red Hat to act like one of the big boys and earn their money.

So you buy Red Hat support only for perl ?
No you don't ! Please be honest.

Also, bugzilla is not a customer support tool. If you want Red Hat to fix a bug then first make sure you actually pay for support and secondly use the normal channels for customer support.

Red Hat seldom pushes bugfix releases until the next update release, because you do not want customers to be exposed to new bugs outside of update releases.

Besides, if this was a bug that was exposed by a lot of companies, it would have already been fixed much sooner and was available from the fastrack repository.

Also, as a paying customer, I would not want a support team to pay more attention to people that blog about bugs or Bugzilla, than tickets in the support tool. We are heading in the wrong direction if blogposts get a higher attention than paying customers.

Re:That's what you get.

[chromatic]

This is certainly making me reconsider whether Perl is an appropriate language to write anything in.

That's your prerogative, but I think you may have learned the wrong lesson here.

Re:That's what you get.

[Kjella]

I've been doing this for a very long time, in addition to compiling my own php, apache etc.

If it's exposed, I compile it. If a 0day hits, I don't have the luxury of leaving the fate of my production security in some vendor's hands.

With all due respect, do you take vacation? Sick leave? Have all-day meetings, busy days or crunch times with a mail backlog? Congrats to you if you can pull it off, but I wouldn't put that kind of 24/365 burden on myself. If anything, I'd look for a better vendor...

Re:That's what you get.

[Anonymous Coward]

man, are you dumb.

Re:That's what you get.

[b4dc0d3r]

I think you clarified for me what happened, even if it is not true it is what seems to have happened.

A vendor releases some version of a package, then applies minimal bug fixes to reduce the amount of regression testing that needs done. Or maybe includes its own bug fixes. When it comes time to merge with the next update, how do you decide what to keep and what to merge over? Seems RH does not have the answer.

Re:That's what you get.

[cr_nucleus]

"If it doesn't have to produce correct results, I can make my version use no memory and run in zero time."

That was actually the starting point of XP.

Surprisingly enough, there isn't much reference of it around but it was well described in the XP book i read.
Found one over at ibm:
http://www.ibm.com/developerworks/java/library/j-beck/ (read "creation").

Re:That's what you get.

[Raenex]

You could compile your own version when something breaks, but otherwise use the vendor's. Sounds to me, though, that you're just a control freak, and not really into pragmatics.

Re:That's what you get.

[bar-agent]

That's your prerogative, but I think you may have learned the wrong lesson here.

What's the right lesson?

Re:That's what you get.

[chromatic]

Distrust vendors who don't work with upstream.

Re:That's what you get.

[Bill_the_Engineer]

So you buy Red Hat support only for perl ?

No you don't ! Please be honest.

Of course not. But Red Hat does include the Perl packages, so therefore they should support them. I mean we complain about not getting "unlimited" bandwidth from our cable system, but it's OK not to get complete support for a product that we paid for, especially given the fact that the product itself is free.

I know it hurts, but let me lay out the issue for you:

Instead of downloading Linux for free, an enterprise pays Red Hat to provide an actively supported distribution with an expected level of reliability and performance. Red Hat does not charge one-time fees, instead they offer subscription based support.

So if a company is paying Red Hat yearly subscription fees per seat, shouldn't they expect Red Hat to fix the problem?

As for the source of the bug report.... That's lame. I would hope that Red Hat is proactive with their issue resolution...

yum

[ilovesymbian]

How difficult is it to install the latest Perl with yum install perl?

You'd get the latest version anyway. :)

Re:yum

[jm4]

It would probably be pretty difficult considering Red Hat doesn't use yum, but that's really just semantics. The main problem is the vendor-blessed version- and the one in the repository whether you use yum on CentOS or up2date on Red Hat- is the one that contains the bug.

Re:yum

[foobat]

the latest version supplied by redhat that is. Which is what the problem is all about.

Re:yum

[ccharles]

Actually, they do. yum is the default package manager in RHEL ever since version 4, I believe. In any case, you are correct that the repository that yum talks to contains the buggy Perl. It looks like this should be fixed for RHEL 5.3, but I wouldn't bet on it.

Re:yum

[jimmyharris]

Nope, up2date was/is still used in RHEL4 - yum was introduced for RHEL5 (thank the Flying Spaghetti Monster).

Red Hat on CentOS?

[Anonymous Coward]

Why blame Red Hat when it's a package running on a non-Red Hat system?

Re:Red Hat on CentOS?

[Ritz_Just_Ritz]

Um...because CentOS is basically Redhat Enterprise Linux with the proprietary bits stripped out. So if the bug appears in RHEL, it will also appear in CentOS.

Personally, I always stash a copy of the latest version in /usr/local as another poster already mentioned. I've tripped on this bug before as well.

Cheers,

Re:Red Hat on CentOS?

[ccharles]

I can confirm that the bug also exists in the current 5.2 release of Red Hat Enterprise Linux.

Re:Red Hat on CentOS?

[KaZen]

Except that if they were paying Red Hat for support, they would have been given the hotfix when they called in to diagnose the problem.

It seems a little odd to complain about Red Hat support, when in fact they weren't paying Red Hat for support.

Re:Red Hat on CentOS?

[Anonymous Coward]

So, basically, CentOS is LIKE Red Hat, but it's not.

Basically, a raw steak is like a cooked steak, but I wouldn't eat the raw steak.

Re:Red Hat on CentOS?

[Endo13]

No, the steak is the same. It just didn't come with the house vegetables and mashed potatoes.

Re:Red Hat on CentOS?

[fracai]

Mmmm... vendor perl.

Re:Red Hat on CentOS?

[Anonymous Coward]

CentOS is basically Redhat Enterprise Linux with the proprietary bits stripped out.

Just what are these proprietary 'bits' - surely more than Redhat's trademark, right?

Re:Red Hat on CentOS?

[Anonymous Coward]

Um...because CentOS is basically Redhat Enterprise Linux with the proprietary bits stripped out.

Proprietary bits? You sir have a vast misunderstanding of what Red Hat Enterprise Linux is.

Re:Red Hat on CentOS?

[neurovish]

No, the steak is the same. It just didn't come with the house vegetables and mashed potatoes.

...and you can't send it back to the kitchen if it comes out overdone.

Re:Red Hat on CentOS?

[Anonymous Coward]

How obtuse can you be? It is exactly the same distribution minus the proprietary drivers and bits that cannot be freely given away. So if you're an RHEL customer, you get the exact same bug. When Redhat fixes it, CentOS users will also get the fix.

So if Redhat's "value" is that they'll ship broken components and supply you with fixes only if you complain, then I'm not really sure I see the value in that.

Functionally, CentOS IS RHEL with the caveat that there is a slight delay between Redhat releasing patches and CentOS users also receiving them. It's no accident that many hosting providers have opted for CentOS instead of RHEL.

Good thing it was open source

[stjobe]

Yeah, well, good on mr Prakash I guess. Good thing he had the option of rebuilding from source, I can think of a few other operating systems and applications where that simply isn't an option.

So, score one for open source I guess, headline be damned.

Re:Good thing it was open source

[Anonymous Brave Guy]

So, score one for open source I guess, headline be damned.

OK, you win a point for open source, but only if you promise to criticise everyone who ever writes a snarky one-liner about how wonderful the Linux world is compared to any alternative platform because you can just "{package manager} {get command} {package name}".

Oh, and if you explain how all the companies forming behind major OSS projects like Linux are actually doing anything valuable in exchange for the money they get paid if they can't even provide robust support for the packages concerned and a high quality default installation. If the default packages don't work properly and you just have to rebuild from source anyway, why not just set up your own Linux installation with only those packages you actually care about and use, rely on the community for support, and let the Linux distro companies go bust?

Re:Good thing it was open source

[Chandon Seldon]

OK, you win a point for open source, but only if you promise to criticise everyone who ever writes a snarky one-liner about how wonderful the Linux world is compared to any alternative platform because you can just "{package manager} {get command} {package name}".

Why are those benifits mutually exclusive?

Over 99% of the time you can install common software with a single apt/yum command and have it work perfectly. Occasionally, when a packaging mistake is made, it's reasonably straightforward for an expert to install the software manually.

This is the general usability case for FOSS software: In the normal case, it Just Works. In the exceptional case, you can make it work. With proprietary software, the supplier's business model frequently makes you sacrifice *both* of those properties. You have to deal with license compliance stuff to make it work at all, and if it breaks you don't have what you need to fix it.

Re:Good thing it was open source

[Anonymous Coward]

Good thing he had the option of rebuilding from source, I can think of a few other operating systems and applications where that simply isn't an option.

Err... isn't perl open source on *all* platforms?

Don't throw out the baby with the bath water

[SirGarlon]

Just because Red Hat made one high-profile mistake, doesn't mean their support service is without value. Jump to conclusions much?

Re:Don't throw out the baby with the bath water

[canUbeleiveIT]

Just because Red Hat made one high-profile mistake, doesn't mean their support service is without value.

Perhaps not, but I know that they will never get another dime of my money.

For years, I always purchased Red Hat even though I never had occasion to use the support that came with it. I was (and still am) bought into the FOSS concept and wanted to make it work for me and my business. But I stopped sending RH my money sometime about 8.0, when I called their support to try and get some help with a printer issue. I would have been satisfied if they had been able to get either one of my printers (HP LaserJet 1100 or LaserJet 4L) to work with RH. A surly woman with almost unrecognizable English--obviously reading off of a cue card--tried for a few minutes and then dismissed my support case with the comment that "RedHat doesn't work with all printers." When I mentioned that I had paid for the RedHat just so that I could have support, she hung up on me. I called back to get another support person with an equally incompetent and rude tech.

Eventually, someone at experts-exchange.com gave me the answer to my problem. Now I just download Centos and if I need support, I pay someone on a case-by-case basis.

Re:Don't throw out the baby with the bath water

[porkThreeWays]

It's a jump to conclusions mat!

Re:Don't throw out the baby with the bath water

[Mr. Underbridge]

Just because Red Hat made one high-profile mistake, doesn't mean their support service is without value. Jump to conclusions much?

That's not the point. The point is that 1) Red Hat won't fix it, and 2) if you fix it yourself, you invalidate your service contract.

There appear to be workarounds as mentioned, but it shouldn't come to that.

Re:Don't throw out the baby with the bath water

[Anonymous Coward]

Red Hat issued a hotfix for RHEL customers. So they *did* fix it. It'll eventually make its way to CentOS. I guess you get what you pay for.

Re:Don't throw out the baby with the bath water

[Just Some Guy]

Eventually, someone at experts-exchange.com gave me the answer to my problem.

You had me until then.

Re:Don't throw out the baby with the bath water

[dacut]

Oh, they've made a bunch of lower-profile mistakes with the company I work for. We have about 50k RHEL installs across our sites; actual issues with their image (especially the kernel "improvements" in RHEL3) that affect production servers are routinely ignored or dismissed.

We've considered dumping them, but this is the main supported Oracle platform -- and we're too invested in Oracle to switch away. I realize the plural of anecdote is not data, but our experience is not unique.

Re:Don't throw out the baby with the bath water

[Anonymous Coward]

LOL. Take a look at the bug report lists.

Comment #18 From Peter 2008-08-08 04:55:19 EDT -------

> Since Bugzilla is not a customer support tool, it's unlikely that you can
> receive response in this way.
>
> I recommend to contact Global Support Services
> [https://www.redhat.com/apps/support/] to escalate this request.

I did it 3 days ago (many of my friends over a week ago)
and RH does not even bother to comment on it.
RH Support is not the best place to ask for support.

------- Comment #26 From Peter 2008-08-21 07:07:02 EDT -------

I just received hotfix from RedHat, the test script above runs 50 times faster
on patched perl comparing to default on RHEL4.5 but i still see slowness when
using perl::DBI with Mysql
I will hopefully post some more detailed examples soon

They have the patched version out (perl-5.8.8-15.el5_2.1), and apparently its very fast now. So I suppose the only thing lacking is RedHat support effectively communicating what they're doing. Perhaps they don't think this bug was particularly high profile as it has been around for a while before anyone noticed, or perhaps Peter thought contacting bugzilla instead of support woudl progress it faster (assuming he has a support contract...)

Re:Don't throw out the baby with the bath water

[jonny55555]

In my small and humble experience I've had several failures that were right in the Enterprise Linux flavor from the Red Hat world themselves.

A couple years ago I was installing 4 with a license contract for a large state college (the largest employer in the state actually) and Red Hat EL4 couldn't even complete an installation without requiring an update to allow for the rest of the GUI setup which if you can't get to, makes it almost impossible to finish the setup.

So now, modern day, I am working with RH EL5.2 and the SAME THING happens. I try to call since I am on a demo license this time (doing the work at home for a company I am starting up) to let them know about the same problem with their installer and another problem they have with the kernel line in grub for the type of processor (completely wrong placement of 'noapic' when you are setting up their virtualization kernel). They tell me they can't take my advise or suggestion until I pay them money. Right - I have to pay them to help them fix their product. Guess that's what companies do anymore, try to convince their customers not to be customers unless they NEED them and then they are willing to pay exuberant fees anyway.

Re:Don't throw out the baby with the bath water

[Anonymous Coward]

You had an individual license over half a decade ago as the .com bubble was still bursting. The company has completely changed since then. Hold a grudge much?

Re:Don't throw out the baby with the bath water

[kelnos]

Good thing they added the hyphen there... 'expert sex change.com' might not attract the same kind of audience.

anonymous coward

[Anonymous Coward]

I would be thankful I had the option of fixing the issue myself. Support contracts on freebsd and linux have always been overrated in my mind when you
are used to solving issues yourself or with the community that uses it. Mailing lists are more helpful to me then the vendor in most cases.

It's not just perl

[Reality Master 201]

I swear everything runs slower with Redhat compiled binaries, even basic shell utils like grep, sed, and awk.

Re:It's not just perl

[Anonymous Coward]

I totally agree! I thought Ubuntu was slow, but then I tried Fedora.. boy was I wrong! How can anyone even use Fedora or anything Red Hat based productively? How the hell did they manage to slow things down like that? Did they compile everything with profiling and debugging on ?

Re:It's not just perl

[tchuladdiass]

I noticed this too, but then found the cause. By default, your language setting is en_US.UTF-8 (echo "$LANG" variable). This includes unicode support, so any text utilities such as grep have a bit more work to do when searching through files (UTF-8 uses 8 bits for normal ascii charcters 0-127, but uses 16 or 24 bits for extended unicode characters). To fix this, change LANG environment variable to "en_US" (http://linux.slashdot.org/article.pl?sid=08/08/29/1423201#
Previewleave off the "UTF-8") prior to running these tools (if you don't need unicode support). To change the language permanently on bootup, change it in /etc/sysconfig/i18n.

Heck, this might even fix this perl "bug" (I haven't looked at it yet though).

Re:It's not just perl

[Reality Master 201]

Maybe, but I'm not completely convinced. I have my locale to en_US.UTF8 on my Gentoo machines and they're still markedly faster.

Re:It's not just perl

[tchuladdiass]

But, are the binaries compiled for UTF-8 support? One way test is to use wc on a binary file -- you will bet a bunch of "invalid multibyte character" errors if it is supporting utf-8.

CentOS it NOT Red Hat

[Anonymous Coward]

If it is "supplied by CentOS" then it was compiled by "CentOS" not Red Hat. Red Hat Enterprise Linux enterprise had a hotfix for this weeks ago. So if Vipul had been using a Red Hat product, he would not have had this problem.

Re:CentOS it NOT Red Hat

[Anonymous Coward]

And recompiling doesn't invalidate his support contract; as a CentOS user he doesn't have one.

The summary is bullshit.

Re:CentOS it NOT Red Hat

[ccharles]

I'm fairly certain that Red Hat has not yet released an official fix for this. Also, the bug has been in their Bugzilla for almost a year now. Even if they had released a fix "weeks ago", it would be far too late.

Re:CentOS it NOT Red Hat

[sloanster]

If it is "supplied by CentOS" then it was compiled by "CentOS" not Red Hat. Red Hat Enterprise Linux enterprise had a hotfix for this weeks ago. So if Vipul had been using a Red Hat product, he would not have had this problem.

A hotfix weeks ago for a problem that's been there since 2006? So he would have had the problem only slightly less longer than he did. Let's face it, redhat dropped the ball on this one.

If they aren't fixing simple bugs...

[russotto]

...then what good is the support contract anyway? Either stop paying for it or make enough of a stink that they fix it. Of course, RedHat being an enormous company won't pay attention to anyone making a stink unless they are an enormous customer, which is a problem in both open-source and proprietary worlds. At least there's a workaround in the open-source world, one better than object patching the binary...

Re:If they aren't fixing simple bugs...

[pembo13]

Please see the referenced bug report before claiming they aren't fixing bugs.

Re:If they aren't fixing simple bugs...

[KaZen]

Except he *wasn't* paying for the support contract!

By definition if you are running CentOS, you are saying, "I'm so smart I don't need a support contract. If something goes wrong I'll fix it myself." And then he complains about getting bitten by the "Red Hat Perl Bug".

If Red Hat is so Horrible (And by extension CentOS), why is he running it? If vendor supplied component X is worthless, why isn't he running Gentoo? Oh, that's right, because having a stable platform is *valuable*. But, it would appear, not valuable enough to pay for.

Re:If they aren't fixing simple bugs...

[FishWithAHammer]

How do you know he isn't running CentOS on his local developer machine and RHEL on testing/production?

Re:If they aren't fixing simple bugs...

[jimmyharris]

Because we read TFBP? It's right there - first paragraph, third sentence: "We use OS X boxes for development and Centos 5.2 for production."

Re:If they aren't fixing simple bugs...

[Anonymous Coward]

What's the good of a support contract when you're not paying it anyway.

This joker is using CentOS, and complaining that since CentOS is broken, RedHat is not doing a good job. Therefore, RedHat doesn't deserve the money he didn't pay for the support contract he didn't buy, but RedHat should still be blamed for the bug.

Funny thing is about 20 of the follow up posts imply that most people who are actually running RedHat can't reproduce his slow results, and another 10 lambast him for shoddy perl coding practices.

But hey, don't let CentOS's shoddy, non-existent support stop you from trashing another company it's leeching for nearly all of their work.

Re:If they aren't fixing simple bugs...

[FishWithAHammer]

I was blocked at work. Mea culpa. :)

Diagnostic perl one-liner

[Anonymous Coward]

FTA: diagnostic perl one-liner

time perl -e 'use overload q( sub {};my%h;for (my $i=0; $i 'main';print STDERR "." if $i % 1000 == 0;}'

Re:Diagnostic perl one-liner

[Anonymous Coward]

You are missing a few things in there....

(Like the test condition, the what to do if we don't meet the test condition, etc)

And I think some of the other things got chopped out as it does absolutely nothing except immediately quit.

The ECODE tag can help a lot in cases like this (I wanted PRE, but it isn't in the allowed HTML list anymore)

(OK, I pulled this from http://community.livejournal.com/perl/150523.html and while I haven't run it, it seems like it should work, as it has the missing parts. (Searched on Google for the "use overload q" thing and found this)

#!/usr/bin/perl
use overload q(<) => sub {};
my %h;
for (my $i=0; $i<50000; $i++) {
$h{$i} = bless [ ] => 'main';
print STDERR '.' if $i % 1000 == 0;
}

stuff into file, then time ./filename
)

Modules

[haeger]

We used "modules" in a similar situation. http://modules.sourceforge.net/

There was a lot of development where I worked before. Things were written with dependancies to specific versions of programs. They would "probably" work with a later version but the developer didn't want to risk that and neither did the business side, so we implemented modules and let the devoper load the version s/he needed.

"module load perl" would always load the latest version but if you depended on a specific version you could always do a "module load perl/5.5.8" which would set environment variables to only get things from that version.

Worked like a charm.

.haeger

My guess is it's the UTF bug

[goombah99]

Well the reason to want to use vendor perl is that perl does some very fancy file management stuff that makes it extremely fast for opening and closing files. So one presume a vendor could optimize this to their kernel.

My guess here is that this is just the recurrence of the UTF-8/16 bug.

Grep and Sort on unix set to UTF-8/16 as opposed to LANG = C (or Posix) are about three orders of magnitude slower on large files. (No I'm not exagerating). Recently this Bug showed up once again on the Leopard on mac.

About ten years ago this showed up in Perl on Redhat. Then it went away. I bet it is back.

I think the reason it comes and goes is it depends on if things are compiled to default to UTF-8/16 or to ascii.

Article is a troll

[wrook]

Cent OS is *not* an OS that Red Hat provides support for. So, in terms of support, you get what you pay for. The bug is fixable by recompiling Perl? Great. Submit the fix to the maintainers. End of story.

But, supposing that you *did* pay for support and you ran into this problem... It's a known bug with low priority. So get them to fix it. You're paying for support. Hold your vendor to their promises.

And if they don't fix it, find another vendor. That's the beauty of open source. If you need support and your current supplier sucks, you can find another.

But it's completely disingenuous to complain that recompiling your Perl binary will void your support contract *when you have no such contract*.

Re:Article is a troll

[A beautiful mind]

Still think it's a troll?

This is what a perl core hacker has to say about the issue:

It seems that there is still a problem with RedHat's packaged perl 5.8."8"**. RedHat seem to have an aggressive policy of incorporating pre-release changes in their released production code. This would not be so bad if they actually communicated back with upstream (i.e. me and the other people on the perl5-porters mailing list), or demonstrated that they had sufficient in-house knowledge that they didn't need to. But evidence suggests that neither is true, certainly for 5.8.x

Let me stress that there has never been this problem in any released Perl, 5.8.7, 5.8.8, 5.10.0, and it won't be in 5.8.9 either when it comes out. The problem was caused by changes I made in the 5.8.x tree that RedHat integrated. End users reported the first bug something like 2 years ago, and RedHat closed it as "upstream patch" rather than reporting back "you know that pre-release change you made, that we integrated - well, it seems to have some problems"

(...)

For their versions affected, RedHat merely need to put out a patch integrating changes 31996, 32018, 32019 and 32025 which FIX IT, are documented as FIXING IT, and are from NOVEMBER 2007.

Re:Article is a troll

[Chirs]

And if they don't fix it, find another vendor. That's the beauty of open source. If you need support and your current supplier sucks, you can find another.

Theoretically, yes. But if your current distro provides board support for a dozen specific hardware boards, you've made version-specific modifications to the kernel and userspace, all your hardware vendors have validated against your current software vendor, and you've gotten your end product certified based on the current software vendor.....in practice it becomes extremely difficult to switch distributions.

Re:Article is a troll

[imroy]

RedHat seem to have an aggressive policy of incorporating pre-release changes in their released production code.

Great, so it's a repeat of the GCC 2.96 debacle.

Oh, and the link went to the wrong journal entry. Here's the right one.

Compiling your own

[UnknowingFool]

Well that's one issue with using pre-built things is that you it may not have been optimized for your usage or in the case of Perl, general usage. I kid! I kid! :P At least open source allowed an option. But I would be careful about compiling your own for everything. There are cautionary examples of how this could lead to problems.

Linux Support Contract

[janeuner]

*facepalm*

RHEL != CentOS

[Anonymous Coward]

hey redhat do you provide any support for CentOS actually

Example of the UTF-8 default "bug"

[goombah99]

here's an example of this on a mac OS 10.5
[CODE]
perl -we 'for (0..1000000) { print rand(1),"\n"}' > /tmp/foo

time sort /tmp/foo
  real 0m36.169s
  user 0m35.731s
  sys 0m0.264s

echo $LANG
  en_US.UTF-8

setenv LANG C

time sort /tmp/foo > /tmp/foo2
0.966u 0.201s 0:01.27 91.3%

SO for a small file it's 40 times faster when not defaulting to UTF-8

Perl on leopard however appears to assume LANG=C bey default and is quite fast. As is grep.

Re:Example of the UTF-8 default "bug"

[jc42]

Hmmm ... I tried that example on my Mac running 10.4.11 and perl 5.8.6, and the two sort times differed only by 1 in the third decimal place.

Maybe I should update the perl on that machine and try it again.

Re:Example of the UTF-8 default "bug"

[goombah99]

apparently this is a leopard "feature".

Re:Example of the UTF-8 default "bug"

[kelnos]

Wow, shit, yeah, this hits me too. On Leopard here, C locale gives me 1.371 seconds, en_US.UTF-8 gives me 41.483 seconds. catted the file to /dev/null first in both cases to ensure the file was cached. Redirected the sort to /dev/null as well to avoid any i/o.

link

[Toffins]

Here's the full story.

easy...

[Anonymous Coward]

There are a couple fixes here... re implement the Perl code in Java and gain 50%, or re implement in C if you are a man.

Using your support contract?

[Anonymous Coward]

If this benchmark was a real issue for your production systems and you had a support contract with Red Hat you would obviously have requested and gotten a hotfix from them already. Done.

That is what support contracts are for to get support and fixes timely even if the generally available package is still going through the full support cycles. Sure CentOS is great and very cheap, but you would go to Red Hat to get timely support and updates if something like this is a real business concern.

Why is it disturbing?

[pembo13]

The summary said it is disturbing. According to the Bugzilla report, the performance issue arouse as a result of a necessary patch. I can see how this would be disagreeable... but how is it disturbing?

Bitten by Perl, not Red Hat

[layer3switch]

Wrong title, wrong article, wrong on just about every level of troubleshooting application performance benchmark.

It only affects x64_86 branch on old 5.8.8 perl package, not just Red Hat.

Re:Bitten by Perl, not Red Hat

[makomk]

Nope - as far as I can tell it's a Red Hat only bug due to a Red Hat specific patch, and it affects all platforms. (Well, the patch is included in official Perl releases, but only together with other changes that avoid the performance issues.)

Compiling your own is the right answer.

[j.e.hahn]

If is a critical to your business, then why are you relying on a general-purpose vendor to provide it to you?

Compiling your own textutils or findutils? That's ridiculous. Compiling your own X11 Server? Lunacy. But if you're building a perl application, it probably makes sense to have tight control of your perl interpreter, down to the compiler options, packages installed and binary builds.

An example from my real life: At some of my employers we've built our own Apache binaries, our own Ruby interpreters, etc. Why? Because we didn't want someone else making version choices for us -- we wanted verified, controlled versions of these key pieces of our software platform.

I know dev teams that build their own GCC because they want very specific feature sets and don't get what they need from their distributions, or because they want conformity across a wide range of platforms.

Control the things that are closest to your application. Let outside vendors cover the infrastructure that isn't. That's just good engineering practice.

Re:Compiling your own is the right answer.

[spaceyhackerlady]

Control the things that are closest to your application. Let outside vendors cover the infrastructure that isn't. That's just good engineering practice.

I agree. Using a generic build of any mission-critical software wastes resources, and every unused-but-still-enabled feature is a potential security problem.

I'm typing this on my development Linux box. It runs a custom kernel, precisely matched to the hardware and my requirements. My policy is that stuff I use all the time (e.g. networking) is compiled in to the kernel, while stuff I don't use all the time (e.g. usb mass storage, udf file system) is modules that I can load as needed. My main applications (apache, python, gcc) are all up to date, and compiled from source. They are a precise match for what I need.

Needless to say, this box rocks.

...laura

Re:Compiling your own is the right answer.

[KaZen]

Sweet. Now Manage 40 boxes just like it. Oh and you have to still get all your normal work done. Yeah, btw we just laid-off two people from your team, so their work is on the rest of your shoulders...

This is why even though I've been running Linux since 1993 (slackware, and I used to recompile all the updates by hand), eventually after the number of machines grew to a certain size, I realized there was no way I could keep them all updated by hand myself. I switched to Red Hat. I get *value* from my RH subscriptions. I'm *happy* to *pay* for that value.

YMMV

CentOS is compiled using the same tools and source

[SuperBanana]

Every time Redhat releases a hotfix, CentOS grabs the source and compiles it. They use the exact same toolchain to compile the exact same source. The only difference between a redhat package and a CentOS package is that CentOS has replaced "Redhat" everywhere, because Redhat started using trademark law to keep them from doing what the GPL entitled them to do (it got so bad that at one point, Redhat was threatening CentOS over even mentioning Redhat on their website.)

Let's keep our eye on the ball, here: this is a known bug, in Redhat's bug tracker, since 2006. Fixes have been commonplace since 2007, and only just now did Redhat get around to fixing the problem. The question remains: what good is Redhat over CentOS (the only difference being logos and a support contract) if they ignore a major performance bug for two years?

Don't link to bugzilla

[MSG]

I don't know how many projects have asked Slashdot not to link to bugzilla. It makes the system unusable for the developers trying to get work done.

Here's the text currently in the bugzilla entry (edited to meet slashdot's filter requirements):

Bug 379791 - perl bless/overload performance problem
Summary: perl bless/overload performance problem
Status: VERIFIED
Product: Red Hat Enterprise Linux 5
Component: perl (Show Red Hat Enterprise Linux 5/perl bugs)
Version: 5.2
Platform: All Linux
Priority: urgent Severity: high
Target Milestone: rc
Assigned To: Marcela Maslanova
QA Contact: desktop-bugs@redhat.com
URL:
Whiteboard: GSSApproved
Keywords: ZStream
Depends on:
Blocks:

Reported: 2007-11-13 07:14 EDT by Nigel Metheringham
Modified: 2008-08-29 10:30 EDT (History)
Fixed In Version:
Release Notes:

Description From Nigel Metheringham 2007-11-13 07:14:04 EDT

RHEL5 perl shows the same performance issues as the Fedora 7 perl did - see
Bug #196836 and Bug #253728

This has been demonstrated in the recent perl update perl-5.8.8-10.el5_0.2

Same fix needs taking across to RHEL, ideally as a update release rather than
waiting for next major release cycle.

I do not have RHEL5.1 to test against right now, but the timing of the Fedora
fixes leads me to believe these would be much too late for the 5.1 release
cycle.

-- Comment #2 From Martin Kutter 2007-11-30 05:24:01 EDT --
The issue can be observed running the benchmark script from the recent
SOAP::WSDL package.

To do so, download SOAP-WSDL-2.00_24 (and its dependencies) from CPAN, run perl
Build.PL && perl Build, cd into benchmark and run perl -I../blib/lib 01_expat.t

This is the Output from RHEL4:
perl -I../lib 01_expat.t
Name "DB::packages" used only once: possible typo at 01_expat.t line 2.
Benchmark: timing 5000 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL): 4 wallclock secs ( 3.48 usr + 0.01 sys = 3.49 CPU) @1432.66/s (n=5000)
XML::Simple (Hash): 7 wallclock secs ( 7.19 usr + 0.03 sys = 7.22 CPU) @692.52/s (n=5000)
XSD (SOAP::WSDL): 6 wallclock secs ( 6.06 usr + 0.01 sys = 6.07 CPU) @823.72/s (n=5000)

And this (with reduced n) is from RHEL5 (different machine, perl-5.8.8-10):

Benchmark: timing 500 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL): 1 wallclock secs ( 0.59 usr + 0.00 sys = 0.59 CPU) @847.46/s (n=500)
XML::Simple (Hash): 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @471.70/s (n=500)
XSD (SOAP::WSDL): 11 wallclock secs (11.34 usr + 0.01 sys = 11.35 CPU) @44.05/s (n=500)

Increasing the number of runs shows the O(n^2) nature of the performance problem
- increasing the number of runs by a factor of 10 increases the runtime for the
XSD bench by a factor of nearly 100:

Name "DB::packages" used only once: possible typo at 01_expat.t line 2.
Benchmark: timing 5000 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL): 6 wallclock secs ( 6.19 usr + 0.03 sys = 6.22 CPU) @ 803.86/s (n=5000)
XML::Simple (Hash): 11 wallclock secs (11.20 usr + 0.02 sys = 11.22 CPU) @ 445.63/s (n=5000)
XSD (SOAP::WSDL): 851 wallclock secs (847.36 usr + 2.28 sys = 849.64 CPU) @ 5.88/s (n=5000)

-- Comment #3 From RHEL Product and Program Management 2007-12-03 15:47:35 EDT --
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the cur

Re:Don't link to bugzilla

[Anonymous Coward]

Yup, that's what it says all right. Just wanted to check to be sure. Thanks.

Re:Don't link to bugzilla

[Anonymous Coward]

Of course it's slow - it's written in Perl ;-)

Old versions in repos: Not unique to redhat

[Spy der Mann]

My distro is PCLinuxOS and the latest available kernel is 2.6.22.something. I've found myself having to compile a lot of packages from source because they haven't been added to the repos.

And I've heard of similar problems in Gentoo.

Maybe it's part of the deficient development cycle of some apps? i.e. no stable versions, and keep fixing bugs and adding features (and bugs) at the same time.

Re:Old versions in repos: Not unique to redhat

[Anonymous Coward]

Gentoo is a *source* distro. You compile pretty much everything. Binaries are available for OpenOffice, Firefox, and perhaps a few other things, but you compile it all. Out of all distros I've used, Gentoo was the fastest. However, it is quite easy to break if you don't have a clue as to what you're doing.

Re:Old versions in repos: Not unique to redhat

[neurovish]

I have that problem in Gentoo all the time...seems like every time I want to install or update a program, I have to compile it.

Netcraft confirms it!

[Anonymous Coward]

Red Hat is dead!

Snark Comment

[Anonymous Coward]

Any clown relying on Perl for performance ... well ... they deserve what they get. Perl is for non time critical applications.

Re:Snark Comment

[Anonymous Coward]

I'll spend the Computer's time rather than MY time any day of the week.

Has happened before

[propanol]

Red Hat has a history of supplying packages with flaky patches. One of the kernel updates for RHEL5.1 was supplied with a backported patch for an NFS-related problem which ended up breaking caching, introducing serious performance issues that made NFS almost unusable when working with multiple files/directories. Red Hat was quick to come up with a patch after the bug had been reported - however, it only applied cleanly to their testing kernel (candidate for the next RHEL update) and refused to introduce it in the mainline kernel package, which forced us to backport the fix ourselves and keep local revisions for about half a year.

Apparently, an issue of similar origin (i.e. bug introduced by Red Hat-specific patch) is currently affecting 3ware/Areca RAID performance when used in conjunction with the RHEL5.2 mainline kernel and Red Hat is refusing to release a maintenence update to correct the error. Won't be fun for those who've got such hardware and are bound by support contracts to use only Red Hat-supplied kernels.

Re:Has happened before

[flyingfsck]

Hmm, now if you are able to do all that yourself, why exactly are you paying for a support contract?

or install ActivePerl ...

[hobbs]

for free from http://www.activestate.com/Products/activeperl/ since it is not effected by this bug. Seriously, this is far from the first time that Red Hat has botched the dynamic language environments it ships.

Why is he using RH anyway?

[flyingfsck]

If he can fix his own problems, then why exactly is he paying RedHat for 'support'? I have filed and fixed numerous bugs in my distro of choice. He should do the same.

Re:Why is he using RH anyway?

[prestomation]

He's running CentOS so we can only assume/hope he's not giving a cent to Redhat.

Re:Why is he using RH anyway?

[quarterbuck]

He is not using Redhat. He is using CentOS.
He never reported the issue to Redhat - so there really is no way of commenting about how good or how bad the RH Support is
I used to work for a large (very large) h/w manufacturer and dealt both with Microsoft and Redhat. I also have filed bugs externally. From what I know, Redhat support is excellent if you communicate either directly with them or through their website - You will get atleast a developer comment on it (with atleast enough info to patch it yourself if needed).
Microsoft support is great too - especially with comments on the developer blogs that they have. Unfortunately when they say, "Thanks for isolating the issue, the bug is in XYZ section.We'll fix it later" there is no way of patching the code. But otherwise (especially when you don't want to patch OS code) , Microsoft is good too.
On the otherhand what you said above (fixing own problems) is exactly what the author did - Used an unsupported OS, Fixed issues and blamed RedHat to boot.

...why?

[XanC]

Can you be specific about how the vendor compiling Perl is inherently worse than anyone else doing so?

This is what distributions are for: to package and/or compile software so that users don't have to. What makes Perl so special that it's suddenly "inferior" when handled that way?

Doesn't surprise me... RedHat isn't that awesome

[Anonymous Coward]

For goodness sakes people- use a different distribution! Even employees of RedHat don't even exclusively use RedHat. While RedHat was a cool idea back in the late 90s, and I'll admit the IPO was rather exciting (before it happened at least), they are the M$ of Linux distributions.

Mod parent up

[Kludge]

Let's see him rebuild VBscript or C# from source to get his speed up.

This is what open source is about.

Re:Mod parent up

[FishWithAHammer]

I rebuild Mono (C# is a language in the CLI framework, and Mono implements the CLI) nightly. What's your point?

Re:CentOS is compiled using the same tools and sou

[Anonymous Coward]

You got it wrong:

"Every time Red Hat releases a hotfix, CentOS grabs the source and compiles it, sometimes weeks or months after Red Hat made it available."

There, fixed it for you.

Seriously, if you want to judge Red Hat's performance using a lagging-behind derivative distro, you are on crack. Even the CentOS guys recommend you to use RHEL if you're so picky about faster releases and/or fixes.

A lot of people hapily not giving a cent to RedHat

[pembo13]

Hope you're just as happy if they stop "giving" cents to FOSS.

Use RHEL5 patch with RHEL4 SRPMs

[Kostya]

That's what we did. Download the SRPM for RHEL5's perl (currently unreleased), and pull the perl-5.8.8-U28775.patch. Download the SRPM for your RHEL4-based OS, edit the spec to include:

Patch28775: perl-5.8.8-U28775.patch

And then make sure to then add:

%patch28775 -p1

Somewhere in the patch section.

You then get "official" RH perl with the patch for the reblessing. It took me about 30 minutes to do all that my first time, including dusting off my RPM setup and knowledge.

Should RH have fixed it 2 years ago. Oh yes. For this, RH should be burned in effigy. You could say that everyone already knows about this, except that some of us poor slobs keep discovering it anew :-(

http://blog.rshtech.com/2008/04/rh-el-5-perl-interpreter-bug.html

Yeah, that's me discovering it all on my own. Fun!

But honestly, when has RH ever gotten the perl interpreter completely right. From RH 4 all the way to today, the general opinion is that you should build your own perl interpreter from the freshest, stable release. RH has a long history of screwing up perl patches or being slow to get the latest in. This one (#28775) is just the worst in a long history of messing up perl for RedHat :-(

And I *like* RedHat. I have used them for years now, and I continue to recommend them as the gold standard for enterprise deployments. So I'm not saying this as a hater, just an annoyed schmuck who continues to use them (I'm probably just lazy at this point).

Re:Use RHEL5 patch with RHEL4 SRPMs

[Jimmy King]

Man, I wish I would have been able to see this post 8 months ago. I fought with this at work for many weeks. We had a CMS that some contractors developed which used DBIx::Class heavily. They developed it on a SuSE box and had no issues. Then it was deployed on Red Hat (yeah, yeah. Not my choice to have the dev environment different than the live one. I've brought it up several times in the past.) It ran like total crap on the RH machines. Then the contractors left and the project was passed to me, where I had to profile the code and then do a ton of searching to find the bug. I tried several of the reported fixes that were documented at the time and nothing resolved the performance issues.

Car Analogy Version

[Tetsujin]

No, the steak is the same. It just didn't come with the house vegetables and mashed potatoes.

Or... It's like the Civilian Hummer - in place of weapon and armor fittings, they put in cup holders...

Er, wrong versions

[Kostya]

Sorry about that, this is in RHEL 5. I used the FC7 SRPM when I made my patch (back in April).

Still principle's the same--just get your current SRPM (e.g. perl-5.8.8-10.el5_0.2.src.rpm), go get the latest FC SRPM (e.g. perl-5.8.8-28.fc7.src.rpm), find the U28775 patch file, add it to the spec, rebuild your original SRPM with that patch file in its spec, etc.

Geez, what was with me and question marks in that post? I need to preview a little longer.

Wrong tool for the job?

[Ukab the Great]

If speed is so important for the author of the article, wouldn't something written in C using libxml be a better choice than using (what I'm assuming) are XML modules written exclusively in Perl?

Re:Wrong tool for the job?

[Anonymous Coward]

$_ you insenstive clod!

Re:Wrong tool for the job?

[Paradigm_Complex]

First off, you shouldn't have been modded Troll. It's a legit question. *shakes fist at mods*

It's not all-or-nothing. The code doesn't have to be super fast, just fast enough. Perl can (when running at it's normal speed) accomplish the needed task at a sufficient speed. If Perl couldn't get it done fast enough when running properly, then yes it would be the wrong tool for the job.

Re:Redhat?

[darkpixel2k]

You mean real h4x0rz like Sam Hocevar, who ran Debian last, put predictable pseudo-random number generators in their SSH packages.

Yeah--that was pretty 31337. Seriously--how many boxes could that dude have potentially hacked. That almost in the level of that unix login bug that was buried in the C compiler. ...but eventually, all the 31337 hax0rs get caught.

Thanks...I just built new RPMs to fix it...

[Anonymous Coward]

https://www.redhat.com/archives/fedora-extras-commits/2007-June/msg04242.html

Bullshit

[Anonymous Coward]

He's not running RedHat Enterprise. -- see that, that's a period. He's blaming the wrong party intentionally to score points for CentOS. Reading RedHat's forums and buglist online is not the same as having enterprise support. F-u-c-k him.

Re:CentOS is compiled using the same tools and sou

[0racle]

The GPL does not invalidate Trademarks. Red Hat owns the trademarks in question and was well within their rights to ensure it is used the way they wanted it to be used. At no point did Red Hat try to use trade mark law to keep the CentOS project "from doing what the GPL entitled them to do." In fact, RH goes out of its way to make repackaging RHEL much easier. They are not required to release source RPMs, but they do anyway. Red Hat just doesn't want you identifying your recreated distro as RHEL.

It's in the name

[Anonymous Coward]

Well, it's CentOS, and with a cent being what it is, it shouldn't be surprising performance is at 1/100th...

Re:CentOS is compiled using the same tools and sou

[Crispy Critters]

"Redhat started using trademark law to keep them from doing what the GPL entitled them to do"

This is a serious attack. Frankly, I cannot accept it at face value without your backing it up. Can you explain how Redhat was violating the GPL?

Just because all the code is GPLed doesn't mean that every bit on the distro disk is GPLed.

Re:CentOS is compiled using the same tools and sou

[digitalhermit]

I use both RHEL and CentOS. RHEL is on our production machines and CentOS is what I use for proof-of-concept and non-production machines. RedHat fixes show up a few weeks before CentOS, though I do often just rebuild the SRPM directly on CentOS. Never had a problem building the source RPM either.

CentOS doesn't do the same thing as RedHat. They take RH packages and rebuild them without modification except for vendor tags. Though they do maintain their own bug lists, it's RedHat that does the heavy lifting.

Optimizing the slowest code...

[Anonymous Coward]

Smart coders always optimize the slowest code? Damn - I guess I better get working on optimizing my apps preference panel. Here I was focusing on the most frequently used code!

Re:Redhat?

[jc42]

...but eventually, all the 31337 hax0rs get caught.

And how exactly do we know this? ;-)

Re:Redhat?

[darkpixel2k]

...but eventually, all the 31337 hax0rs get caught.

And how exactly do we know this? ;-)

I was referring to the guy who committed the not-so-random ssh key generation bug. (I was actually trying to be funny, but failed miserably. I read my comment again and am ashamed as the lack of quality control)

Totaally lame article

[stupido]

1) CentOS is *not* a RedHat product. So, Vipul, you don't lose any support... because you're entitled to none!

2) For the sake of clearing the FUD with RedHat's bugzilla's policies, here are some details on how it works:
- There's no rule that patches applied on bugzilla are to be reported upstream! Some (large) packages have over 100 patches applied. How would *you* feel if RedHat spammed you every time they applied a patch to whatever software you wrote?
- Upstream devs can register to receive any and all buzilla tickets for their products, if they chose to do so. Some do, some don't. Those that don't sometimes end up complaining that "RedHat/Fedora is keeping patches secret!". Duh.
- When a bug is reported on bugzilla, especially a crashing bug, only a _minimal_ fix is applied that fixes the crash. The patched package goes on QA for a while.

In the Perl story above an upstream patch was applied, so it made *no* sense for RedHat to report it upstream. Turns out that the patch (alone) caused another problem (slowdown). The normal thing to do was to file another bug report on it. Which did happen! And, lo and behold the bug will be fixed:

-- Comment #14 From Marcela Maslanova 2008-08-08 02:42:52 EDT --
This bug will be solved in RHEL5U3.

So Vipul, which didn't contribute a cent to RedHat by using CentOS, is biatchin' that RedHat is simply following the cautious QA policy that RHEL is known for. Perhaps he needs to use another distro that provides more bleeding edge updates, with the obvious risk entailed by less QA. Not that RedHat would give a hoot what Vipul is using (for free)...

Re:Totaally lame article

[chromatic]

There's no rule that patches applied on bugzilla are to be reported upstream! ... Upstream devs can register to receive any and all buzilla tickets for their products, if they chose to do so.

Wow, how generous. They distribute software that I write to their users under the same name as my version while potentially applying patches that I may never see unless I go looking for them. Can you see how a bad patch like this might give users the wrong impression about the software I wrote, or how having to check every potential distributor of my software might not be the most enjoyable use of my time?

And, lo and behold the bug will be fixed:

A bug which was never present in a stable release of Perl -- the only reason it was present in the Red Hat version of Perl is because Red Hat took a patch from a development version of Perl and kept applying it even to stable releases of Perl when it was no longer appropriate.

Re:Totaally lame article

[stupido]

There's no rule that patches applied on bugzilla are to be reported upstream! ... Upstream devs can register to receive any and all buzilla tickets for their products, if they chose to do so.

Wow, how generous. They distribute software that I write to their users under the same name as my version while potentially applying patches that I may never see unless I go looking for them. Can you see how a bad patch like this might give users the wrong impression about the software I wrote, or how having to check every potential distributor of my software might not be the most enjoyable use of my time?

You seem to want others to redistribute only your pristine sources. Then don't write FOSS, which, horrors, other people can patch, redistribute, and never tell you about it. Or better, sue RedHat for libel or for diminishing the value of your company's stock, that would teach them!

And, lo and behold the bug will be fixed:

A bug which was never present in a stable release of Perl -- the only reason it was present in the Red Hat version of Perl is because Red Hat took a patch from a development version of Perl and kept applying it even

Which was entirely appropriate at the time, given then minimal-patch policy. You seem to imply that they should have upgraded perl instead; that may have caused more regression failures than you think, and would have been far costlier as QA. RHEL is not even Fedora, let alone Rawhide.

I sometimes run system with patches that aren't even in Rawhide yet. Why? Because I have submitted them upstream, so I stake my reputation that they'll work.

to stable releases of Perl when it was no longer appropriate.

That's indeed a *uck-up. You seem to want to hold RedHat to a level of QA that is hardly appropriate. Like Perl never had any bugs of its own or something.

Re:Totaally lame article

[True Grit]

It seems to me you've completely missed the point: he's getting complaints & bug reports from folks who think the problem is with standard Perl, when the problem is with RHEL's patched version of Perl. Given that he didn't create this problem, but is getting flack about it nonetheless, I can understand his frustration.

Re:Totaally lame article

[chromatic]

You seem to want others to redistribute only your pristine sources.

No, I want distributors to talk to upstream. If Red Hat had asked "Hey, do you know about this bug? Is there a fix? Can we backport a patch to the old version of Perl we distribute?" they could have avoided this problem.

Is that really too much to ask?

All companies are created equal...

[aztracker1]

...to the profits they bring to your business). That's better.

For custom packages build your own yum+rpm repo

[dex.pdx]

With CentOS or RHEL (fundamentally the same, but still two different entities) the sysadmins should setup a central yum repository for in-house and custom built software. Developers should be building in-house packages into RPMS with dependencies for the correct build of Perl.

I ran into this problem a while ago building a DBIC API on the stock CentOS 5 perl build. DBIC really complains about the slow Perl bug... So, I build an rpm for Perl 5.10 and setup a yum repo. Also I deploy my Perl applications through yum+rpm as well - all my app rpms depend on my in-house perl rpm which keeps dependency issues to a minimum. If you are worried about CPAN dependencies just make sure the applications install spec does the correct stuff with your Makefile.PL and you are using a perl build environment that does CPAN require+installs (that way you don't have to build rpms for each CPAN module :).

I doubt if I was using RHEL with a support contract that they would give me any guff what I've done.

The upside of this method is that if the sysadmins are motivated enough they can build entire kickstart installs that install groups of home-grown software for specific business purposes.

A fix is coming

[chadj79]

Hey, looking at the official Bugzilla page for this bug :

https://bugzilla.redhat.com/show_bug.cgi?id=379791

(scroll down to the bottom)

It looks like RedHat is using the exact test package I and a co-worker of mine put together that illustrates this issue. And clearly RH has a Perl build in house that completely fixes this issue.

Look forward to a complete fix soon folks!!!

-CJ

In this issue of "Easy Answers to Easy Questions"

[Anonymous Coward]

Q: If a Linux vendor can't provide comprehensive maintenance and support for the open source software projects you depend on, McAllister asks, who ever will?

A: Nobody.

That has been this week's issue of "Easy Answers to Easy Questions. Be sure to check back next week, when we address the Easy Question, "Who is really in charge of Linux?"

Shocking Redhat Support...

[doomicon]

Blows!

Bug since 2006 in an "Enterprise" Operating System. So they haven't been able to get some spare body to rebuild perl?

Maybe I'm a bit jaded as I have had extensive dealings with Redhat support. I find it funny that for the longest time Redhat pushed their certifications so damn hard, but don't require their own support technicians to know anything beyond scripted responses.

It was to be expected

[Progman3K]

I really appreciate Linux when I hear a story like this:

It's like if I had a car and was able to change its constituent parts at will!

Of course that may void its warranty, but for some that is an acceptable alternative.

Let freedom ring!

Re:CentOS is compiled using the same tools and sou

[Anonymous Coward]

The question remains: what good is Redhat over CentOS (the only difference being logos and a support contract) if they ignore a major performance bug for two years?

Who fixed the bug? Did CentOS fix the bug or did CentOS compile the fixed by Red Hat code?

To be clear, Red Hat fixed the bug and CentOS again rode on their coat tails. Take out Red Hat and CentOS goes away too.

So much RH bashing

[Anonymous Coward]

Seems like slashdot bashes redhat every chance they get. The last 3 rh stories were the same way Foxnews talks about Obama.... if recompiling perl was so difficult a solution that you needed redhat support to help you how to figure it out, then kill yourself. We all have problems and bugs waiting to be fixed that make our jobs and lives a little harder.

Re:CentOS is compiled using the same tools and sou

[mikechant]

The question remains: what good is Redhat over CentOS (the only difference being logos and a support contract) if they ignore a major performance bug for two years?

Even if you concede that RH support is worthless (which I don't - your conclusion seems to be that because RH were crap on one support issue, they would be crap in all cases - other posters seem to have better experiences) the value of Redhat over CentOS is that quite a lot of non-technical business types have heard of it and probably have at least a vague impression of it being a significant force with some solidity. This sort of thing is very important for many companies when buying software or support.

Will Somebody 'Splains It To Me???

[Anonymous Coward]

What is that Red Hat does that borks perl so badly?
Why do they do it?
And why won't they stop?

Or maybe it's just me that's clueless . . .

Yet another reason...

[Yfrwlf]

...to not rely on some stupid vendor to be the one to provide you with access to what is supposed to be free software. If access to software isn't easily possible and available, it's not very "free". All software should be easily installable from the original maintainers, and distros should not have to or need to maintain any of that software.

All of Sourceforge and the entire Internet should be my Linux software repository. Support cross-distro packaging standards wherever they might arise that will help address the Linux packaging mess.

I doubt if I was using RHEL with a support contrac

[/dev/trash]

How bout you test that and report back to us?

Re:CentOS is compiled using the same tools and sou

[Anonymous Coward]

CentOS does not have access to the RHEL Build System. They use their own, which may or may not be built with the exact same combinations of tool revisions. I have not seen a comparison on any given package.

CentOS does seem to release errata. No clue if they have any access to HotFixes, which are *not* errata updates.

yet another back-handed compliment for Open Source

[darkonc]

A bug that the official vendor doesn't want to fix, so the customer goes back to the original source, compiles it and fixes the problem himeself...

When's the last time you saw this sort of 'complaint' about a Microsoft product. It may start the same, but the ending is very very different.

Re:I doubt if I was using RHEL with a support cont

[dex.pdx]

I rather save the $$$ :)

update is available

[Anonymous Coward]

Red Hat has updated perl packages that fix the problem. A RH support person told me it is due out for RHEL 5 update 3.

Upon a bit of insistence on my part he provided me the updated packages. The performance issue on our servers is gone after installing the new packages.

Hah! There are much worse bugs!

[Cinquero]

I know an extreme SSL security bug delivered with Debian Perl packages (and probably many other distributions) since years.

If you want to know it, pay me. I have told them and no one reacted. Now I'm pissed off and let 'em bleed.

Wow !! My slow web site is now bearable !!!!!

[Anonymous Coward]

http://searchitlocally.com/ads/M4Qr638AAAEAAHacengAAAAA.shtml

went from about 5 sec a page to 2 sec a page. wonderful to have source.