Bitten By the Red Hat Perl Bug

snydeq writes "Smart coders always optimize the slowest thing. But what if 'the slowest thing' is the code supplied by your vendor? That was exactly the situation Vipul Ved Prakash discovered when he tinkered with a company Linux box on which Perl code was running at least 100 times slower than expected. The code, he found, was running on CentOS Linux, using Perl packages built by Red Hat. So Prakash got rid of the Perl executable that came with CentOS, compiled a new one from stock, and the bug disappeared. 'What's more disturbing,' McAllister writes, 'is that this Red Hat Perl performance issue is a known bug,' first documented in 2006 on Red Hat's own Bugzilla database. Folks affected by the current bug have two options: sit tight, or compile the Perl interpreter from source — effectively waiving your support contract. If a Linux vendor can't provide comprehensive maintenance and support for the open source software projects you depend on, McAllister asks, who ever will?"

Comments

CentOS it NOT Red Hat

[Anonymous Coward]

If it is "supplied by CentOS" then it was compiled by "CentOS" not Red Hat. Red Hat Enterprise Linux enterprise had a hotfix for this weeks ago. So if Vipul had been using a Red Hat product, he would not have had this problem.

Re:Article is a troll

[A beautiful mind]

Still think it's a troll?

This is what a perl core hacker has to say about the issue:

It seems that there is still a problem with RedHat's packaged perl 5.8."8"**. RedHat seem to have an aggressive policy of incorporating pre-release changes in their released production code. This would not be so bad if they actually communicated back with upstream (i.e. me and the other people on the perl5-porters mailing list), or demonstrated that they had sufficient in-house knowledge that they didn't need to. But evidence suggests that neither is true, certainly for 5.8.x

Let me stress that there has never been this problem in any released Perl, 5.8.7, 5.8.8, 5.10.0, and it won't be in 5.8.9 either when it comes out. The problem was caused by changes I made in the 5.8.x tree that RedHat integrated. End users reported the first bug something like 2 years ago, and RedHat closed it as "upstream patch" rather than reporting back "you know that pre-release change you made, that we integrated - well, it seems to have some problems"

(...)

For their versions affected, RedHat merely need to put out a patch integrating changes 31996, 32018, 32019 and 32025 which FIX IT, are documented as FIXING IT, and are from NOVEMBER 2007.

Example of the UTF-8 default "bug"

[goombah99]

here's an example of this on a mac OS 10.5
[CODE]
perl -we 'for (0..1000000) { print rand(1),"\n"}' > /tmp/foo

time sort /tmp/foo
  real 0m36.169s
  user 0m35.731s
  sys 0m0.264s

echo $LANG
  en_US.UTF-8

setenv LANG C

time sort /tmp/foo > /tmp/foo2
0.966u 0.201s 0:01.27 91.3%

SO for a small file it's 40 times faster when not defaulting to UTF-8

Perl on leopard however appears to assume LANG=C bey default and is quite fast. As is grep.

Re:yum

[ccharles]

Actually, they do. yum is the default package manager in RHEL ever since version 4, I believe. In any case, you are correct that the repository that yum talks to contains the buggy Perl. It looks like this should be fixed for RHEL 5.3, but I wouldn't bet on it.

Don't link to bugzilla

[MSG]

I don't know how many projects have asked Slashdot not to link to bugzilla. It makes the system unusable for the developers trying to get work done.

Here's the text currently in the bugzilla entry (edited to meet slashdot's filter requirements):

Bug 379791 - perl bless/overload performance problem
Summary: perl bless/overload performance problem
Status: VERIFIED
Product: Red Hat Enterprise Linux 5
Component: perl (Show Red Hat Enterprise Linux 5/perl bugs)
Version: 5.2
Platform: All Linux
Priority: urgent Severity: high
Target Milestone: rc
Assigned To: Marcela Maslanova
QA Contact: desktop-bugs@redhat.com
URL:
Whiteboard: GSSApproved
Keywords: ZStream
Depends on:
Blocks:

Reported: 2007-11-13 07:14 EDT by Nigel Metheringham
Modified: 2008-08-29 10:30 EDT (History)
Fixed In Version:
Release Notes:

Description From Nigel Metheringham 2007-11-13 07:14:04 EDT

RHEL5 perl shows the same performance issues as the Fedora 7 perl did - see
Bug #196836 and Bug #253728

This has been demonstrated in the recent perl update perl-5.8.8-10.el5_0.2

Same fix needs taking across to RHEL, ideally as a update release rather than
waiting for next major release cycle.

I do not have RHEL5.1 to test against right now, but the timing of the Fedora
fixes leads me to believe these would be much too late for the 5.1 release
cycle.

-- Comment #2 From Martin Kutter 2007-11-30 05:24:01 EDT --
The issue can be observed running the benchmark script from the recent
SOAP::WSDL package.

To do so, download SOAP-WSDL-2.00_24 (and its dependencies) from CPAN, run perl
Build.PL && perl Build, cd into benchmark and run perl -I../blib/lib 01_expat.t

This is the Output from RHEL4:
perl -I../lib 01_expat.t
Name "DB::packages" used only once: possible typo at 01_expat.t line 2.
Benchmark: timing 5000 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL): 4 wallclock secs ( 3.48 usr + 0.01 sys = 3.49 CPU) @1432.66/s (n=5000)
XML::Simple (Hash): 7 wallclock secs ( 7.19 usr + 0.03 sys = 7.22 CPU) @692.52/s (n=5000)
XSD (SOAP::WSDL): 6 wallclock secs ( 6.06 usr + 0.01 sys = 6.07 CPU) @823.72/s (n=5000)

And this (with reduced n) is from RHEL5 (different machine, perl-5.8.8-10):

Benchmark: timing 500 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL): 1 wallclock secs ( 0.59 usr + 0.00 sys = 0.59 CPU) @847.46/s (n=500)
XML::Simple (Hash): 1 wallclock secs ( 1.06 usr + 0.00 sys = 1.06 CPU) @471.70/s (n=500)
XSD (SOAP::WSDL): 11 wallclock secs (11.34 usr + 0.01 sys = 11.35 CPU) @44.05/s (n=500)

Increasing the number of runs shows the O(n^2) nature of the performance problem
- increasing the number of runs by a factor of 10 increases the runtime for the
XSD bench by a factor of nearly 100:

Name "DB::packages" used only once: possible typo at 01_expat.t line 2.
Benchmark: timing 5000 iterations of Hash (SOAP:WSDL), XML::Simple (Hash), XSD
(SOAP::WSDL)...
Hash (SOAP:WSDL): 6 wallclock secs ( 6.19 usr + 0.03 sys = 6.22 CPU) @ 803.86/s (n=5000)
XML::Simple (Hash): 11 wallclock secs (11.20 usr + 0.02 sys = 11.22 CPU) @ 445.63/s (n=5000)
XSD (SOAP::WSDL): 851 wallclock secs (847.36 usr + 2.28 sys = 849.64 CPU) @ 5.88/s (n=5000)

-- Comment #3 From RHEL Product and Program Management 2007-12-03 15:47:35 EDT --
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the cur

Re:waiving your support contract?

[no1home]

Very true. And this has been an ongoing issue with Linux adoption... I have a friend who runs mega-million-dollar, mission-critical systems and they've had to move off of Linux in favor of (Sun? Don't remember right now). It isn't about functionality. It isn't about open source. It's about support. Red Hat, et. al. want to be enterprise systems, and claim to offer enterprise support. But they don't perform enterprise support. As indicated here, change something to fix a bug, and you don't get support for that piece anymore. More, they won't support a system that doesn't have the latest updates, which is a problem on mission-critical systems. We don't update needlessly, and we certainly don't update to 'today's' patch. We have to wait and be sure the patch is stable and provides an improvement without risking our mission.

Until the players selling support realize all of this, Linux will be a difficult sell for such key systems (and the PHBs all think ALL their systems are mission-critical).

Keep in mind, I say this lovingly.... I want Linux to succeed and prefer it over the popular alternative.

Re:That's what you get.

[chromatic]

Fast and correct always wins, and the real Perl hackers are working on that.

No released version of Perl ever had this bug. Red Hat pulled a patch from a development version of Perl and maintained it over released versions of Perl which did not need it. That's the source of this bug. The Perl developers fixed this bug before releasing the next stable version of Perl.