Slashdot Mirror

• The speed with which the kernel can schedule and context-switch among threads
  For some recent data on this, see http://marc.theaimsgroup.com/?l=apache-httpd-dev&m =103228014211983. The O(1) scheduler patch for 2.4 seems to help here.
• Memory usage per thread
• Concurrency limitations of the Apache code itself
  This has been improving gradually with successive 2.0 releases, as the remaining global locks are removed or optimized.
• General robustness of the thread implementation
  The current (2.4) Linux threading implementation doesn't work well with debuggers.

There was a followup message on the lkml from hp discussing how these numbers can improved even further, you can see it at http://marc.theaimsgroup.com/?l=linux-kernel&m=103 237327405644&w=2

This will certainly not win me friends in the "everything should use threads because it's easier to do linear programming than to build a session reentrant state machine" camp, but...

It's hard to overstate the importance of the ease-of-programming issue. The current Apache 2 MPMs (concurrency plug-ins) use threads rather than state machines because a pure state machine model would make life very, very difficult for third party module developers. The main reason for the current multithreaded model is that it offers a reasonable compromise between performance and ease of development.

If Apache were a closed system, it probably would be a big event loop already. But Apache is really a platform as much as it is a product, and state machines don't allow for easy module development.

Note that, because the concurrency model is a plug-in in Apache 2.0, it's possible to drop in a state-machine replacement for the current threaded design. But it wouldn't be usable with third party modules that make assumptions about being able to do blocking I/O

Among the httpd-2.0 developers, there's been some discussion of moving to a hybrid thread/async design: run most of the module callback hooks in a thread pool so that modules can use blocking code, but complete the network writes in a separate thread that runs a big event loop, so that we don't need to keep a huge number of worker threads around just to wait for network writes to complete. (See, for example, http://marc.theaimsgroup.com/?t=103083818700002&r= 1&w=2)

More interesing ASF news:

Re: El-Kabong -- HTML Parser

In the past few weeks I've been working on moving my main (home) server from Slackware on i386 to Debian on UltraSparc.

The Debian 3.0r0 install went fine (although for those trying it, be sure to select "rescue" when you boot off the CD). I recompiled my kernel using egcs64 and added RAID1 support into the kernel (with RAID0 as a module). I was able to setup my RAID partitions without difficulty, and the RAID0 arrays mount just fine. Unfortunately, when I try to mount the RAID1 arrays I get an oops, and any attempts to access the RAID device after that simply hang (that's a technical term ;)).

After a few searches on various mailing list archives, I found this post on the Linux-Sparc list. I tried this particular patch and was able to mount the RAID1, but after a few minutes copying data to the drive gave me another oops.

So, one supposition was that the oops was due to a compiler bug, but since egcs64 is so old (from 1998 I think) it's not going to get fixed. So I was looking at GCC 3.1.1 yesterday and got it installed but I was unable to compile a kernel with it (make couldn't find the compiler).

Is GCC 3.2 usable for Sparc? The GCC site had a report of a successful build of 2.4.18 using GCC 3.1.1 so I expect 3.2 would also work for UltraSparc. However, I tried to get GCC 3.1 working on a Gentoo install I did on my U30 and it died a horrible death. If GCC 3.2 will work, how do I install it as a replacement for GCC 2.95 and egcs64? When I installed the debs for 3.1.1 they didn't seem to replace either GCC or egcs. Can I pass some arguments to make-kpkg to provide the location of the compiler, as well as the -m64 option for UltraSparc?

First, thank you Gogo for the link, I have been wanting to know what was in SP3 all week. Second, to be fair microsoft has writers and secretaries and corporate accounts to cater too. Alas the SP3 update is a major update, NT service packs are almost as important as new OS's to enterprise customers. Whereas second decimal point Kernel releases are frequent. Changelogs are written by Kernel developers that can better spend time kernel hacking than making organized concise changelogs. All things considered Marcelo has done a good job compiling this log, if you want more information on any change just do a search for it or the developer who contributed it in the Kernel mailing list.

"hmm, sacrilicous" HS

Mandrake Cooker (sort of the equivalent of Debian Woody or Sid) has used gcc-3.1 as its sytem compiler since May 6.

The encoding of the letters and numbers in Morse seems wasteful, but it can't be done by five bits in one byte the way Alan Cox suggested, since not all Morse characters are 5 symbols long. There would be space in a byte to store all of this information though if the remaining bits were used to store the number of encoded bits.

Maybe Alan didn't come up with the optimal scheme this time, but you are way wrong about what's possible. In fact, a variable length string of up to 7 bits can be stored unambiguously in a single byte. See my post, which explains how the code works. Not my idea by the way, but once somebody points it out to you, it's obvious.

Darren Reed also asked in the OpenBSD misc mailing list
for prior art and points to pf probably being affected,
too (read here).

Daniel Hartmeier, swiss Author of PF, the OpenBSD packet
filter, has a good reply finding prior art and Darren even thanks him explicitly a lot, which is not what we _were_ used to read from him.

I personally do not have any objections against him,
still - though I use pf as it is in OpenBSD - the operating system of my choice, and not
even the recent OpenSSH bug could prevent me from
trusting that team.

Darren Reed also asked in the OpenBSD misc mailing list
for prior art and points to pf probably being affected,
too (read here).

Daniel Hartmeier, swiss Author of PF, the OpenBSD packet
filter, has a good reply finding prior art and Darren even thanks him explicitly a lot, which is not what we _were_ used to read from him.

I personally do not have any objections against him,
still - though I use pf as it is in OpenBSD - the operating system of my choice, and not
even the recent OpenSSH bug could prevent me from
trusting that team.

Darren Reed also asked in the OpenBSD misc mailing list
for prior art and points to pf probably being affected,
too (read here).

Daniel Hartmeier, swiss Author of PF, the OpenBSD packet
filter, has a good reply finding prior art and Darren even thanks him explicitly a lot, which is not what we _were_ used to read from him.

I personally do not have any objections against him,
still - though I use pf as it is in OpenBSD - the operating system of my choice, and not
even the recent OpenSSH bug could prevent me from
trusting that team.

I like Mandrake but your comments regarding them not caring about the license is just plain wrong. 8.2 no longer includes Netscape. The next version will not include pine because of license issues. Mandrake has made a large attempt to remove all software that isn't free software from the GPL CDs. The only way to get anything else is to belong to the MandrakeClub or buy the PowerPack.

Yeah, I know it's just a big list of links...

Apache and FW Performance Tips
Apache.org Performance Tuning
Apache Tuning Tips
Apache Tuning Directives
Tuning Your Apache Server
TUNING.txt
PHP-DEV: Database Connection Problems
PHP Everywhere: Tuning Apache and PHP for Speed
Tuning Apache Web Servers for Speed

and last, but not least, my favorite:

Web Server Tuning

I'd also recommend reading up on tuning the linux kernel.

-techwolf

Me and woolley chatted on irc tonite and i verified his patch does indeed work. You will have to manually adjust apache_1.3.26/src/ap/Makefile.tmpl to add the three object files to line 7:

ap_hook.o ap_ctx.o ap_mm.o

The patch will cause a rejection due to modifications between 1.3.24 and 1.3.26 to the file.

The patch applies to apache-1.3.24, btw. And be sure to use mod_ssl-2.8.8-1.3.24 and add --force on the mod_ssl configure line.

Woolley's patch works great.

I'm pretty sure webpages have very little validity when compared to patents, but my favorite part of this debate is the fact that no one from SCC said anything until the use of SELinux in a commercial package was brought up on the mailing list. Even better is this page, which, after being around for about 2 years, 'magically' disappeared from SCC's website after the debate began on the mailing list. Take a look at Questions 5 & 6, which pretty much spell out that they released the work under the "letter and spirit of the GPL."

This is just another example of software patent BS. Doesn't the GPL forbid/advise against patents anyway? If that's the case then why would SCC bother to say they were releasing the work under the GPL? It looks more like they just noticed that there could be money to be made on this, so now it's time to break out the patents and scream about royalties.

Way to go, SCC. I think you've confused the 'spirit of the GPL' with something else far more ugly.

I'm working with heartbeat from the Linux-HA project and it is very much alive and well, as is the linux-ha mailing list.

These problem have been around for a long time and were not fixed in the existing system. Some of them cannot be fixed using pure make rules, they are deficencies in make itself. The pre-processor code is to work around make problems that cannot be solved any other way.

The change to a pre-processor and from recursive make to a global view of the kernel build cannot be done in pieces. There is no sensible way to do part of the build using the old system and part of the build using the new system.

It appears we have a biased moderator on the loose. Please read the above article and tell me which part is 'flamebait'. Notice that the article references Larry's very own words, it is not my opinion, but Larry's.

McVoy is hardly anti-free-software. The very fact that he gives away *anything* for free symbolizes that.

Loss leader? 30 day free trial? What has that got with 'free as in freedom'? Larry lets open source projects use BitKeeper for free (for now) so that he can stamp 'Official Source Code Management System of Linux' on the box, and get free advertising. It's not because he loves free (as in freedom) software. In fact, he takes every opportunity to trash the ideals of free-as-in-freedom software and argues that it is anti-business, almost parroting the words of Microsoft. With friends like that...

The very fact that he gives away *anything* for free symbolizes that. (He doesn't have to give anything away.) He makes the simple requirement that the free users use the newest versions for bug reporting reasons.

That's what he says. The less trusting suggest it's so that he can change the licence any time he wants, in particular, he can end the free usage any time he wants. So he's enshrined his right to pull a bait and switch.

This comes up a lot on the kernel mailing list, and it boils down to this- If anyone else were to create a viable alternative to bitkeeper, They'll try it. But that's not quite as easy as it sounds.. Here's a quote from a larger posting by Larry McVoy, bitkeeper's author:

To underscore Rik's point: it took 4 years of at least 6 day/week efforts by a team that varied in size from 3-8 engineers to get BitKeeper where it is today. Pavel is welcome to try and do better, but he sure as hell isn't going to do it in his spare time. Nobody is, it's a much harder problem than it looks like. I really wish people would try it and find out. Maybe I'm wrong, in which case we can all get on to doing something more fun.

And RMS isn't discluded- If he really cares so much about a proprietary tool being USED to enhance the development of a Free project, then he's very welcome to organize development of a suitable replacement.

Maybe you should STFA.

These might be also helpful:

OpenBSD Packet Filter

Re: pf and statesfull filtering on a bridge

The OpenBSD Packet Filter HOWTO

After reading some of the other posts in this article, I noticed someone mentioned Kernel Traffic. A quick Google search turned up:

kt.zork.net

And following some links from there got me to:

marc.theaimsgroup.com/?l=linux-ultrasparc&r=1&w=2

This should keep my occupied for a while, but are there any other sites that anyone can recommend?

I've heard this repeated several places, and I don't understand where people are getting it. In fact, plex86 is a dead (or rather dormant) project, while bochs is chugging along healthily.

If you don't believe me, read the development mail archives for the month of May:

bochs

plex86

Daniel

One of these days PHP will support Apache 2.0 and then we can revisit these benchmarks. Until then I'll write this off as a Zdnet troll for Slashdot attention.

I wrote that article. :) [As I mentioned in another comment on this /. story, the article is now posted online, btw.] At the time I wrote the article, it's true that Apache 2.0 wasn't performing as well as 1.3. But I wrote it in like October. In the months since then, especially within just the last month since we added the bucket brigade freelists to avoid all the bazillion malloc/free calls and since much other performance tuning has been done by various members of the development team, 2.0 has started to really kick 1.3's butt. Have a look at this post to dev@apr.apache.org from Brian Pane on April 1, where he posted some side-by-side performance comparison numbers.

This is a lot like the EXPORT_SYMBOL_GPL travesty currently taking place on lkml.

The one time linux crashed on me was the time I was trying to configure a nVidia card

[linux-kernel] Re: Kernel hosed or Nvidia modules?

Riddle: What do you do if you're running two closed binary modules from different companies?

http://marc.theaimsgroup.com/?l=linux-kernel&m=101 770381312800&w=2

If anonymous posting hasn't been disabled, then tell me, what's my name?

You don't know. Slash keeps exactly as much information about an AC as it does about a logged in users, except for an email address. Which is usually anonymous. So even if this had been real, you'd still be just as funny.

List: mandrake-cooker
Subject: [Cooker] 8.2
From: Warly <warly@mandrakesoft.com>
Date: 2002-03-15 18:07:56
[Download message RAW]

I am in the process of building the final 8.2 isos.

These isos will be tested this week-end, and released on Monday if OK.

As a consequence if you find some free minutes this week-end and test all the uploads that have been done today, and report any regression, that would be quite a great help.

--
Warly

The original can be found here.

He hath taken the shilling of Bitkeeper

> Long live IPCop!

interesting that most of these comments about ipcop popped up after this was posted to one of their lists ...

personally I'm more interested in *BSD variants to Crunch's new box rather than listen to a rehashing of whatever many other discussions / arguments / holy wars have gone by over Linux-based distros ...

And more fuel for the fire:

a debate between a core developer of Cocoon and the author of the above article

I'll be watching this thread closely. If nothing else, it may lead to some interesting ideas for both camps.

And you'll note that most of these threads have to do with the performance of Xalan and not specifically Cocoon. If someone rolled their own implementation using Xalan, they'd run into the same issues. Cocoon supports more XSLT implementations than just Xalan. And with current work going on to integrate XSLTC, the transformation speed issues should be brought to reasonable levels soon.

http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=1 01458195902683&w=2

The developers were talking about including native code because some parts (Xalan!) were too slow. It was a random thought and a thread-starter to foster coversation. The URL that I posted is Stefano pulling back from his beliefs that native code should be used. This was because an extremely fast solution (that does not affect end user sites - internal optimization) was found in an all-Java XSLT processor. This refutes your last and second-to-last links as simply being stale info.

Some of your other links were authored by outside individuals, not core developers, and many of them are a few months old (and thus out of date with the rate of development in Cocoon). And if you follow the above links to their conclusion, you will see that (a) a configuration change solves the issue, (b) it's an issue that has been resolved, or (c) it's an issue that has been discussed and a plan of attack is well underway and close to completion.

For example, XSP has had known performance issues and this I acknowlege. The developers know what is causing the slowdown and are either working on or have already fixed many of its speed issues. One such issue is the speed at which XSPs are processed. The generation of bytecodes from XSP (much like the concept of servlet bytecode generation from JSPs) is bottlenecked by Xalan.

If you say, "yeah, that's all well and good, but it's too slow now -- I don't care about how fast it's gonna be," I can sympathize. Yes, perhaps I went overboard, but then I tend to believe that you make your interfaces and processes correct first and then focus completely on speed later. This means that over time, your apps get faster just by upgrading the framework in much the same way that native programs get faster when the compiler is updated or with optimizations to a dependant library. As long as your contract is respected (in this case, the contracts are interfaces), this is possible. Here, the contracts are XML, XSLT, XHTML, SVG, the sitemap, etc.

I see the issues with Cocoon 2 today as the same as the issues with C vs. assembly years ago. If someone coded in C when C was functionally complete but not fully optimized, they would run slower than the equivalent assembly (minus development time). On the other hand, when it came time to maintain and/or extend the code, C was easier. And as time went on, the C code "magically" became comparable in speed to assembly with little to no effort (comparatively) with the upgrade of a compiler. Then the machine for which it was originally written falls into disuse. With C, you begin porting. With assembly, you effectively start over again.

In short, if you really can't wait, then your decision is made for you: roll your own (like assembly in the above analogy) or use something else that fits your current needs. If the speed/scalability concerns are not as pressing and you can think about the long haul, Cocoon 2 makes sense (like C in the above analogy).

-----------

Note that my original post was solely with regard to comparisons of Cocoon 2 with Cocoon 1 and roll-your-owns. I explicitly excluded comparisons with Zope or any of its ilk due to my lack of knowledge about them. Right now -- today -- Cocoon 1 is slower than Cocoon 2, and rolling your own will most likely put you on the path that the Cocoon project started walking a few years ago.

My advocacy is not blind. What I said, in a nutshell, was that Cocoon 2.x is currently faster than Cocoon 1.x and probably a better choice than roll-your-own.

And last but not least, if those of you out there go through and read the mailing list archives, you'll find some exciting discussions that rarely (if ever) say that performance is bad and they've hit a brick wall. Or that performance is bad and they don't have a plan to fix it. It is a very active mailing list full of people with ideas and implementations that show Cocoon speeding up by leaps and bounds on a weekly basis.

Most of the posted links that talked about problems are merely that beginning of threads. I encourage people to read the threads in their entirety.

I've been looking forward to the perchild MPM. It can run different server processes under different UID/GIDs. This is important because mod_{perl,php,python,snake} run in-process with the Apache server. It's the only way to run them securely for different people other than a completely seperate webserver for each person (with its own IP address, configuration file, memory footprint, etc.)

But perchild doesn't really work:

• It's not portable to non-Linux platforms. (There was talk on the mailing list of marking it experimental because of this.)
• It hasn't compiled (even on Linux) out of the box in several releases. In 2.0.29, easy to fix but still doesn't work right. (Not compiling is a sure sign it hasn't been maintained.) Not quite as easy on 2.0.32. There's a patch, but it doesn't look right to me.
• It's easy to misconfigure it into running virtual hosts as root. (Bug report)

So, Apache 2.0 may be promising in the future...but when a feature I've been looking forward to for a long time is broken, I'm kind of disappointed.

Because geocrawler seems to be /.ed: Athlon/AGP issue update.

Somebody should inform Aunt Tillie about that ...

... [malloc] does not fail can also mean does not return for a VERY long time. Plus the memory system on Irix has a mechanism where various subsystems which consume memory can register callouts which the memory system can call to ask them to release memory. Linux does not have the latter except for the explicit calls in page_launder or what ever it is called this week.

Yes you are trolling! Especially since the original poster could not get the quote right.

Linus wrote: "...Which, btw, explains why I don't consider you a kernel maintainer, Rik, ..."
See for yourself!

The reason was that Rik didn't care about everybody else if his bugfixes were not applied. Would YOU like a maintainer that didn't care about the rest of the world?

"I would tell Linus to fuck off" ... who is being immature now?

b.

Full ACK, in Outlook it is at least possible to write in a good style ('though you generally go through a lot of pain to be able to do so), but in Notes it is just not possible (at least I haven't found a sane way to do it, after I've used it for 3 years).
I see the entire topic mainly from an Usenet-Perspective. Usenet is a bit better in this respect. The three groups I read frequently have almost consistent quoting style, others I read every now and then have not so great, but still good quoting habbits. It seems that the more technical a discussion gets, the better the quoting gets.
And should you ever read the LKML archive you'll find that although you'll find 'Outlook default quotings' none of the gurus use it ...

isn't one of the big "selling points" about Linux the fact that there aren't branches and forks everywhere? The appearance of one may prompt the appearance of others.

There are others. Lots of others. Always have been. Those branches are essential to the development of Linux, as Linus explains. It is important to note that all those branches are compatible: their implementation is different, but they all look the same from userland.

I think the terminology here should be used very carefully; these are patches to the official 2.4 kernel. Not kernel branches.

The terminology is used very carefully. Patches and branches are two quite different things.

Among other software, there's FreeIPdb.

In the last month or two this subject has been discussed at least once on NANOG. You might also try searching the inet-access list archives at MARC

-Nathan

sigh, its been well explained that you don't need a journaling filesystem to be safe with transfering data to the harddrive. In fact, if you're clever enough, you can even get away safely writing without having to hold the entire system up (hence, softupdates). If you actually look through the interview, you'll find Theo actually pointing you to resources that quite seriously make this point (journaling not needed).
take a look at this
it can be frustrating being right, all journaling really seems to do is attempt to fix the problems ext2fs has by laying another piece of code on top of it, instead of fixing the primary problem, that is that ext2 is broken as far as the BSD hackers are concerned.
Is waiting for fsck to finish really that much of a problem for you?

Check out this thread on the wuftpd-questions list:

http://marc.theaimsgroup.com/?l=wuftpd-questions&m =100698257011540&w=2

You know what? As Linus posted to LKML[1], it doesn't matter if there are a million pre releases, as long as it's a pre release, most people don't download it and run it on their hardware and workloads. Not to mention the fact that Linus doesn't like to maintain kernels and turns them over to other maintainers (Alan and Marcelo) for maintenance.

Hence, bugs don't turn up until after real releases are made.

Anyone who goes out and runs a shiny new kernel on a mission critical machine which was released 20 minutes ago is just asking for trouble. These kernels simply don't get the QA they need to be determined to be stable for a number of days after they're released.

If you want a QA tested kernel, go to RedHat, Suse or any of the other Linux distributions, shell out whatever they charge for bundling it up and use their kernel. When that kernel breaks, go whine to the distribution maintainers. (I've done this personally with RedHat, and found them to be very responsive to bug reports.)

Its either that, or fix it yourself, it's that simple. What, you want something for nothing? That's not how free software works.

Whining about the problem will not fix it. Going out and fixing it yourself, will.

1. See posts about Linus and maintaining stable kernels here and here.

You know what? As Linus posted to LKML[1], it doesn't matter if there are a million pre releases, as long as it's a pre release, most people don't download it and run it on their hardware and workloads. Not to mention the fact that Linus doesn't like to maintain kernels and turns them over to other maintainers (Alan and Marcelo) for maintenance.

Hence, bugs don't turn up until after real releases are made.

Anyone who goes out and runs a shiny new kernel on a mission critical machine which was released 20 minutes ago is just asking for trouble. These kernels simply don't get the QA they need to be determined to be stable for a number of days after they're released.

If you want a QA tested kernel, go to RedHat, Suse or any of the other Linux distributions, shell out whatever they charge for bundling it up and use their kernel. When that kernel breaks, go whine to the distribution maintainers. (I've done this personally with RedHat, and found them to be very responsive to bug reports.)

Its either that, or fix it yourself, it's that simple. What, you want something for nothing? That's not how free software works.

Whining about the problem will not fix it. Going out and fixing it yourself, will.

1. See posts about Linus and maintaining stable kernels here and here.

The last post in that thread is this one by Andrea Arcangeli sometime this morning and from the looks of things (if you read the entire thread) there is conflict between Alexander Viro and Andrea on which is the better solution.

Linus saying he prefers a patch on an initial viewing isn't the end of the situation for now. I'd suggesting waiting a week and revisiting the thread to find out what the final word was.

While browsing through the kernel mailing list a while back... I noticed a post that went unanswered about some specific problems Google was having with 2.4 kernels and the new/old VM.
It seems obvious from this post that Google can't use 2.4 until this gets worked out, and Google is one of Linux's big success stories...

How does that weigh in on the todo scale versus my problem with the Visor driver? What if it was Connectiva that had the problem? (Or Redhat?)

(and a quick second question: How can any human read teh 7000 posts a month to the mailing list and still get any coding done?)