Slashdot Mirror

reptyle asks: "Over Xmas dinner, a friend of the family told us she was considering launching a partnership between her employer FNMA (colloquially known as Fannie Mae; they are a semi-private company that guarantees mortgages) and Microsoft to provide training and eventually, surplus hardware to private individuals and non-profit outfits. I lamented this choice and she suggested that I e-mail her names and URLs of companies as alternative recommendations. So far I have come up with: Debian, Red Hat (distributions), VA Linux, Penguin (hardware companies), and maybe a few non-profit advocacy groups, but I think the list is still a bit too short. I can't think of any other entities that might be appropriate so I figured I'd ask Slashdot readers for help in providing other organizations that I may have missed. I don't care whether the organization uses Linux or BSD, just as long as it's not a proprietary model."

lizrd writes "The New York Times is reporting (yeah, yeah, you gotta sign to read it) that several Linux distros will be shipping stable versions of Linux for Intel's new 64-bit Itanium chip on the day that it is released to the public. Microsoft however will not be supplying a version of Windows for Itanium until sometime in the fall of next year, several months after the expected May release of the new processor."

zonker writes "This article at C|Net says the army is going to try an embedded version of Linux to use on portable vehicle diagnosis devices. Red Hat got the contract. Here is an article at Red Hat's site." Not a huge deal fiscally, but this is one of the areas where Linux is probably going to grow the most - hence companies like Lineo and others like them.

zonker writes "This article at C|Net says the army is going to try an embedded version of Linux to use on portable vehicle diagnosis devices. Red Hat got the contract. Here is an article at Red Hat's site." Not a huge deal fiscally, but this is one of the areas where Linux is probably going to grow the most - hence companies like Lineo and others like them.

zonker writes "This article at C|Net says the army is going to try an embedded version of Linux to use on portable vehicle diagnosis devices. Red Hat got the contract. Here is an article at Red Hat's site." Not a huge deal fiscally, but this is one of the areas where Linux is probably going to grow the most - hence companies like Lineo and others like them.

linugen writes "According to this article on LinuxGram, the majority of Linux distributions won't install on the Pentium 4. Apparently this is caused by the CPUID database, which contains no information on the Pentium IV. Currently only Red Hat and Turbolinux have updated versions of the CPUID databases."

Sam "Criswell" Hart asks: "I have been using Red Hat on my machines since RH4.2. The reasons were largely due to bandwidth issues and the fact that I could always get the latest RH ISOs for free. However, since 6.2 I have been disappointed with RH, and 7.0 with its not-quite compatible GCC would be very bad for my free-software projects. I have been thinking of switching to Caldera or possibly to Mandrake. However, I am spoiled by the fact that I can just pop in a new RH CD and upgrade my existing system quite painlessly. Because there are so many other RPM-based Linux distros out there, and they have update options, are there any (esp. Mandrake or Caldera) which can relatively easily update systems based on other distros? For example, anyone have any success or troubles trying to upgrade a RH6.2 system with Mandrake 7.2? Is it possible, or is there something significant blocking such a thing?" While updating a system with software intended for another distribution isn't necessarily good idea, are there utilities out there that can perform the necessary operations needed to convert an existing system from one flavor of Linux to another?

Bob Young, Chairman of the Board for Red Hat has written an open letter to the Community, in response to some of the recent criticism of Red Hat 7. If you've been following these stories at all, or the Linux scene, this is worth checking out.

Subject: Freedom & personal responsibility good, serfdom & tyrannical control bad.

From: bob@redhat.com

Dear Slashdotter,

The wild and heated debate about Red Hat 7 in recent days has been interesting to follow. It demonstrates the strength of the open source model. By comparison (I'm not sure if anyone noticed this) Computerworld had a front page story a couple of weeks ago about how there were problems with Solaris on Sun's Enterprise systems, but that these bugs were not well known because Sun was making their customers sign NDA's (non-disclosure agreements) before helping them fix the problem.

Consider the contrast between a proprietary vendor's unwillingness to debate the merits of their technology with the open debate that Red Hat Linux enjoys.

This discussion is of such value to the users of Red Hat products that we feel little need to even attempt to comment. Informed readers can read all sides of the debate, download our products, test them, and decide for themselves whether our critics or supporters are correct. Of course the readers who post things like "well I haven't tried RH7 but I've heard..." aren't very helpful, but I trust most Slashdot readers to see through that kind of stuff.

There is one recurring comment that I could not resist addressing. Namely the regular habit of our critics of comparing Red Hat to Microsoft. I just don't get it.

There are many things for which we should be justifiably criticised (I have no idea what these might be, but I'm certain they exist ;-) but trying to act like Microsoft is not one of them. Red Hat's business is built on solving the problem thatMicrosoft's business model has imposed on the software user since Bill Gates disagreed with the members of the Homebrew computing club back in 1980.

The software industry that Microsoft has been the role model for is built on the premise that customers are not to be trusted with the technology that they are building their organizations on. The legacy software industry is built on the proprietary binary-only model where not only does the user not get the source code he needs to make changes, but worse he receives the product under a license that essentially says that if you make any improvements to the technology you are using, if you solve a bug that is causing your systems to crash, or add a feature that your users or customers desperately need the vendor can have you thrown in jail. (If you don't believe me, just read any shrinkwrapped software license). This kind of business model, where the customer is completely beholden to his supplier exists in no other industry in any free market that I know of. It harks back to the old feudal systems of 12th century Europe.

Red Hat's business success is owed to one simple benefit our products and services offer that our larger binary-only OS competitors do not. Namely that our commitment to publish the code that we write and distribute under open source licenses enable us to give our customers control over the technology they are using to build their systems. We cannot promise to deliver perfection. All we can promise is to acknowledge the problems immediately and work with you to fix them publicly and in real time. With control over their systems our users can simply build more stable and reliable systems than the binary-only model allows.

This is why the fear that Red Hat is somehow going to wake up one morning and abandon our commitment to open source is so mis-placed. Open source provides us with -the- competitive advantage that enables us to compete effectively against much larger competitors. To abandon open source is simply not in our customers interest and hence not in Red Hat's financial interest.

So if you want to criticise us for shipping gcc 2.96, you have every right to do so - you'd be wrong, but it is at least a legitimate debate and I'd respect your opinion. But to compare Red Hat to Microsoft indicates an ignorance of what is driving our success.

Remember that this debate was begun by someone going to Red Hat's public site and trying to add up all the registered bugs in Red Hat 7. When was the last time Microsoft (or any other legacy software vendor for that matter) gave you access to their complete bug registration system? Which software model do you really want to see succeed? One where you have to trust your vendor (who can and frequently restrict access to information you does need) or one where you are in control of the technology you are using?

We may be making mistakes - that up to you to decide. Some of them may be important to you and while I have no doubt you will point them out to us, you have control over the technology you are using. We work hard to build products that please most of our users most of the time. But if you don't like something about Red Hat Linux you don't have to use that feature or function. We simply are not pursing a business model that bears any resemblance to Microsoft's, so just quit it.

The next slashdotter who compares anything Red Hat does to Microsoft will be punished. The punishment will be to find the nearest blackboard and write "freedom & personal responsibility good, serfdom & tyrannical control bad" seven hundred times.

Cheers, Bob.

Herz writes: "I got this email today from Red Hat. RH7 will crash out of the box in 3 weeks! The new Update Agent provided with Red Hat Linux 7.0 contains a daemon, rhnsd, which periodically polls Red Hat Network for updates. This daemon leaks file descriptors. On a default installation, all available file descriptors will be used by rhnsd in approximately three weeks, making the system unusable." The Red Hat folks have also provided a fix, though -- updated packages for those who want to use their update network, and the two-line method of disabling per machine for those who don't. After all, everyone wants uptime > 3 weeks, eh? And you don't need to wait for a "service pack," either.

John E. Cosgrove writes "I read in this article that IBM signed a deal with Red Hat to include RedHat linux on all of their mainframe servers. It's a little short, but worth the look."

Scooter[AMMO] was one of the first with this report: "According to Turbolinux CEO Paul Thomas, the future will see Linux boiled down to one generic distribution. Catch the scoop here. How can this claim be made? How can a system like Debian throw away all its guidelines to create a generic distribution? I won't give up apt, and I sure don't see RH accepting it. Why should Peter let the uber-stable power user-oriented Slackware head towards the bug infested hail-the-script-kiddies Red Hat? Standards are great, but aren't the ideals of different distributions what give Linux its flavour?"

Scooter[AMMO] was one of the first with this report: "According to Turbolinux CEO Paul Thomas, the future will see Linux boiled down to one generic distribution. Catch the scoop here. How can this claim be made? How can a system like Debian throw away all its guidelines to create a generic distribution? I won't give up apt, and I sure don't see RH accepting it. Why should Peter let the uber-stable power user-oriented Slackware head towards the bug infested hail-the-script-kiddies Red Hat? Standards are great, but aren't the ideals of different distributions what give Linux its flavour?"

A reader writes: "Linux and MacOS versions of Parsec LAN-Test have been released! Windows version will follow soon. The game will also be included on the European version of Red Hat Linux 7, with more than 50 minutes of music by Stefan Poiss. You can download the game here. Please use the mirrors."

TBHiX writes "Apparently, according to reports on bugzilla and on linuxnewbie.internet.com, Red Hat 7.0 is being described by some people as one of the buggiest distros they've seen in recent history." Red Hat's point-oh releases have been historically been pretty bad over the years, so I the only thing that surprises me is that people didn't realize it before they downloaded it. The point release has typically been fine, but the bugzilla report lists over a thousand bugs: 200 appearing this week. Take this as a warning folks: didn't 4.0, 5.0 and 6.0 teach you anything? *grin* But a DB with 2500 bugs in it doesn't necessarily mean a buggy distribution either.

musicmaster asks: "Do you know what happened to OpenCCVS (Open Credit Card Verification system). Originally it was built by Dave Cinege (original homepage). After he stopped working on it about mid-1999 it was taken over by BlackHoleSun, whose site now contains the message that work has ceased since 'many legal snags and licensing issues' didn't justify the effort. OpenCCVS is a clone of CCVS (originally located at www.hks.net, now taken over by Red Hat and placed here). Although Red Hat publishes some of the source to buyers it is not Open Source." Are there any other free software projects developing credit card verification systems?

A curious Anonymous Coward wrote in with this important issue: "RedHat Linux 7.0 does not include libc5 compatibility libraries. Trying to run an old binary (RH4.2 or earlier) generates a "no such file or directory" error. To what extent does Red Hat commit to binary backward compatibility? Is it possible to get libc5-based programs, such as MATLAB, working under RH 7 without recompiling?"

weeble writes "Red Hat 7 is now out. The updates to the Red Hat web site have been made; however the ftp site has not yet been updated." Remember to use mirrors folks. Its gonna be a bit before they all catch up so be patient.

chamont writes: "Even though Red Hat is still number 1, this article states that Caldera, SuSE, and Turbo are gaining ground fast. The article also mentions that Corel is pretty much history." Interesting to see -- what's cool is that the Linux market, at least what they measured, grew 89% overall. Turbolinux had monstrous growth overall -- you can see a lot more is happening on Pacific Rim.

James Cownie asks: "The Linux kernel development is an open process, we all know that, but, as an unknown in the community it seems impossible to attract the attention of anyone on the kernel list. I'm not trying to reimplement huge kernel subsystems or do anything major, but I found a genuine kernel bug, documented it and submitted a patch on the kernel mail list; to be met with complete and utter silence. Just as if no-one had read my mail at all. I can stand and react to abuse, or requests to fix my patch, or whatever, but what can I do in response to silence?" UPDATED

"I can't mail Alan Cox (who seems the right person for a fix to 2.2) directly because he rejects all mail from folks he doesn't know. Since the bug causes problems for gdb I mailed the gdb developer list, but also met with silence. So, how can I get someone to take notice? If no-one does, what's the point of an 'open' process? I may as well not bother."

First off, a good deal of patience is necessary when dealing with developers, they can be extremely busy when it comes to dealing with the pressures not only from their day jobs, but also from their code, their other hobbies and whatever time is left over for them to have lives to themselves. Even on internet time, certain things (like bug reports) will slip thru the cracks and seemingly fall into the ether...a few times, this might be the case, most often though, it is not and the developer just hasn't gotten to your bug/comment/suggestion yet.

A suggestion to developers: If you haven't looekd into this, it might not hurt to automate some form of reply stating your situation so that you don't alienate users by your non-response.

Thoughts?

Update: 09/05 11:50 PM by C : Alan Cox had this to say via email:

"I can't mail Alan Cox (who seems the right person for a fix to 2.2) directly because he rejects all mail from folks he doesn't know."

This is wrong. I reject mail from sites in ORBS, RBL or other major spam block lists.

A few things I'd suggest:

1. There is a REPORTING-BUGS file in 2.2/2.3 kernels
2. You should start with MAINTAINERS in the kernel for kernel bug reports
3. If its a vendor supplied kernel start with the vendor bug report system such as http://bugzilla.redhat.com/bugzilla [for Red Hat]. ( C : There's also Debian's bug reporting system at http://www.debian.org/Bugs, and the one for Mandrake-Linux at http://www.linux-mandrake.com/bugs, for other flavors of Linux, check your vendor's homepage)

#3 is important. Vendor kernels rarely match the default due to timing issues, desire to ship stuff like USB, Reiserfs, 3D acceleration and more."

As I said, the developers are listening. You just might need to take the time to find the right communication channel. For a bug report to be worth something, it has to end up in the right place.

Mike Bouma writes "Amiga and Red Hat are working together to provide the foundation for exciting games and consumer content for the desktop, set-top-box, game console, and handheld market. This announcement follows monts after the disclosure of Amiga`s relationship with the Corel Corporation. The Amiga SDK can now also be bought at Redhat.com. Also take a look at this review which includes benchmark comparisons of the Java performance of the Amiga SDK running hosted on Redhat 6.1 and native Linux Java implementations."

Mike Bouma writes "Amiga and Red Hat are working together to provide the foundation for exciting games and consumer content for the desktop, set-top-box, game console, and handheld market. This announcement follows monts after the disclosure of Amiga`s relationship with the Corel Corporation. The Amiga SDK can now also be bought at Redhat.com. Also take a look at this review which includes benchmark comparisons of the Java performance of the Amiga SDK running hosted on Redhat 6.1 and native Linux Java implementations."

Jacek Fedorynski writes "Indrema (the Linux gaming console guys) and Red Hat have announced that they have formed an alliance to "jointly manage branding and distribution" of DV Linux, a Linux distribution for gaming consoles."

Ant Mitchell was first out of the gate with the submission revealing that C2Net had signed a definitive agreement to be acquired by RedHat. An e-mail message was sent to all registered Stronghold customers. Interested in the details? Check out the FAQ.

It's quickie time! Let's get into the action with some blood and guts. cthulhubob told us about Overkill. In light of all the irritating infringement news, An anonymous coward pointed us at http://www.groucho.com. I don't know why, but Jesper Skov thinks that having turbojets on model airplanes is a fantastic idea. Hey Doc, are you reading? Gabriel Golden submitted The Boomtrain Manifesto. leviathan shared the l33t surfer. According to our anonymous source, "Wired News is reporting that Danni Ashe has become the Most Downloaded Woman on the Internet. In your face, Cindy Margolis!" Bluedove (with a little help from Wes) showed us how to pirate music and video without a computer. alienmole gave us the origin of @. Mantis! is a Sega Saturn obsessive. Check out his site. Garth, while busy turning it up to eleven, told us about the upcoming re-release of Spinal Tap. On a personal note, just wanted to tell my friend Raja to get well, and that we're all thinking about him.

KS writes: "Old time GNOME hacker and Slashdot familiar Guillaume Laurent has finally written up an explanation on why he left the GTK-- project. In summary, he disagrees with some of the fundamental features of GTK-- but sees a bright future for Red Hat's Inti. I don't know why but I always find these sorts of things eye opening." Update: 08/10 02:50 PM by H :Guillaume wrote me asking me to mention the an update to the story.

qbasicprogrammer writes: "Security Portal has a review on the security of Red Hat, SuSE, TurboLinux, and Caldera Linux distributions." Debian and Slackware are absent, but its a decent piece.

Last Tuesday you asked Ingo Molnar, Red Hat kernel hacker, about the means by which his TUX Web server recently achieved such fantastic results in SpecWeb99 . He was kind enough to respond with at-length answers addressing licensing, the reality of threads under Linux, the realism of benchmarks, and more. Thanks, Ingo!

1) TUX Architecture
by dweezil

"You appear to have take an "architectural" approach to designing TUX, so I have some architectural questions.

1.The choice of a kernel space implementation is probably going to be a controversial one. You suggest that HTTP is commonly used enough to go in the kernel just as TCP/IP did years ago. What performance or architectural advantages do you see to moving application protocols into the kernel that cannot be achieved in user space?

Ingo Molnar The biggest advantage i see is to have encapsulation, security and performance available to dynamic web applications *at the same time*.

There are various popular ways to create dynamic web content. Encapsulation, security is provided by CGI, various scripting and virtual machine models - unpriviledged/prototype/buggy CGIs are sufficiently isolated both from HTTP protocol details, from the webserving context, from the web-client and from each other.

But all CGI/scripting/virtual-machine models (including fast-CGI) lack the possibility of performing with 'maximum performance' if the webserver is in another process context. 'maximum performance' means that there should be only one context running (no context switching done), and the application writer should have the freedom to use C code, or even assembly code. (not that using assembly in web-applications would be too common.)

ISAPI, NSAPI dynamic webserver applications can have maximum performance because they run in the webserver's context, but they do not provide enough security - ie. a buggy ISAPI DLL which is loaded into IIS's address space can take down IIS and the whole site, including unrelated virtual sites. Not a very safe programming model. Additionally ISAPI/NSAPI modules often have to care about HTTP protocol details, and can cause HTTP-nonconform replies being generated. Additionally, debugging ISAPI/NSAPI modules is hard and a pain, and nobody does it on live servers. [i intentionally omitted Apache modules, because Apache modules are primarily used to extend the capabilities of Apache, they are not ment to provide a platform for dynamic applications.]

The TUX dynamic API is implemented through a kernel-based subsystem, which is accessible via a system-call which is available to 'unpriviledged' user-space code. There is no context switching (ie. we can have maximum performance) and still no user-space code can take the webserver down, because kernel-space is isolated from user-space. Obviously the TUX subsystem checks user-space parameters with extreme prejudice. It's a goal of TUX to also enforce RFC-conform replies being sent to the web-client - just like CGIs. Plus TUX modules can be in separate process contexts as well. (in this case there is going to be casual context-switching between these process contexts.)

TUX also has kernel-space modules - for the truly performance-hungry and kernel-savvy web coder. Some TUX features (such as CGI execution) are implemented as TUX kernel-space modules. Here are the various layers of modularity in TUX:

• 'accelerated requests', automated by TUX. (static GETs, etc.)
• 'kernel-space TUX modules'
• 'user-space TUX modules'
• 'fast socket-redirection to other webservers'
• 'external CGIs'

2. What is your approach to concurence? In particular, you refer to "event driven". What do you mean by that and what did you choose as the core to your event engine? Also, how do you handle threading to scale on SMP machines?

Ingo: This is how it works, simplified:

there is a 'HTTP request' user-space structure that is manipulated by the TUX kernel-space subsystem. Whenever a requested operation (eg. 'send data to client', or 'read web-object from disk') is about to block then TUX 'schedules' another request context and returns it to user-space. User-space can put event codes (or other private state) into the request structure, so that it can track the state of the request. This programming model is harder to code than other, more 'synchronous' solutions, but avoids the context-switch problem. Whenever there is no more work to be done by user-space, TUX suspends the process until there are new connections or any other progress.

3. Are there any plans to generalize the infrastructure elements of TUX so that other protocol servers can take advantage of the TUX architecture and also go fast as hell?"

Ingo: Yep i think so - i'm thinking about writing a FTP TUX component ...

2) Relative Impact
by Caballero

TUX includes a variety of kernel and apache changes. Can you give a rough measure of how each of the changes improved the http performance? I'm interested in the amount of improvement as well as why it improved performance. Do those particular changes have negative impact on the performance of other applications?

Ingo: i have no exact numbers because TUX advances in an 'evolutionary' way and closely followed/follows the 2.3/2.4 kernel line - the impact of particular changes is hard to judge exactly. Here are a couple of new features of the 2.4 kernel that are used by TUX and make a visible performance impact (without trying to list all the improvements):

• the single biggest change that enabled TUX was the inclusion of the new Linux TCP/IP architecture into 2.3 early this year. This is the 'softnet' architecture which has been written by David Miller and Alexey Kuznetsov, and is a very SMP-centric rewrite of the Linux TCP/IP stack. In Windows speak: 'the Linux 2.4 TCP/IP stack is completely deserialized'.
• another important change for TUX were the many VFS cleanups and scalability improvements done by Alexander Viro and Linus.
• a patch from Manfred Spraul went into the kernel recently, the per-CPU SLAB cache (the concept is this: when freeing buffers the kernel keeps them in variable-size per-CPU pools, and if the same type of buffer gets allocated shortly afterwards then the kernel picks from the pool that belongs to that CPU. This leads to dramatically better cache-locality.) Per-CPU SLAB pools are used by TUX, obviously.
• another feature which made a noticeable difference is the I/O scheduler rewrite by Andrea Arcangeli and Jens Axboe.
• large file support (up to 2TB files) from Matti Aarnio and Ben LaHaise is crucial as well - the logfile of the 4200 connections SPECweb99 result was bigger than 5 GB.

other TUX-related "generic kernel changes" are included in the mainstream 2.4.0-test3 kernel already: IRQ affinity, process pinning, polling-idle, better IRQ scalability, highmem support.

there are other important scalability improvements all around the place in 2.4 Linux, i think it's fair to say that almost every single line in the 'main kernel' got replaced by something else.

TUX includes no Apache changes right now - the 'redirection' feature can be used to feed connections to an (unchanged) Apache setup. But enabling a future mod_tux.c was definitely a goal, to get deeper Apache integration.

3) Caching
by JohnZed

I have a few questions about TUX's caching system. Before I go any further, I want to say that I'm incredibly impressed by the results. I've been following specWeb99 for a while and have been wondering when someone would manage to build a great dynamic cache like this one. I hope it'll get the wide acceptance it seems to deserve.

First, it seems that basically the entire test file set was loaded into memory ahead of time for use by TUX. [...]

Ingo: no, this was not the case actually. In the 1-CPU Dell system the fileset size was 4182 MB, RAM size was 2GB. In the 4-CPU Dell system the total fileset size was 13561 MB, RAM size was 8GB. So the IO and cache capabilities of TUX and the hardware were tested heavily. Eg. the best Windows 2000 + IIS result had a fileset size of 5241 MB, with 8GB RAM. (ie. fully cached, only logging IO.)

[...] How adaptable is TUX to more dynamic, limited-memory environments in terms of setting cache size limitations, selectivity (e.g. "cache all .GIFs, but not .html files"), and expiration/reloading algorithms?

Ingo: i do not want to raise expectations unnecesserily, but it's an integral part of TUX to perform async IO and 'cachemiss' operations effectively. Cached objects are timed out by a LRU mechanizm, so objects accessed less frequently will get deallocated if memory pressure rises.

Second, can a tux module programmer modify the basic tux commands, or do they always do the same thing? For instance, if I were adapting TUX to work with a web proxy cache, I'd want TUX_ACTION_GET_OBJECT to actually go out over the network and do a GET request if it couldn't find a requested object in the cache. You can imagine lots of other circumstances where this would come up as well. [...]

Ingo: TUX is a webserver, and has no proxy capabilities yet. It would be a reasonable and natural extension of the GET_OBJECT mechanizm to fetch objects from a cache hierarchy or from origin servers, yes.

Third, is it possible to execute more than one user-space TUX module at one time? [...]

Ingo: yes. TUX modules right now are compiled as shared libraries and thus can be loaded/unloaded runtime. An unlimited number of user-space modules can be used without performance impact. Fourth, when can we play with the code? Thanks a lot!

Ingo: SPEC rules which regulate the acceptance of not-yet-released products require us to release TUX sometime in August - the whole TUX codebase is going to be released under the GPL.

4)Integration into RedHat?
by ErMaC

How will the TUX Webserver integrate with RedHat's Linux distributions? Will RedHat create a special distribution with an identical setup to yours?

Ingo: there are going to be RPMs which can be installed on Red Hat 6.2, plus source-code which can be used with any distribution. Since TUX is a kernel-subsystem mainly, TUX is fundamentally distribution-neutral.

Will RedHat start releasing more specialized distributions, preferably ones more suited to a secure server environment but focused on performance like your setup was?

Ingo: well i cannot comment on specific products (being a kernel guy), but we always try to maximize the security and performance of all our products. TUX itself can be safer than user-space webservers, simply due to the fact that the kernel is a much more controlled, predictable and dedicated programming environment. Nevertheless we try to minimize the amount of code put into the kernel.

5) Performance using dynamic content
by Paladin128

How would TUX perform using CGI/Servlets/PHP/etc. compared to Apache or IIS? The ability to serve static pages fast is not that useful in the real world, as all the sites that get really big hits-per-second are those with dynamic content (Yahoo, Slashdot, Amazon.com, etc.)"

Ingo: SPECweb99 is based on real-life logged web-traffic and as a result of that it includes 30% dynamic content -- which dynamic content uses the same fileset as static replies. So the dynamic workload of SPECweb99 is neither trivial, nor isolated.

as explained earlier, TUX is designed to generate very fast dynamic content, the possibility to accelerate static content is a natural step enabled by the kernel-space HTTP protocol stack. After all the kernel does eg. TCP handshake fully in kernel-space as well, and the user gets the simpler 'connect()' functionality. Nevertheless static content is still very important, if take a look at the workload of really big sites like Yahoo, Slashdot or CNN then you'll notice that lots of content (especially content that attracts many hits) is still static. But TUX tries to provide an environment for 'next generation' web-content (ie. web applications), where dynamic output is just as natural as static.

6) Threading and Linux
by Chainsaw

Unix programmers seems to dislike using threads in their applications. After all, they can just fork(); and run along instead of using the thread functions. But, that's not important right now.

What is your opinion on the current thread implementation in the Linux kernel compared to systems designed from the ground up to support threads (like BeOS, OS/2 and Windows NT)? In which way could the kernel developers make the threads work better?"

Ingo: thats a misconception. The Linux kernel is *fundamentally* 'threaded'. Within the Linux kernel there are only threads. Full stop. Threads either share or do not share various system resources like VM (ie. page tables) or files. If a thread has 'all-private' resources then it behaves like a process. If a thread has shared resources (eg. shares files and page tables) then it's a 'thread'. Some OSs have a rigid distinction between threads and processes - Linux is more flexible, eg. you can have two threads that share all files but have private page-tables. Or you can have threads that have the same page-tables but do not share files. Within the kernel i couldnt even make a distinction between 'processes' and 'threads', because everything is a thread to the kernel.

This means that in Linux every system-call is 'thread-safe', from grounds up. You program 'threads' the same way as 'processes'. There are some popular shared-VM thread APIs, and Linux implements the pthreads API - which btw. is a user-space wrapper exposing already existing kernel-provided APIs. Just to show that the Linux kernel has only one notion for 'context of execution': under Linux the context-switch time between two 'threads' and two 'processes' is all the same: around 2 microseconds on a 500MHz PIII.

programming 'with threads' (ie.: with Linux threads that share page tables) is fundamentally more error-prone that coding isolated threads (ie. processes). This is why you see all those lazy Linux programmers using processes (ie. isolated threads) - if there is no need to share too much state, why go the error-prone path? Under Linux processes scale just as fine on SMP as threads.

the only area where 'all-shared-VM threads' are needed is where there is massive and complex interaction between threads. 98% of the programming tasks are not such. Additionally, on SMP systems threads are *fundamentally slower*, because there has to be (inevitable, hardware-mandated) synchronization between CPUs if shared VM is used.

this whole threading issue i believe comes from the fact that it's so hard and slow to program isolated threads (processes) under NT (NT processes are painfully slow to be created for example) - so all programming tasks which are performance-sensitive are forced to use all-shared-VM threads. Then this technological disadvantage of NT is spinned into a magical 'using threads is better' mantra. IMHO it's a fundamentally bad (and rude) thing to force some stupid all-shared-VM concept on all multi-context programming tasks.

for example, the submitted SPECweb99 TUX results were done in a setup where every CPU was running an isolated thread. Windows 2000 will never be able to do stuff like this without redesigning their whole OS, because processes are just so much fscked up there, and all the APIs (and programming tools) have this stupid bias towards all-shared-VM threads.

7) Kernel modules decrease portability?
by 11223

You mentioned in the second Linux Today article that you intend to integrate TUX with Apache. However, Apache has always been a cross-platform server and is heavily used on *BSD and Solaris. Do you feel that this integration will undermine the portability work of the Apache team, or will it simply provide an incentive for web servers to be running Linux? [...]

Ingo: TUX is a kernel subsystem with a small amount of user-space glue code to make it easier to use the TUX system-call. I believe that integrating kernel-based HTTP protocol stacks into Apache makes sense - i dont think this will 'undermine' anything, to the contrary, it will enable similar solutions on other OSs as well.

" If you intend to encourage people to move to Linux, can a similar idea as TUX be applied to an SQL server to make up for the speed deficit between Linux SQL servers and Microsoft SQL?

Ingo: I dont care about Microsoft SQL Server being too slow. ;-)

Database servers are not networking protocols - so the concepts of TUX and SQL servers cannot be compared. While there are a couple of simple RFCs describing the HTTP protocol, SQL is a few orders more complex, has programming extensions and all sorts of proprietary flavors.

8) Load balancing
by Ex Machina

Does/Will TUX provide any sort of load balancing for a cluster of heterogenous TUX servers?

Ingo: not the initial release, but yes, it's on the feature list :-)

9) It works with Apache...
by Ex Machina

It works with Apache but is TUX generic enough to be interfaced with another server?

Ingo: yes, i think so.

10) Version 1
By An Unnamed Correspondent

"This is a version 1 of the web server, and it has proven itself to be pretty nifty when it comes to serving both static webpages (through a kernel level httpd) and dynamic webpages. Do you see TuX getting more lean and faster as time wears on, past versions 2, 3, ... or do you see it getting bogged down in mostly unnecessary cruft and bloat?

Ingo: actually if you watch the development of Apache, Apache got faster with every major release. No, i dont think that additional features will slow TUX down, once the internal architecture is extensible enough there is no problem if additional features are *added*. It brings an obvious performance penalty if you *use* a given feature (eg. SSI). A webserver is badly designed if it gets slower doing the very same task.

Will there be a way to port an existing Apache configuration across to the TuX configuration? How about IIS, Netscape, Zeus, etc? Will TuX have the option of a GUI setup screen for those who don't like the command line? Will TuX have a simple installer?

Ingo: i dont know. The initial release will have no 'fancy' tools, this will implicitly 'encourage' the early adopters to be technically savvy. (and thus helping TUX development indirectly.) The initial TUX release is expected to be raw and uncut.

11) Re:Versions for other OSs?
by Jason Earl

Actually there is a specific feature that would probably make TUX incompatible with the BSDs. TUX is licensed under the GPL and the BSD maintainers would probably be very reluctant to port it to their OSes. Especially since it is possible that this would require them to release the derivative work under the GPL.

Which leads to the obvious question for Ingo. You mention a specific disclaimer that would allow the Apache to be linked with TUX, do the BSDs get the same privilege?

Ingo: TUX is not 'linked' to Apache. Apache can use the TUX system-call, and applications are 'isolated' from the GPL license of the kernel. Can BSD-licensed software use the TUX system-call? Of course!

Not that I particularly care, as I am not a BSD user, but the putting such a nifty program as TUX under the GPL is bound to cause weeping and gnashing of teeth in the BSD camp. Which brings up another question. How much pressure do you get from your BSD compatriots to release software like this under a more liberal BSD-friendly license?

Ingo: TUX has to be under the GPL because it's a Linux kernel subsystem, and because Red Hat releases all source code as open-source. That having said, i wouldnt write Linux kernel code if i didnt agree with the ethics of the GPL. Putting it simply, the GPL guarantees that all derivatives of source code that *we* have written stay under the GPL as well.

Isnt this a reasonable thing? Nobody is forced to use *our* source code as a base for their project, but if you freely chose to use our source-code, then isnt it very reasonable to ask for the same kind of openness that enabled you using this source code in the first place?

To put in another way, you get our source code only if you agree to be just as open to us as open we are to you. You always have the option to not use our source code. You are free to weigh the benefits of using our source code, against the 'disadvantage' of having to show us your redistributed improvements.

but again, the GPL is only about *our* *source* code. You are completely free to put your source code under whatever license you wish to. I respect BSD kernel hackers and closed-source kernel hackers just as much as Linux kernel hackers, based on the code they produce. It's their private matter to decide under what license they put their code. It's their godsent right to do whatever they wish to do with the source code they wrote.

Greyfox asks: "I'm looking for a good set of C++ bindings for ORBit. The only thing I can find seems to be OrbitCPP, which doesn't appear to be particularly robust yet. Are there any other projects out there to implement C++ bindings for ORBit?"

Several people noted that Red Hat is now partnering with Dell and offering Red Hat as one of its strategic operating systems, as well as shipping Linux web/proxy boxes to Lexus and Toyota dealerships. But thats not all... Big Blue apparently is now shipping T20 Thinkpads with Caldera and Star Office preloaded.

appletnc from linhardware.com points us to their article about Athlons and Linux. They're trying to sort out the compatibility problems rumored to exist with the boards and chipsets. He says "Despite SuSE's Athlon workaround and RedHat's (in)compatibility note, etc.) and rumors, we have not seen many reports of problems by LhD users. Are Linux users actually experiencing problems with Athlon motherboards? Given that the outstanding price/performance value of the Athlon, the question is how well do Athlon motherboards work under Linux?"

appletnc from linhardware.com points us to their article about Athlons and Linux. They're trying to sort out the compatibility problems rumored to exist with the boards and chipsets. He says "Despite SuSE's Athlon workaround and RedHat's (in)compatibility note, etc.) and rumors, we have not seen many reports of problems by LhD users. Are Linux users actually experiencing problems with Athlon motherboards? Given that the outstanding price/performance value of the Athlon, the question is how well do Athlon motherboards work under Linux?"

Matthew Miller recently went through the RH300 training course, as well as the RHCE Certification Exam. He was kind enough to write an overview and give us his opinions on both of them, as well as his opinions on the relevance and quality of the training and the exam. Certification has been discussed extensively with regards to Linux, and here's a big scoop of food for thought.

The following was written by Slashdot Reader Matthew Miller

I'm fortunate enough to work at a place that realizes the importance of keeping employees educated and up-to-date. Since my largest current project is Linux-related, and based on Red Hat's distribution in specific, we thought it'd be worthwhile to send me to Red Hat for their RH300 course. I'm pretty familiar with Linux, but I'm a long way from knowing everything, and it's always interesting to learn what the vendor thinks are the most important parts of their product. We chose RH300 because it's the highest-level systems administration class currently offered. It's also the one linked to the RHCE exam, which was an added bonus, but learning was my main goal, not getting the certification. This is my report on the experience -- hopefully, it will help you decide if this is a good choice for you, either as a sysadmin or as an employer.

The Training Center

This course is not only available directly from Red Hat, but also from various partner organizations, including Global Knowledge, which has a training center here in Boston. However, we decided that if we were going to go to the expense of sending me, I might as well go directly to Red Hat, to increase the chances of getting a good instructor, and to insure adequate access to resources. We've had experiences in the past with third-party instructors who didn't know much beyond what was written in the materials. Of course, I don't know that this would be the case with Global Knowledge's version of RH300 -- perhaps someone else can comment on any experience they've had there.

So, it was off to the Red Hat headquarters in Durham, NC. Incidentally, I stayed in the Residence Inn there -- it was on Red Hat's site as being nearby. They didn't mention that it was on the other side of a major highway, with no provision for pedestrians to get across. Moral: stay at one of the closer hotels, or else get a car. Anyway, the RH building is very nice -- much bigger than I expected. (I suppose the IPO cash is going to good use.) Of course, as students, we weren't shown much of it -- no tour, and we weren't introduced to any of the celebrity employees. (Fair enough -- with several classes coming through every week, they'd never get anything done.) The people I did meet seemed pretty cool, and in general I got the impression that it's a fun place to work.

The classroom was about as I expected -- projection screen up front, rows of decent-enough small-brand Celeron-based systems (one per student). The machines were on a private network -- reasonable for the course, but unfortunately there was no provision for Internet access, which at the least would have been nice to have when I finished labs early.

We did have access to a breakroom with free soft drinks / juice and various snack items. This is also where the lunches were served -- to my surprise, these were quite good, and there were even decent non-meat choices.

The Teacher

The instructor was very knowledgeable -- not necessarily a complete guru, but he knew his stuff, including the "why" behind the course material. He was able to present the material in a good way, and was good at answering questions. I think the decision to go to Red Hat directly was wise; unlike a third-party consultant, he had some idea of what was going on inside of Red Hat and of their potential future plans. For example, during the section on the printing subsystem, he mentioned that they're considering a replacement for LPR in future releases -- perhaps LPRng or even CUPS. It's unlikely that someone from a different company would have had access to that kind of information.

Other Students

The other students in the course had a wide range of skills and backgrounds. I think that everyone probably met the listed better than pico. However, I could tell that some people were struggling. The instructor mentioned that the pass rate for the exam is about 65%, and I wouldn't be surprised if our class came out at that level or worse. It's not that anyone was stupid -- just that some people were out of their depth. On the other end of the spectrum, there were some people who were over-qualified: a few highly experienced sysadmins, and some folks from IBM taking the class because they are soon going to teach it.

The Course

The course was generally similar to the outline found on Red Hat's site, although I think the online information is a bit out of date. (Notice that the Web page makes reference to ipfwadm instead of ipchains or netfilter.) The eight units had slightly different names, and covered slightly different information. In the most drastic example, Unit 8, listed on the Web site as "Systems Administration and Security II", has turned into "Routers, Firewalls, Clusters and Troubleshooting". Some of the information listed in the online Unit 8 was moved into Unit 7, and some of it (cops, for instance) wasn't talked about at all. Hopefully, the online info will be updated soon.

Overall, the class went into less depth than I was hoping. Some of this was due to limitations of the lab setup -- it's a bit difficult to experiment with RAID in any meaningful way when you've only got one IDE hard drive, and obviously impossible to set up a cluster on one machine (short of running VMware). Other things where just plain introductory -- the section on the kernel, for example, focused on the steps required to build and install a new kernel, rather than being an in-depth discussion of tunable parameters. The part about Apache was similar; I was hoping to hear "You've all configured Apache before; here's things you should be aware of when you need it to do such-and-such", but the most advanced we got was setting up a virtual host. Building RPMs from source was mentioned briefly, but there was no information given on important and largely undocumented topics like --buildpolicy.

That's not to say I didn't learn anything -- the section on LVS / Piranha was enlightening even without hands-on experience, and I appreciated the part about quotas, which isn't something I've worked with much. And, I learned a large number of tiny things which add up to making the experience worthwhile to me. RPM can now do globbing over ftp! Portmap uses tcp_wrappers, but doesn't do reverse name lookups, so be sure to use IP addresses instead of names. RH Linux provides a little script called "service" that lets one avoid the tedium of typing /etc/rc.d/init.d/servicename all the time. And so on....

The "300" designation is a bit misleading. This isn't really what I'd consider an upper-level course -- it's more along the lines of SysAdmin 101. Overall, I think this class is probably worthwhile to someone with a good RH Linux background who hasn't done any systems administration. In fact, I'd even recommend it to people in that situation. On the other hand, if you've been a Linux sysadmin for a while, you'll probably be bored most of the time. It might be valuable to experienced Unix sysadmins who haven't dealt with Linux much (or even Linux admins who haven't used Red Hat Linux), but the course wasn't particularly taught from that angle and there are probably better options.

The Exam

Since I signed a confidentiality agreement, I can't talk about specific details of the test, but I will address the exam in general terms. It's a day-long three part process, with each part being worth 1/3 of the total. To pass, your overall score must be at least 80%, and you can't do worse than 50% on any one part.

One of the sections is a typical multiple-choice test, but the other two are lab based. I was quite impressed with the hands-on tests -- they are certainly what makes the RHCE meaningful. I'm not aware of any other sysadmin certifications that work this way.

For one of the lab tests, students are given a several-page specification, and must install and configure Red Hat Linux and several network services. This wasn't particularly difficult, and shouldn't be for anyone with much experience. For me, the hardest part was resisting the temptation to go beyond the spec -- since I finished the given requirements with plenty of spare time, I considered installing and setting up additional services in a way that would fit in with the listed goals. But, I decided that it'd be better to leave well-enough alone -- there's no concept of extra credit.

The other hands-on test is the cool and exciting one. Students are given preconfigured setups which are broken in some way, and given a task that must be completed. The system's problem doesn't necessarily relate directly to the task, but does interfere with it. The test-taker must find out what's wrong and correct the error. (Reinstalling packages is not allowed.) Being able to list the steps taken and to repeat the fix is important, but ultimately the test is scored on a works / doesn't work basis. One the examiner verifies that the problem is fixed, he or she wipes the system and provides another broken config.

This problem-solving section directly tests skills important to being a sysadmin in the real world; if someone has trouble with these, they're probably not ready for a systems administration job. Of course, just passing this test doesn't guarantee good problem solving skills (let alone all the other needed abilities), but it does seem a genuinely valuable indicator.

I've only two complaints with this part of the test. First, I'd make it a much larger section -- at least 50% -- and I'd increase the number of problems given so that there'd be a better sample size. The various challenges are assigned at random, and some are easier than others, and each tests knowledge of different parts of the system. The way it's done isn't bad, but it wouldn't hurt to have a lot more of it. Second, I'd give each student two computers, and make more of the problems network-related. This has logistical and cost issues (especially in places other than Red Hat's own training centers), but since many of the problems faced in the real world have to do with the way systems interact, I feel it'd be worth it.

The Exam Separated From The Course

You may have noticed that I seem a lot more excited by the exam than by the course itself. I think both are valuable, but they seemed aimed at slightly different levels. The course definitely can serve as a good review for the exam, but if you need the course, you won't do well on the test. If you're tight on cash and the certification seems valuable to you or to your employer, going straight to the exam would be reasonable. (Make sure you take a look at Red Hat's test prep page.) On the other hand, if you need to be quickly brought up to speed on the basic knowledge required of a RH Linux sysadmin, it might make sense to take this course without worrying about the test. Since RH300 is equivalent to RH033 + RH133 + RH253, this could be a much more intensive and time-efficient option.

Red Hat-Specificness

It's probably obvious, but bears mentioning anyway: this is a Red Hat Linux course and certification, not a general Linux one. I found this to be true both explicitly and implicitly. The instructor was good about saying "This is the Red Hat way of doing things -- it's possibly different on other distributions." (I found the increase-the-whole-pie attitude to be common to all of the RH employees I talked to.) There were also quite a few things that were just assumed. If you take the exam without knowing a lot about Red Hat Linux in particular, you're likely to have trouble.

This doesn't make the certification meaningless for organizations running other distributions -- many of the skills and knowledge required for the test (especially the problem solving part) are generally applicable anywhere. In fact, due to the lab-based testing process, I have more respect for this exam than I might for a multiple-choice test covering more distributions. I think this issue is a one-way sort of thing: the RHCE exam requires knowledge of Red Hat Linux, but anyone who can pass it shouldn't have much trouble picking up other flavors.

Stuff

Ok, the Web page promises that they'll give Red Hat promotional items to course participants. Yeah, well, they can do better on this front. Not even a t-shirt! C'mon, everyone gives t-shirts. Vendor shirts are a staple of my wardrobe! All we got was a mousepad, some stickers, and a baseball cap. (No chance of getting a red fedora.) Oh, and of course an official copy of the CD (with the 180 days of support). Many people in the class were surprised to learn that Red Hat doesn't sell anything from their offices -- you can't buy copies of the distro or additional merchandise. They've got a lot of students coming through there, so it seems like this could be a decent (even if relatively small) revenue stream.

A Bit About Study Guides

Before I went, I flipped through RHCE Exam Cram , the sole study guide I found at the local bookstore. Someone in the class actually purchased it and brought it with them, and I got a chance to read more of it then. I wasn't really impressed. The book was especially concerned with what it called "trick questions", and indeed its sample questions were sometimes a bit confusing -- and often poorly worded. After taking the test, I can say that this seems mostly to be a problem with the book, not something encountered on the actual exam, which was mostly straightforward and fair.

There are RHCE study guides, but I wouldn't recommend spending any money on any of them. As the course instructor told us: if you're going to pass, you'll do so even if you don't have a guide. And if you're going to fail, the guide won't be much help.

Conclusion

I think the RH300 course and RHCE certification can be valuable to both employers and individuals. The course provides a nice quick overview of the basics needed to move, for example, from being a systems operator to being an admin. I wouldn't think of it as either a requirement for the test or as something that can make someone not ready suddenly have the skills required for the exam. Since the exam is hands-on and lab based, those abilities can only come from real world experience. Looking at that from the other direction: this is exactly what makes the RHCE worth anything. While it's not a total statement on someone's talent, being able to pass is a strong indicator that they have the basic skills for a systems administration job. If I were making hiring decisions, I wouldn't make the RHCE a requirement, but I would have more confidence in applicants who have it.

Matthew Miller recently went through the RH300 training course, as well as the RHCE Certification Exam. He was kind enough to write an overview and give us his opinions on both of them, as well as his opinions on the relevance and quality of the training and the exam. Certification has been discussed extensively with regards to Linux, and here's a big scoop of food for thought.

The following was written by Slashdot Reader Matthew Miller

I'm fortunate enough to work at a place that realizes the importance of keeping employees educated and up-to-date. Since my largest current project is Linux-related, and based on Red Hat's distribution in specific, we thought it'd be worthwhile to send me to Red Hat for their RH300 course. I'm pretty familiar with Linux, but I'm a long way from knowing everything, and it's always interesting to learn what the vendor thinks are the most important parts of their product. We chose RH300 because it's the highest-level systems administration class currently offered. It's also the one linked to the RHCE exam, which was an added bonus, but learning was my main goal, not getting the certification. This is my report on the experience -- hopefully, it will help you decide if this is a good choice for you, either as a sysadmin or as an employer.

The Training Center

This course is not only available directly from Red Hat, but also from various partner organizations, including Global Knowledge, which has a training center here in Boston. However, we decided that if we were going to go to the expense of sending me, I might as well go directly to Red Hat, to increase the chances of getting a good instructor, and to insure adequate access to resources. We've had experiences in the past with third-party instructors who didn't know much beyond what was written in the materials. Of course, I don't know that this would be the case with Global Knowledge's version of RH300 -- perhaps someone else can comment on any experience they've had there.

So, it was off to the Red Hat headquarters in Durham, NC. Incidentally, I stayed in the Residence Inn there -- it was on Red Hat's site as being nearby. They didn't mention that it was on the other side of a major highway, with no provision for pedestrians to get across. Moral: stay at one of the closer hotels, or else get a car. Anyway, the RH building is very nice -- much bigger than I expected. (I suppose the IPO cash is going to good use.) Of course, as students, we weren't shown much of it -- no tour, and we weren't introduced to any of the celebrity employees. (Fair enough -- with several classes coming through every week, they'd never get anything done.) The people I did meet seemed pretty cool, and in general I got the impression that it's a fun place to work.

The classroom was about as I expected -- projection screen up front, rows of decent-enough small-brand Celeron-based systems (one per student). The machines were on a private network -- reasonable for the course, but unfortunately there was no provision for Internet access, which at the least would have been nice to have when I finished labs early.

We did have access to a breakroom with free soft drinks / juice and various snack items. This is also where the lunches were served -- to my surprise, these were quite good, and there were even decent non-meat choices.

The Teacher

The instructor was very knowledgeable -- not necessarily a complete guru, but he knew his stuff, including the "why" behind the course material. He was able to present the material in a good way, and was good at answering questions. I think the decision to go to Red Hat directly was wise; unlike a third-party consultant, he had some idea of what was going on inside of Red Hat and of their potential future plans. For example, during the section on the printing subsystem, he mentioned that they're considering a replacement for LPR in future releases -- perhaps LPRng or even CUPS. It's unlikely that someone from a different company would have had access to that kind of information.

Other Students

The other students in the course had a wide range of skills and backgrounds. I think that everyone probably met the listed better than pico. However, I could tell that some people were struggling. The instructor mentioned that the pass rate for the exam is about 65%, and I wouldn't be surprised if our class came out at that level or worse. It's not that anyone was stupid -- just that some people were out of their depth. On the other end of the spectrum, there were some people who were over-qualified: a few highly experienced sysadmins, and some folks from IBM taking the class because they are soon going to teach it.

The Course

The course was generally similar to the outline found on Red Hat's site, although I think the online information is a bit out of date. (Notice that the Web page makes reference to ipfwadm instead of ipchains or netfilter.) The eight units had slightly different names, and covered slightly different information. In the most drastic example, Unit 8, listed on the Web site as "Systems Administration and Security II", has turned into "Routers, Firewalls, Clusters and Troubleshooting". Some of the information listed in the online Unit 8 was moved into Unit 7, and some of it (cops, for instance) wasn't talked about at all. Hopefully, the online info will be updated soon.

Overall, the class went into less depth than I was hoping. Some of this was due to limitations of the lab setup -- it's a bit difficult to experiment with RAID in any meaningful way when you've only got one IDE hard drive, and obviously impossible to set up a cluster on one machine (short of running VMware). Other things where just plain introductory -- the section on the kernel, for example, focused on the steps required to build and install a new kernel, rather than being an in-depth discussion of tunable parameters. The part about Apache was similar; I was hoping to hear "You've all configured Apache before; here's things you should be aware of when you need it to do such-and-such", but the most advanced we got was setting up a virtual host. Building RPMs from source was mentioned briefly, but there was no information given on important and largely undocumented topics like --buildpolicy.

That's not to say I didn't learn anything -- the section on LVS / Piranha was enlightening even without hands-on experience, and I appreciated the part about quotas, which isn't something I've worked with much. And, I learned a large number of tiny things which add up to making the experience worthwhile to me. RPM can now do globbing over ftp! Portmap uses tcp_wrappers, but doesn't do reverse name lookups, so be sure to use IP addresses instead of names. RH Linux provides a little script called "service" that lets one avoid the tedium of typing /etc/rc.d/init.d/servicename all the time. And so on....

The "300" designation is a bit misleading. This isn't really what I'd consider an upper-level course -- it's more along the lines of SysAdmin 101. Overall, I think this class is probably worthwhile to someone with a good RH Linux background who hasn't done any systems administration. In fact, I'd even recommend it to people in that situation. On the other hand, if you've been a Linux sysadmin for a while, you'll probably be bored most of the time. It might be valuable to experienced Unix sysadmins who haven't dealt with Linux much (or even Linux admins who haven't used Red Hat Linux), but the course wasn't particularly taught from that angle and there are probably better options.

The Exam

Since I signed a confidentiality agreement, I can't talk about specific details of the test, but I will address the exam in general terms. It's a day-long three part process, with each part being worth 1/3 of the total. To pass, your overall score must be at least 80%, and you can't do worse than 50% on any one part.

One of the sections is a typical multiple-choice test, but the other two are lab based. I was quite impressed with the hands-on tests -- they are certainly what makes the RHCE meaningful. I'm not aware of any other sysadmin certifications that work this way.

For one of the lab tests, students are given a several-page specification, and must install and configure Red Hat Linux and several network services. This wasn't particularly difficult, and shouldn't be for anyone with much experience. For me, the hardest part was resisting the temptation to go beyond the spec -- since I finished the given requirements with plenty of spare time, I considered installing and setting up additional services in a way that would fit in with the listed goals. But, I decided that it'd be better to leave well-enough alone -- there's no concept of extra credit.

The other hands-on test is the cool and exciting one. Students are given preconfigured setups which are broken in some way, and given a task that must be completed. The system's problem doesn't necessarily relate directly to the task, but does interfere with it. The test-taker must find out what's wrong and correct the error. (Reinstalling packages is not allowed.) Being able to list the steps taken and to repeat the fix is important, but ultimately the test is scored on a works / doesn't work basis. One the examiner verifies that the problem is fixed, he or she wipes the system and provides another broken config.

This problem-solving section directly tests skills important to being a sysadmin in the real world; if someone has trouble with these, they're probably not ready for a systems administration job. Of course, just passing this test doesn't guarantee good problem solving skills (let alone all the other needed abilities), but it does seem a genuinely valuable indicator.

I've only two complaints with this part of the test. First, I'd make it a much larger section -- at least 50% -- and I'd increase the number of problems given so that there'd be a better sample size. The various challenges are assigned at random, and some are easier than others, and each tests knowledge of different parts of the system. The way it's done isn't bad, but it wouldn't hurt to have a lot more of it. Second, I'd give each student two computers, and make more of the problems network-related. This has logistical and cost issues (especially in places other than Red Hat's own training centers), but since many of the problems faced in the real world have to do with the way systems interact, I feel it'd be worth it.

The Exam Separated From The Course

You may have noticed that I seem a lot more excited by the exam than by the course itself. I think both are valuable, but they seemed aimed at slightly different levels. The course definitely can serve as a good review for the exam, but if you need the course, you won't do well on the test. If you're tight on cash and the certification seems valuable to you or to your employer, going straight to the exam would be reasonable. (Make sure you take a look at Red Hat's test prep page.) On the other hand, if you need to be quickly brought up to speed on the basic knowledge required of a RH Linux sysadmin, it might make sense to take this course without worrying about the test. Since RH300 is equivalent to RH033 + RH133 + RH253, this could be a much more intensive and time-efficient option.

Red Hat-Specificness

It's probably obvious, but bears mentioning anyway: this is a Red Hat Linux course and certification, not a general Linux one. I found this to be true both explicitly and implicitly. The instructor was good about saying "This is the Red Hat way of doing things -- it's possibly different on other distributions." (I found the increase-the-whole-pie attitude to be common to all of the RH employees I talked to.) There were also quite a few things that were just assumed. If you take the exam without knowing a lot about Red Hat Linux in particular, you're likely to have trouble.

This doesn't make the certification meaningless for organizations running other distributions -- many of the skills and knowledge required for the test (especially the problem solving part) are generally applicable anywhere. In fact, due to the lab-based testing process, I have more respect for this exam than I might for a multiple-choice test covering more distributions. I think this issue is a one-way sort of thing: the RHCE exam requires knowledge of Red Hat Linux, but anyone who can pass it shouldn't have much trouble picking up other flavors.

Stuff

Ok, the Web page promises that they'll give Red Hat promotional items to course participants. Yeah, well, they can do better on this front. Not even a t-shirt! C'mon, everyone gives t-shirts. Vendor shirts are a staple of my wardrobe! All we got was a mousepad, some stickers, and a baseball cap. (No chance of getting a red fedora.) Oh, and of course an official copy of the CD (with the 180 days of support). Many people in the class were surprised to learn that Red Hat doesn't sell anything from their offices -- you can't buy copies of the distro or additional merchandise. They've got a lot of students coming through there, so it seems like this could be a decent (even if relatively small) revenue stream.

A Bit About Study Guides

Before I went, I flipped through RHCE Exam Cram , the sole study guide I found at the local bookstore. Someone in the class actually purchased it and brought it with them, and I got a chance to read more of it then. I wasn't really impressed. The book was especially concerned with what it called "trick questions", and indeed its sample questions were sometimes a bit confusing -- and often poorly worded. After taking the test, I can say that this seems mostly to be a problem with the book, not something encountered on the actual exam, which was mostly straightforward and fair.

There are RHCE study guides, but I wouldn't recommend spending any money on any of them. As the course instructor told us: if you're going to pass, you'll do so even if you don't have a guide. And if you're going to fail, the guide won't be much help.

Conclusion

I think the RH300 course and RHCE certification can be valuable to both employers and individuals. The course provides a nice quick overview of the basics needed to move, for example, from being a systems operator to being an admin. I wouldn't think of it as either a requirement for the test or as something that can make someone not ready suddenly have the skills required for the exam. Since the exam is hands-on and lab based, those abilities can only come from real world experience. Looking at that from the other direction: this is exactly what makes the RHCE worth anything. While it's not a total statement on someone's talent, being able to pass is a strong indicator that they have the basic skills for a systems administration job. If I were making hiring decisions, I wouldn't make the RHCE a requirement, but I would have more confidence in applicants who have it.

Matthew Miller recently went through the RH300 training course, as well as the RHCE Certification Exam. He was kind enough to write an overview and give us his opinions on both of them, as well as his opinions on the relevance and quality of the training and the exam. Certification has been discussed extensively with regards to Linux, and here's a big scoop of food for thought.

The following was written by Slashdot Reader Matthew Miller

I'm fortunate enough to work at a place that realizes the importance of keeping employees educated and up-to-date. Since my largest current project is Linux-related, and based on Red Hat's distribution in specific, we thought it'd be worthwhile to send me to Red Hat for their RH300 course. I'm pretty familiar with Linux, but I'm a long way from knowing everything, and it's always interesting to learn what the vendor thinks are the most important parts of their product. We chose RH300 because it's the highest-level systems administration class currently offered. It's also the one linked to the RHCE exam, which was an added bonus, but learning was my main goal, not getting the certification. This is my report on the experience -- hopefully, it will help you decide if this is a good choice for you, either as a sysadmin or as an employer.

The Training Center

This course is not only available directly from Red Hat, but also from various partner organizations, including Global Knowledge, which has a training center here in Boston. However, we decided that if we were going to go to the expense of sending me, I might as well go directly to Red Hat, to increase the chances of getting a good instructor, and to insure adequate access to resources. We've had experiences in the past with third-party instructors who didn't know much beyond what was written in the materials. Of course, I don't know that this would be the case with Global Knowledge's version of RH300 -- perhaps someone else can comment on any experience they've had there.

So, it was off to the Red Hat headquarters in Durham, NC. Incidentally, I stayed in the Residence Inn there -- it was on Red Hat's site as being nearby. They didn't mention that it was on the other side of a major highway, with no provision for pedestrians to get across. Moral: stay at one of the closer hotels, or else get a car. Anyway, the RH building is very nice -- much bigger than I expected. (I suppose the IPO cash is going to good use.) Of course, as students, we weren't shown much of it -- no tour, and we weren't introduced to any of the celebrity employees. (Fair enough -- with several classes coming through every week, they'd never get anything done.) The people I did meet seemed pretty cool, and in general I got the impression that it's a fun place to work.

The classroom was about as I expected -- projection screen up front, rows of decent-enough small-brand Celeron-based systems (one per student). The machines were on a private network -- reasonable for the course, but unfortunately there was no provision for Internet access, which at the least would have been nice to have when I finished labs early.

We did have access to a breakroom with free soft drinks / juice and various snack items. This is also where the lunches were served -- to my surprise, these were quite good, and there were even decent non-meat choices.

The Teacher

The instructor was very knowledgeable -- not necessarily a complete guru, but he knew his stuff, including the "why" behind the course material. He was able to present the material in a good way, and was good at answering questions. I think the decision to go to Red Hat directly was wise; unlike a third-party consultant, he had some idea of what was going on inside of Red Hat and of their potential future plans. For example, during the section on the printing subsystem, he mentioned that they're considering a replacement for LPR in future releases -- perhaps LPRng or even CUPS. It's unlikely that someone from a different company would have had access to that kind of information.

Other Students

The other students in the course had a wide range of skills and backgrounds. I think that everyone probably met the listed better than pico. However, I could tell that some people were struggling. The instructor mentioned that the pass rate for the exam is about 65%, and I wouldn't be surprised if our class came out at that level or worse. It's not that anyone was stupid -- just that some people were out of their depth. On the other end of the spectrum, there were some people who were over-qualified: a few highly experienced sysadmins, and some folks from IBM taking the class because they are soon going to teach it.

The Course

The course was generally similar to the outline found on Red Hat's site, although I think the online information is a bit out of date. (Notice that the Web page makes reference to ipfwadm instead of ipchains or netfilter.) The eight units had slightly different names, and covered slightly different information. In the most drastic example, Unit 8, listed on the Web site as "Systems Administration and Security II", has turned into "Routers, Firewalls, Clusters and Troubleshooting". Some of the information listed in the online Unit 8 was moved into Unit 7, and some of it (cops, for instance) wasn't talked about at all. Hopefully, the online info will be updated soon.

Overall, the class went into less depth than I was hoping. Some of this was due to limitations of the lab setup -- it's a bit difficult to experiment with RAID in any meaningful way when you've only got one IDE hard drive, and obviously impossible to set up a cluster on one machine (short of running VMware). Other things where just plain introductory -- the section on the kernel, for example, focused on the steps required to build and install a new kernel, rather than being an in-depth discussion of tunable parameters. The part about Apache was similar; I was hoping to hear "You've all configured Apache before; here's things you should be aware of when you need it to do such-and-such", but the most advanced we got was setting up a virtual host. Building RPMs from source was mentioned briefly, but there was no information given on important and largely undocumented topics like --buildpolicy.

That's not to say I didn't learn anything -- the section on LVS / Piranha was enlightening even without hands-on experience, and I appreciated the part about quotas, which isn't something I've worked with much. And, I learned a large number of tiny things which add up to making the experience worthwhile to me. RPM can now do globbing over ftp! Portmap uses tcp_wrappers, but doesn't do reverse name lookups, so be sure to use IP addresses instead of names. RH Linux provides a little script called "service" that lets one avoid the tedium of typing /etc/rc.d/init.d/servicename all the time. And so on....

The "300" designation is a bit misleading. This isn't really what I'd consider an upper-level course -- it's more along the lines of SysAdmin 101. Overall, I think this class is probably worthwhile to someone with a good RH Linux background who hasn't done any systems administration. In fact, I'd even recommend it to people in that situation. On the other hand, if you've been a Linux sysadmin for a while, you'll probably be bored most of the time. It might be valuable to experienced Unix sysadmins who haven't dealt with Linux much (or even Linux admins who haven't used Red Hat Linux), but the course wasn't particularly taught from that angle and there are probably better options.

The Exam

Since I signed a confidentiality agreement, I can't talk about specific details of the test, but I will address the exam in general terms. It's a day-long three part process, with each part being worth 1/3 of the total. To pass, your overall score must be at least 80%, and you can't do worse than 50% on any one part.

One of the sections is a typical multiple-choice test, but the other two are lab based. I was quite impressed with the hands-on tests -- they are certainly what makes the RHCE meaningful. I'm not aware of any other sysadmin certifications that work this way.

For one of the lab tests, students are given a several-page specification, and must install and configure Red Hat Linux and several network services. This wasn't particularly difficult, and shouldn't be for anyone with much experience. For me, the hardest part was resisting the temptation to go beyond the spec -- since I finished the given requirements with plenty of spare time, I considered installing and setting up additional services in a way that would fit in with the listed goals. But, I decided that it'd be better to leave well-enough alone -- there's no concept of extra credit.

The other hands-on test is the cool and exciting one. Students are given preconfigured setups which are broken in some way, and given a task that must be completed. The system's problem doesn't necessarily relate directly to the task, but does interfere with it. The test-taker must find out what's wrong and correct the error. (Reinstalling packages is not allowed.) Being able to list the steps taken and to repeat the fix is important, but ultimately the test is scored on a works / doesn't work basis. One the examiner verifies that the problem is fixed, he or she wipes the system and provides another broken config.

This problem-solving section directly tests skills important to being a sysadmin in the real world; if someone has trouble with these, they're probably not ready for a systems administration job. Of course, just passing this test doesn't guarantee good problem solving skills (let alone all the other needed abilities), but it does seem a genuinely valuable indicator.

I've only two complaints with this part of the test. First, I'd make it a much larger section -- at least 50% -- and I'd increase the number of problems given so that there'd be a better sample size. The various challenges are assigned at random, and some are easier than others, and each tests knowledge of different parts of the system. The way it's done isn't bad, but it wouldn't hurt to have a lot more of it. Second, I'd give each student two computers, and make more of the problems network-related. This has logistical and cost issues (especially in places other than Red Hat's own training centers), but since many of the problems faced in the real world have to do with the way systems interact, I feel it'd be worth it.

The Exam Separated From The Course

You may have noticed that I seem a lot more excited by the exam than by the course itself. I think both are valuable, but they seemed aimed at slightly different levels. The course definitely can serve as a good review for the exam, but if you need the course, you won't do well on the test. If you're tight on cash and the certification seems valuable to you or to your employer, going straight to the exam would be reasonable. (Make sure you take a look at Red Hat's test prep page.) On the other hand, if you need to be quickly brought up to speed on the basic knowledge required of a RH Linux sysadmin, it might make sense to take this course without worrying about the test. Since RH300 is equivalent to RH033 + RH133 + RH253, this could be a much more intensive and time-efficient option.

Red Hat-Specificness

It's probably obvious, but bears mentioning anyway: this is a Red Hat Linux course and certification, not a general Linux one. I found this to be true both explicitly and implicitly. The instructor was good about saying "This is the Red Hat way of doing things -- it's possibly different on other distributions." (I found the increase-the-whole-pie attitude to be common to all of the RH employees I talked to.) There were also quite a few things that were just assumed. If you take the exam without knowing a lot about Red Hat Linux in particular, you're likely to have trouble.

This doesn't make the certification meaningless for organizations running other distributions -- many of the skills and knowledge required for the test (especially the problem solving part) are generally applicable anywhere. In fact, due to the lab-based testing process, I have more respect for this exam than I might for a multiple-choice test covering more distributions. I think this issue is a one-way sort of thing: the RHCE exam requires knowledge of Red Hat Linux, but anyone who can pass it shouldn't have much trouble picking up other flavors.

Stuff

Ok, the Web page promises that they'll give Red Hat promotional items to course participants. Yeah, well, they can do better on this front. Not even a t-shirt! C'mon, everyone gives t-shirts. Vendor shirts are a staple of my wardrobe! All we got was a mousepad, some stickers, and a baseball cap. (No chance of getting a red fedora.) Oh, and of course an official copy of the CD (with the 180 days of support). Many people in the class were surprised to learn that Red Hat doesn't sell anything from their offices -- you can't buy copies of the distro or additional merchandise. They've got a lot of students coming through there, so it seems like this could be a decent (even if relatively small) revenue stream.

A Bit About Study Guides

Before I went, I flipped through RHCE Exam Cram , the sole study guide I found at the local bookstore. Someone in the class actually purchased it and brought it with them, and I got a chance to read more of it then. I wasn't really impressed. The book was especially concerned with what it called "trick questions", and indeed its sample questions were sometimes a bit confusing -- and often poorly worded. After taking the test, I can say that this seems mostly to be a problem with the book, not something encountered on the actual exam, which was mostly straightforward and fair.

There are RHCE study guides, but I wouldn't recommend spending any money on any of them. As the course instructor told us: if you're going to pass, you'll do so even if you don't have a guide. And if you're going to fail, the guide won't be much help.

Conclusion

I think the RH300 course and RHCE certification can be valuable to both employers and individuals. The course provides a nice quick overview of the basics needed to move, for example, from being a systems operator to being an admin. I wouldn't think of it as either a requirement for the test or as something that can make someone not ready suddenly have the skills required for the exam. Since the exam is hands-on and lab based, those abilities can only come from real world experience. Looking at that from the other direction: this is exactly what makes the RHCE worth anything. While it's not a total statement on someone's talent, being able to pass is a strong indicator that they have the basic skills for a systems administration job. If I were making hiring decisions, I wouldn't make the RHCE a requirement, but I would have more confidence in applicants who have it.

Matthew Miller recently went through the RH300 training course, as well as the RHCE Certification Exam. He was kind enough to write an overview and give us his opinions on both of them, as well as his opinions on the relevance and quality of the training and the exam. Certification has been discussed extensively with regards to Linux, and here's a big scoop of food for thought.

The following was written by Slashdot Reader Matthew Miller

I'm fortunate enough to work at a place that realizes the importance of keeping employees educated and up-to-date. Since my largest current project is Linux-related, and based on Red Hat's distribution in specific, we thought it'd be worthwhile to send me to Red Hat for their RH300 course. I'm pretty familiar with Linux, but I'm a long way from knowing everything, and it's always interesting to learn what the vendor thinks are the most important parts of their product. We chose RH300 because it's the highest-level systems administration class currently offered. It's also the one linked to the RHCE exam, which was an added bonus, but learning was my main goal, not getting the certification. This is my report on the experience -- hopefully, it will help you decide if this is a good choice for you, either as a sysadmin or as an employer.

The Training Center

This course is not only available directly from Red Hat, but also from various partner organizations, including Global Knowledge, which has a training center here in Boston. However, we decided that if we were going to go to the expense of sending me, I might as well go directly to Red Hat, to increase the chances of getting a good instructor, and to insure adequate access to resources. We've had experiences in the past with third-party instructors who didn't know much beyond what was written in the materials. Of course, I don't know that this would be the case with Global Knowledge's version of RH300 -- perhaps someone else can comment on any experience they've had there.

So, it was off to the Red Hat headquarters in Durham, NC. Incidentally, I stayed in the Residence Inn there -- it was on Red Hat's site as being nearby. They didn't mention that it was on the other side of a major highway, with no provision for pedestrians to get across. Moral: stay at one of the closer hotels, or else get a car. Anyway, the RH building is very nice -- much bigger than I expected. (I suppose the IPO cash is going to good use.) Of course, as students, we weren't shown much of it -- no tour, and we weren't introduced to any of the celebrity employees. (Fair enough -- with several classes coming through every week, they'd never get anything done.) The people I did meet seemed pretty cool, and in general I got the impression that it's a fun place to work.

The classroom was about as I expected -- projection screen up front, rows of decent-enough small-brand Celeron-based systems (one per student). The machines were on a private network -- reasonable for the course, but unfortunately there was no provision for Internet access, which at the least would have been nice to have when I finished labs early.

We did have access to a breakroom with free soft drinks / juice and various snack items. This is also where the lunches were served -- to my surprise, these were quite good, and there were even decent non-meat choices.

The Teacher

The instructor was very knowledgeable -- not necessarily a complete guru, but he knew his stuff, including the "why" behind the course material. He was able to present the material in a good way, and was good at answering questions. I think the decision to go to Red Hat directly was wise; unlike a third-party consultant, he had some idea of what was going on inside of Red Hat and of their potential future plans. For example, during the section on the printing subsystem, he mentioned that they're considering a replacement for LPR in future releases -- perhaps LPRng or even CUPS. It's unlikely that someone from a different company would have had access to that kind of information.

Other Students

The other students in the course had a wide range of skills and backgrounds. I think that everyone probably met the listed better than pico. However, I could tell that some people were struggling. The instructor mentioned that the pass rate for the exam is about 65%, and I wouldn't be surprised if our class came out at that level or worse. It's not that anyone was stupid -- just that some people were out of their depth. On the other end of the spectrum, there were some people who were over-qualified: a few highly experienced sysadmins, and some folks from IBM taking the class because they are soon going to teach it.

The Course

The course was generally similar to the outline found on Red Hat's site, although I think the online information is a bit out of date. (Notice that the Web page makes reference to ipfwadm instead of ipchains or netfilter.) The eight units had slightly different names, and covered slightly different information. In the most drastic example, Unit 8, listed on the Web site as "Systems Administration and Security II", has turned into "Routers, Firewalls, Clusters and Troubleshooting". Some of the information listed in the online Unit 8 was moved into Unit 7, and some of it (cops, for instance) wasn't talked about at all. Hopefully, the online info will be updated soon.

Overall, the class went into less depth than I was hoping. Some of this was due to limitations of the lab setup -- it's a bit difficult to experiment with RAID in any meaningful way when you've only got one IDE hard drive, and obviously impossible to set up a cluster on one machine (short of running VMware). Other things where just plain introductory -- the section on the kernel, for example, focused on the steps required to build and install a new kernel, rather than being an in-depth discussion of tunable parameters. The part about Apache was similar; I was hoping to hear "You've all configured Apache before; here's things you should be aware of when you need it to do such-and-such", but the most advanced we got was setting up a virtual host. Building RPMs from source was mentioned briefly, but there was no information given on important and largely undocumented topics like --buildpolicy.

That's not to say I didn't learn anything -- the section on LVS / Piranha was enlightening even without hands-on experience, and I appreciated the part about quotas, which isn't something I've worked with much. And, I learned a large number of tiny things which add up to making the experience worthwhile to me. RPM can now do globbing over ftp! Portmap uses tcp_wrappers, but doesn't do reverse name lookups, so be sure to use IP addresses instead of names. RH Linux provides a little script called "service" that lets one avoid the tedium of typing /etc/rc.d/init.d/servicename all the time. And so on....

The "300" designation is a bit misleading. This isn't really what I'd consider an upper-level course -- it's more along the lines of SysAdmin 101. Overall, I think this class is probably worthwhile to someone with a good RH Linux background who hasn't done any systems administration. In fact, I'd even recommend it to people in that situation. On the other hand, if you've been a Linux sysadmin for a while, you'll probably be bored most of the time. It might be valuable to experienced Unix sysadmins who haven't dealt with Linux much (or even Linux admins who haven't used Red Hat Linux), but the course wasn't particularly taught from that angle and there are probably better options.

The Exam

Since I signed a confidentiality agreement, I can't talk about specific details of the test, but I will address the exam in general terms. It's a day-long three part process, with each part being worth 1/3 of the total. To pass, your overall score must be at least 80%, and you can't do worse than 50% on any one part.

One of the sections is a typical multiple-choice test, but the other two are lab based. I was quite impressed with the hands-on tests -- they are certainly what makes the RHCE meaningful. I'm not aware of any other sysadmin certifications that work this way.

For one of the lab tests, students are given a several-page specification, and must install and configure Red Hat Linux and several network services. This wasn't particularly difficult, and shouldn't be for anyone with much experience. For me, the hardest part was resisting the temptation to go beyond the spec -- since I finished the given requirements with plenty of spare time, I considered installing and setting up additional services in a way that would fit in with the listed goals. But, I decided that it'd be better to leave well-enough alone -- there's no concept of extra credit.

The other hands-on test is the cool and exciting one. Students are given preconfigured setups which are broken in some way, and given a task that must be completed. The system's problem doesn't necessarily relate directly to the task, but does interfere with it. The test-taker must find out what's wrong and correct the error. (Reinstalling packages is not allowed.) Being able to list the steps taken and to repeat the fix is important, but ultimately the test is scored on a works / doesn't work basis. One the examiner verifies that the problem is fixed, he or she wipes the system and provides another broken config.

This problem-solving section directly tests skills important to being a sysadmin in the real world; if someone has trouble with these, they're probably not ready for a systems administration job. Of course, just passing this test doesn't guarantee good problem solving skills (let alone all the other needed abilities), but it does seem a genuinely valuable indicator.

I've only two complaints with this part of the test. First, I'd make it a much larger section -- at least 50% -- and I'd increase the number of problems given so that there'd be a better sample size. The various challenges are assigned at random, and some are easier than others, and each tests knowledge of different parts of the system. The way it's done isn't bad, but it wouldn't hurt to have a lot more of it. Second, I'd give each student two computers, and make more of the problems network-related. This has logistical and cost issues (especially in places other than Red Hat's own training centers), but since many of the problems faced in the real world have to do with the way systems interact, I feel it'd be worth it.

The Exam Separated From The Course

You may have noticed that I seem a lot more excited by the exam than by the course itself. I think both are valuable, but they seemed aimed at slightly different levels. The course definitely can serve as a good review for the exam, but if you need the course, you won't do well on the test. If you're tight on cash and the certification seems valuable to you or to your employer, going straight to the exam would be reasonable. (Make sure you take a look at Red Hat's test prep page.) On the other hand, if you need to be quickly brought up to speed on the basic knowledge required of a RH Linux sysadmin, it might make sense to take this course without worrying about the test. Since RH300 is equivalent to RH033 + RH133 + RH253, this could be a much more intensive and time-efficient option.

Red Hat-Specificness

It's probably obvious, but bears mentioning anyway: this is a Red Hat Linux course and certification, not a general Linux one. I found this to be true both explicitly and implicitly. The instructor was good about saying "This is the Red Hat way of doing things -- it's possibly different on other distributions." (I found the increase-the-whole-pie attitude to be common to all of the RH employees I talked to.) There were also quite a few things that were just assumed. If you take the exam without knowing a lot about Red Hat Linux in particular, you're likely to have trouble.

This doesn't make the certification meaningless for organizations running other distributions -- many of the skills and knowledge required for the test (especially the problem solving part) are generally applicable anywhere. In fact, due to the lab-based testing process, I have more respect for this exam than I might for a multiple-choice test covering more distributions. I think this issue is a one-way sort of thing: the RHCE exam requires knowledge of Red Hat Linux, but anyone who can pass it shouldn't have much trouble picking up other flavors.

Stuff

Ok, the Web page promises that they'll give Red Hat promotional items to course participants. Yeah, well, they can do better on this front. Not even a t-shirt! C'mon, everyone gives t-shirts. Vendor shirts are a staple of my wardrobe! All we got was a mousepad, some stickers, and a baseball cap. (No chance of getting a red fedora.) Oh, and of course an official copy of the CD (with the 180 days of support). Many people in the class were surprised to learn that Red Hat doesn't sell anything from their offices -- you can't buy copies of the distro or additional merchandise. They've got a lot of students coming through there, so it seems like this could be a decent (even if relatively small) revenue stream.

A Bit About Study Guides

Before I went, I flipped through RHCE Exam Cram , the sole study guide I found at the local bookstore. Someone in the class actually purchased it and brought it with them, and I got a chance to read more of it then. I wasn't really impressed. The book was especially concerned with what it called "trick questions", and indeed its sample questions were sometimes a bit confusing -- and often poorly worded. After taking the test, I can say that this seems mostly to be a problem with the book, not something encountered on the actual exam, which was mostly straightforward and fair.

There are RHCE study guides, but I wouldn't recommend spending any money on any of them. As the course instructor told us: if you're going to pass, you'll do so even if you don't have a guide. And if you're going to fail, the guide won't be much help.

Conclusion

I think the RH300 course and RHCE certification can be valuable to both employers and individuals. The course provides a nice quick overview of the basics needed to move, for example, from being a systems operator to being an admin. I wouldn't think of it as either a requirement for the test or as something that can make someone not ready suddenly have the skills required for the exam. Since the exam is hands-on and lab based, those abilities can only come from real world experience. Looking at that from the other direction: this is exactly what makes the RHCE worth anything. While it's not a total statement on someone's talent, being able to pass is a strong indicator that they have the basic skills for a systems administration job. If I were making hiring decisions, I wouldn't make the RHCE a requirement, but I would have more confidence in applicants who have it.

Matthew Miller recently went through the RH300 training course, as well as the RHCE Certification Exam. He was kind enough to write an overview and give us his opinions on both of them, as well as his opinions on the relevance and quality of the training and the exam. Certification has been discussed extensively with regards to Linux, and here's a big scoop of food for thought.

The following was written by Slashdot Reader Matthew Miller

I'm fortunate enough to work at a place that realizes the importance of keeping employees educated and up-to-date. Since my largest current project is Linux-related, and based on Red Hat's distribution in specific, we thought it'd be worthwhile to send me to Red Hat for their RH300 course. I'm pretty familiar with Linux, but I'm a long way from knowing everything, and it's always interesting to learn what the vendor thinks are the most important parts of their product. We chose RH300 because it's the highest-level systems administration class currently offered. It's also the one linked to the RHCE exam, which was an added bonus, but learning was my main goal, not getting the certification. This is my report on the experience -- hopefully, it will help you decide if this is a good choice for you, either as a sysadmin or as an employer.

The Training Center

This course is not only available directly from Red Hat, but also from various partner organizations, including Global Knowledge, which has a training center here in Boston. However, we decided that if we were going to go to the expense of sending me, I might as well go directly to Red Hat, to increase the chances of getting a good instructor, and to insure adequate access to resources. We've had experiences in the past with third-party instructors who didn't know much beyond what was written in the materials. Of course, I don't know that this would be the case with Global Knowledge's version of RH300 -- perhaps someone else can comment on any experience they've had there.

So, it was off to the Red Hat headquarters in Durham, NC. Incidentally, I stayed in the Residence Inn there -- it was on Red Hat's site as being nearby. They didn't mention that it was on the other side of a major highway, with no provision for pedestrians to get across. Moral: stay at one of the closer hotels, or else get a car. Anyway, the RH building is very nice -- much bigger than I expected. (I suppose the IPO cash is going to good use.) Of course, as students, we weren't shown much of it -- no tour, and we weren't introduced to any of the celebrity employees. (Fair enough -- with several classes coming through every week, they'd never get anything done.) The people I did meet seemed pretty cool, and in general I got the impression that it's a fun place to work.

The classroom was about as I expected -- projection screen up front, rows of decent-enough small-brand Celeron-based systems (one per student). The machines were on a private network -- reasonable for the course, but unfortunately there was no provision for Internet access, which at the least would have been nice to have when I finished labs early.

We did have access to a breakroom with free soft drinks / juice and various snack items. This is also where the lunches were served -- to my surprise, these were quite good, and there were even decent non-meat choices.

The Teacher

The instructor was very knowledgeable -- not necessarily a complete guru, but he knew his stuff, including the "why" behind the course material. He was able to present the material in a good way, and was good at answering questions. I think the decision to go to Red Hat directly was wise; unlike a third-party consultant, he had some idea of what was going on inside of Red Hat and of their potential future plans. For example, during the section on the printing subsystem, he mentioned that they're considering a replacement for LPR in future releases -- perhaps LPRng or even CUPS. It's unlikely that someone from a different company would have had access to that kind of information.

Other Students

The other students in the course had a wide range of skills and backgrounds. I think that everyone probably met the listed better than pico. However, I could tell that some people were struggling. The instructor mentioned that the pass rate for the exam is about 65%, and I wouldn't be surprised if our class came out at that level or worse. It's not that anyone was stupid -- just that some people were out of their depth. On the other end of the spectrum, there were some people who were over-qualified: a few highly experienced sysadmins, and some folks from IBM taking the class because they are soon going to teach it.

The Course

The course was generally similar to the outline found on Red Hat's site, although I think the online information is a bit out of date. (Notice that the Web page makes reference to ipfwadm instead of ipchains or netfilter.) The eight units had slightly different names, and covered slightly different information. In the most drastic example, Unit 8, listed on the Web site as "Systems Administration and Security II", has turned into "Routers, Firewalls, Clusters and Troubleshooting". Some of the information listed in the online Unit 8 was moved into Unit 7, and some of it (cops, for instance) wasn't talked about at all. Hopefully, the online info will be updated soon.

Overall, the class went into less depth than I was hoping. Some of this was due to limitations of the lab setup -- it's a bit difficult to experiment with RAID in any meaningful way when you've only got one IDE hard drive, and obviously impossible to set up a cluster on one machine (short of running VMware). Other things where just plain introductory -- the section on the kernel, for example, focused on the steps required to build and install a new kernel, rather than being an in-depth discussion of tunable parameters. The part about Apache was similar; I was hoping to hear "You've all configured Apache before; here's things you should be aware of when you need it to do such-and-such", but the most advanced we got was setting up a virtual host. Building RPMs from source was mentioned briefly, but there was no information given on important and largely undocumented topics like --buildpolicy.

That's not to say I didn't learn anything -- the section on LVS / Piranha was enlightening even without hands-on experience, and I appreciated the part about quotas, which isn't something I've worked with much. And, I learned a large number of tiny things which add up to making the experience worthwhile to me. RPM can now do globbing over ftp! Portmap uses tcp_wrappers, but doesn't do reverse name lookups, so be sure to use IP addresses instead of names. RH Linux provides a little script called "service" that lets one avoid the tedium of typing /etc/rc.d/init.d/servicename all the time. And so on....

The "300" designation is a bit misleading. This isn't really what I'd consider an upper-level course -- it's more along the lines of SysAdmin 101. Overall, I think this class is probably worthwhile to someone with a good RH Linux background who hasn't done any systems administration. In fact, I'd even recommend it to people in that situation. On the other hand, if you've been a Linux sysadmin for a while, you'll probably be bored most of the time. It might be valuable to experienced Unix sysadmins who haven't dealt with Linux much (or even Linux admins who haven't used Red Hat Linux), but the course wasn't particularly taught from that angle and there are probably better options.

The Exam

Since I signed a confidentiality agreement, I can't talk about specific details of the test, but I will address the exam in general terms. It's a day-long three part process, with each part being worth 1/3 of the total. To pass, your overall score must be at least 80%, and you can't do worse than 50% on any one part.

One of the sections is a typical multiple-choice test, but the other two are lab based. I was quite impressed with the hands-on tests -- they are certainly what makes the RHCE meaningful. I'm not aware of any other sysadmin certifications that work this way.

For one of the lab tests, students are given a several-page specification, and must install and configure Red Hat Linux and several network services. This wasn't particularly difficult, and shouldn't be for anyone with much experience. For me, the hardest part was resisting the temptation to go beyond the spec -- since I finished the given requirements with plenty of spare time, I considered installing and setting up additional services in a way that would fit in with the listed goals. But, I decided that it'd be better to leave well-enough alone -- there's no concept of extra credit.

The other hands-on test is the cool and exciting one. Students are given preconfigured setups which are broken in some way, and given a task that must be completed. The system's problem doesn't necessarily relate directly to the task, but does interfere with it. The test-taker must find out what's wrong and correct the error. (Reinstalling packages is not allowed.) Being able to list the steps taken and to repeat the fix is important, but ultimately the test is scored on a works / doesn't work basis. One the examiner verifies that the problem is fixed, he or she wipes the system and provides another broken config.

This problem-solving section directly tests skills important to being a sysadmin in the real world; if someone has trouble with these, they're probably not ready for a systems administration job. Of course, just passing this test doesn't guarantee good problem solving skills (let alone all the other needed abilities), but it does seem a genuinely valuable indicator.

I've only two complaints with this part of the test. First, I'd make it a much larger section -- at least 50% -- and I'd increase the number of problems given so that there'd be a better sample size. The various challenges are assigned at random, and some are easier than others, and each tests knowledge of different parts of the system. The way it's done isn't bad, but it wouldn't hurt to have a lot more of it. Second, I'd give each student two computers, and make more of the problems network-related. This has logistical and cost issues (especially in places other than Red Hat's own training centers), but since many of the problems faced in the real world have to do with the way systems interact, I feel it'd be worth it.

The Exam Separated From The Course

You may have noticed that I seem a lot more excited by the exam than by the course itself. I think both are valuable, but they seemed aimed at slightly different levels. The course definitely can serve as a good review for the exam, but if you need the course, you won't do well on the test. If you're tight on cash and the certification seems valuable to you or to your employer, going straight to the exam would be reasonable. (Make sure you take a look at Red Hat's test prep page.) On the other hand, if you need to be quickly brought up to speed on the basic knowledge required of a RH Linux sysadmin, it might make sense to take this course without worrying about the test. Since RH300 is equivalent to RH033 + RH133 + RH253, this could be a much more intensive and time-efficient option.

Red Hat-Specificness

It's probably obvious, but bears mentioning anyway: this is a Red Hat Linux course and certification, not a general Linux one. I found this to be true both explicitly and implicitly. The instructor was good about saying "This is the Red Hat way of doing things -- it's possibly different on other distributions." (I found the increase-the-whole-pie attitude to be common to all of the RH employees I talked to.) There were also quite a few things that were just assumed. If you take the exam without knowing a lot about Red Hat Linux in particular, you're likely to have trouble.

This doesn't make the certification meaningless for organizations running other distributions -- many of the skills and knowledge required for the test (especially the problem solving part) are generally applicable anywhere. In fact, due to the lab-based testing process, I have more respect for this exam than I might for a multiple-choice test covering more distributions. I think this issue is a one-way sort of thing: the RHCE exam requires knowledge of Red Hat Linux, but anyone who can pass it shouldn't have much trouble picking up other flavors.

Stuff

Ok, the Web page promises that they'll give Red Hat promotional items to course participants. Yeah, well, they can do better on this front. Not even a t-shirt! C'mon, everyone gives t-shirts. Vendor shirts are a staple of my wardrobe! All we got was a mousepad, some stickers, and a baseball cap. (No chance of getting a red fedora.) Oh, and of course an official copy of the CD (with the 180 days of support). Many people in the class were surprised to learn that Red Hat doesn't sell anything from their offices -- you can't buy copies of the distro or additional merchandise. They've got a lot of students coming through there, so it seems like this could be a decent (even if relatively small) revenue stream.

A Bit About Study Guides

Before I went, I flipped through RHCE Exam Cram , the sole study guide I found at the local bookstore. Someone in the class actually purchased it and brought it with them, and I got a chance to read more of it then. I wasn't really impressed. The book was especially concerned with what it called "trick questions", and indeed its sample questions were sometimes a bit confusing -- and often poorly worded. After taking the test, I can say that this seems mostly to be a problem with the book, not something encountered on the actual exam, which was mostly straightforward and fair.

There are RHCE study guides, but I wouldn't recommend spending any money on any of them. As the course instructor told us: if you're going to pass, you'll do so even if you don't have a guide. And if you're going to fail, the guide won't be much help.

Conclusion

I think the RH300 course and RHCE certification can be valuable to both employers and individuals. The course provides a nice quick overview of the basics needed to move, for example, from being a systems operator to being an admin. I wouldn't think of it as either a requirement for the test or as something that can make someone not ready suddenly have the skills required for the exam. Since the exam is hands-on and lab based, those abilities can only come from real world experience. Looking at that from the other direction: this is exactly what makes the RHCE worth anything. While it's not a total statement on someone's talent, being able to pass is a strong indicator that they have the basic skills for a systems administration job. If I were making hiring decisions, I wouldn't make the RHCE a requirement, but I would have more confidence in applicants who have it.

Matthew Miller recently went through the RH300 training course, as well as the RHCE Certification Exam. He was kind enough to write an overview and give us his opinions on both of them, as well as his opinions on the relevance and quality of the training and the exam. Certification has been discussed extensively with regards to Linux, and here's a big scoop of food for thought.

The following was written by Slashdot Reader Matthew Miller

I'm fortunate enough to work at a place that realizes the importance of keeping employees educated and up-to-date. Since my largest current project is Linux-related, and based on Red Hat's distribution in specific, we thought it'd be worthwhile to send me to Red Hat for their RH300 course. I'm pretty familiar with Linux, but I'm a long way from knowing everything, and it's always interesting to learn what the vendor thinks are the most important parts of their product. We chose RH300 because it's the highest-level systems administration class currently offered. It's also the one linked to the RHCE exam, which was an added bonus, but learning was my main goal, not getting the certification. This is my report on the experience -- hopefully, it will help you decide if this is a good choice for you, either as a sysadmin or as an employer.

The Training Center

This course is not only available directly from Red Hat, but also from various partner organizations, including Global Knowledge, which has a training center here in Boston. However, we decided that if we were going to go to the expense of sending me, I might as well go directly to Red Hat, to increase the chances of getting a good instructor, and to insure adequate access to resources. We've had experiences in the past with third-party instructors who didn't know much beyond what was written in the materials. Of course, I don't know that this would be the case with Global Knowledge's version of RH300 -- perhaps someone else can comment on any experience they've had there.

So, it was off to the Red Hat headquarters in Durham, NC. Incidentally, I stayed in the Residence Inn there -- it was on Red Hat's site as being nearby. They didn't mention that it was on the other side of a major highway, with no provision for pedestrians to get across. Moral: stay at one of the closer hotels, or else get a car. Anyway, the RH building is very nice -- much bigger than I expected. (I suppose the IPO cash is going to good use.) Of course, as students, we weren't shown much of it -- no tour, and we weren't introduced to any of the celebrity employees. (Fair enough -- with several classes coming through every week, they'd never get anything done.) The people I did meet seemed pretty cool, and in general I got the impression that it's a fun place to work.

The classroom was about as I expected -- projection screen up front, rows of decent-enough small-brand Celeron-based systems (one per student). The machines were on a private network -- reasonable for the course, but unfortunately there was no provision for Internet access, which at the least would have been nice to have when I finished labs early.

We did have access to a breakroom with free soft drinks / juice and various snack items. This is also where the lunches were served -- to my surprise, these were quite good, and there were even decent non-meat choices.

The Teacher

The instructor was very knowledgeable -- not necessarily a complete guru, but he knew his stuff, including the "why" behind the course material. He was able to present the material in a good way, and was good at answering questions. I think the decision to go to Red Hat directly was wise; unlike a third-party consultant, he had some idea of what was going on inside of Red Hat and of their potential future plans. For example, during the section on the printing subsystem, he mentioned that they're considering a replacement for LPR in future releases -- perhaps LPRng or even CUPS. It's unlikely that someone from a different company would have had access to that kind of information.

Other Students

The other students in the course had a wide range of skills and backgrounds. I think that everyone probably met the listed better than pico. However, I could tell that some people were struggling. The instructor mentioned that the pass rate for the exam is about 65%, and I wouldn't be surprised if our class came out at that level or worse. It's not that anyone was stupid -- just that some people were out of their depth. On the other end of the spectrum, there were some people who were over-qualified: a few highly experienced sysadmins, and some folks from IBM taking the class because they are soon going to teach it.

The Course

The course was generally similar to the outline found on Red Hat's site, although I think the online information is a bit out of date. (Notice that the Web page makes reference to ipfwadm instead of ipchains or netfilter.) The eight units had slightly different names, and covered slightly different information. In the most drastic example, Unit 8, listed on the Web site as "Systems Administration and Security II", has turned into "Routers, Firewalls, Clusters and Troubleshooting". Some of the information listed in the online Unit 8 was moved into Unit 7, and some of it (cops, for instance) wasn't talked about at all. Hopefully, the online info will be updated soon.

Overall, the class went into less depth than I was hoping. Some of this was due to limitations of the lab setup -- it's a bit difficult to experiment with RAID in any meaningful way when you've only got one IDE hard drive, and obviously impossible to set up a cluster on one machine (short of running VMware). Other things where just plain introductory -- the section on the kernel, for example, focused on the steps required to build and install a new kernel, rather than being an in-depth discussion of tunable parameters. The part about Apache was similar; I was hoping to hear "You've all configured Apache before; here's things you should be aware of when you need it to do such-and-such", but the most advanced we got was setting up a virtual host. Building RPMs from source was mentioned briefly, but there was no information given on important and largely undocumented topics like --buildpolicy.

That's not to say I didn't learn anything -- the section on LVS / Piranha was enlightening even without hands-on experience, and I appreciated the part about quotas, which isn't something I've worked with much. And, I learned a large number of tiny things which add up to making the experience worthwhile to me. RPM can now do globbing over ftp! Portmap uses tcp_wrappers, but doesn't do reverse name lookups, so be sure to use IP addresses instead of names. RH Linux provides a little script called "service" that lets one avoid the tedium of typing /etc/rc.d/init.d/servicename all the time. And so on....

The "300" designation is a bit misleading. This isn't really what I'd consider an upper-level course -- it's more along the lines of SysAdmin 101. Overall, I think this class is probably worthwhile to someone with a good RH Linux background who hasn't done any systems administration. In fact, I'd even recommend it to people in that situation. On the other hand, if you've been a Linux sysadmin for a while, you'll probably be bored most of the time. It might be valuable to experienced Unix sysadmins who haven't dealt with Linux much (or even Linux admins who haven't used Red Hat Linux), but the course wasn't particularly taught from that angle and there are probably better options.

The Exam

Since I signed a confidentiality agreement, I can't talk about specific details of the test, but I will address the exam in general terms. It's a day-long three part process, with each part being worth 1/3 of the total. To pass, your overall score must be at least 80%, and you can't do worse than 50% on any one part.

One of the sections is a typical multiple-choice test, but the other two are lab based. I was quite impressed with the hands-on tests -- they are certainly what makes the RHCE meaningful. I'm not aware of any other sysadmin certifications that work this way.

For one of the lab tests, students are given a several-page specification, and must install and configure Red Hat Linux and several network services. This wasn't particularly difficult, and shouldn't be for anyone with much experience. For me, the hardest part was resisting the temptation to go beyond the spec -- since I finished the given requirements with plenty of spare time, I considered installing and setting up additional services in a way that would fit in with the listed goals. But, I decided that it'd be better to leave well-enough alone -- there's no concept of extra credit.

The other hands-on test is the cool and exciting one. Students are given preconfigured setups which are broken in some way, and given a task that must be completed. The system's problem doesn't necessarily relate directly to the task, but does interfere with it. The test-taker must find out what's wrong and correct the error. (Reinstalling packages is not allowed.) Being able to list the steps taken and to repeat the fix is important, but ultimately the test is scored on a works / doesn't work basis. One the examiner verifies that the problem is fixed, he or she wipes the system and provides another broken config.

This problem-solving section directly tests skills important to being a sysadmin in the real world; if someone has trouble with these, they're probably not ready for a systems administration job. Of course, just passing this test doesn't guarantee good problem solving skills (let alone all the other needed abilities), but it does seem a genuinely valuable indicator.

I've only two complaints with this part of the test. First, I'd make it a much larger section -- at least 50% -- and I'd increase the number of problems given so that there'd be a better sample size. The various challenges are assigned at random, and some are easier than others, and each tests knowledge of different parts of the system. The way it's done isn't bad, but it wouldn't hurt to have a lot more of it. Second, I'd give each student two computers, and make more of the problems network-related. This has logistical and cost issues (especially in places other than Red Hat's own training centers), but since many of the problems faced in the real world have to do with the way systems interact, I feel it'd be worth it.

The Exam Separated From The Course

You may have noticed that I seem a lot more excited by the exam than by the course itself. I think both are valuable, but they seemed aimed at slightly different levels. The course definitely can serve as a good review for the exam, but if you need the course, you won't do well on the test. If you're tight on cash and the certification seems valuable to you or to your employer, going straight to the exam would be reasonable. (Make sure you take a look at Red Hat's test prep page.) On the other hand, if you need to be quickly brought up to speed on the basic knowledge required of a RH Linux sysadmin, it might make sense to take this course without worrying about the test. Since RH300 is equivalent to RH033 + RH133 + RH253, this could be a much more intensive and time-efficient option.

Red Hat-Specificness

It's probably obvious, but bears mentioning anyway: this is a Red Hat Linux course and certification, not a general Linux one. I found this to be true both explicitly and implicitly. The instructor was good about saying "This is the Red Hat way of doing things -- it's possibly different on other distributions." (I found the increase-the-whole-pie attitude to be common to all of the RH employees I talked to.) There were also quite a few things that were just assumed. If you take the exam without knowing a lot about Red Hat Linux in particular, you're likely to have trouble.

This doesn't make the certification meaningless for organizations running other distributions -- many of the skills and knowledge required for the test (especially the problem solving part) are generally applicable anywhere. In fact, due to the lab-based testing process, I have more respect for this exam than I might for a multiple-choice test covering more distributions. I think this issue is a one-way sort of thing: the RHCE exam requires knowledge of Red Hat Linux, but anyone who can pass it shouldn't have much trouble picking up other flavors.

Stuff

Ok, the Web page promises that they'll give Red Hat promotional items to course participants. Yeah, well, they can do better on this front. Not even a t-shirt! C'mon, everyone gives t-shirts. Vendor shirts are a staple of my wardrobe! All we got was a mousepad, some stickers, and a baseball cap. (No chance of getting a red fedora.) Oh, and of course an official copy of the CD (with the 180 days of support). Many people in the class were surprised to learn that Red Hat doesn't sell anything from their offices -- you can't buy copies of the distro or additional merchandise. They've got a lot of students coming through there, so it seems like this could be a decent (even if relatively small) revenue stream.

A Bit About Study Guides

Before I went, I flipped through RHCE Exam Cram , the sole study guide I found at the local bookstore. Someone in the class actually purchased it and brought it with them, and I got a chance to read more of it then. I wasn't really impressed. The book was especially concerned with what it called "trick questions", and indeed its sample questions were sometimes a bit confusing -- and often poorly worded. After taking the test, I can say that this seems mostly to be a problem with the book, not something encountered on the actual exam, which was mostly straightforward and fair.

There are RHCE study guides, but I wouldn't recommend spending any money on any of them. As the course instructor told us: if you're going to pass, you'll do so even if you don't have a guide. And if you're going to fail, the guide won't be much help.

Conclusion

I think the RH300 course and RHCE certification can be valuable to both employers and individuals. The course provides a nice quick overview of the basics needed to move, for example, from being a systems operator to being an admin. I wouldn't think of it as either a requirement for the test or as something that can make someone not ready suddenly have the skills required for the exam. Since the exam is hands-on and lab based, those abilities can only come from real world experience. Looking at that from the other direction: this is exactly what makes the RHCE worth anything. While it's not a total statement on someone's talent, being able to pass is a strong indicator that they have the basic skills for a systems administration job. If I were making hiring decisions, I wouldn't make the RHCE a requirement, but I would have more confidence in applicants who have it.

Matthew Miller recently went through the RH300 training course, as well as the RHCE Certification Exam. He was kind enough to write an overview and give us his opinions on both of them, as well as his opinions on the relevance and quality of the training and the exam. Certification has been discussed extensively with regards to Linux, and here's a big scoop of food for thought.

The following was written by Slashdot Reader Matthew Miller

I'm fortunate enough to work at a place that realizes the importance of keeping employees educated and up-to-date. Since my largest current project is Linux-related, and based on Red Hat's distribution in specific, we thought it'd be worthwhile to send me to Red Hat for their RH300 course. I'm pretty familiar with Linux, but I'm a long way from knowing everything, and it's always interesting to learn what the vendor thinks are the most important parts of their product. We chose RH300 because it's the highest-level systems administration class currently offered. It's also the one linked to the RHCE exam, which was an added bonus, but learning was my main goal, not getting the certification. This is my report on the experience -- hopefully, it will help you decide if this is a good choice for you, either as a sysadmin or as an employer.

The Training Center

This course is not only available directly from Red Hat, but also from various partner organizations, including Global Knowledge, which has a training center here in Boston. However, we decided that if we were going to go to the expense of sending me, I might as well go directly to Red Hat, to increase the chances of getting a good instructor, and to insure adequate access to resources. We've had experiences in the past with third-party instructors who didn't know much beyond what was written in the materials. Of course, I don't know that this would be the case with Global Knowledge's version of RH300 -- perhaps someone else can comment on any experience they've had there.

So, it was off to the Red Hat headquarters in Durham, NC. Incidentally, I stayed in the Residence Inn there -- it was on Red Hat's site as being nearby. They didn't mention that it was on the other side of a major highway, with no provision for pedestrians to get across. Moral: stay at one of the closer hotels, or else get a car. Anyway, the RH building is very nice -- much bigger than I expected. (I suppose the IPO cash is going to good use.) Of course, as students, we weren't shown much of it -- no tour, and we weren't introduced to any of the celebrity employees. (Fair enough -- with several classes coming through every week, they'd never get anything done.) The people I did meet seemed pretty cool, and in general I got the impression that it's a fun place to work.

The classroom was about as I expected -- projection screen up front, rows of decent-enough small-brand Celeron-based systems (one per student). The machines were on a private network -- reasonable for the course, but unfortunately there was no provision for Internet access, which at the least would have been nice to have when I finished labs early.

We did have access to a breakroom with free soft drinks / juice and various snack items. This is also where the lunches were served -- to my surprise, these were quite good, and there were even decent non-meat choices.

The Teacher

The instructor was very knowledgeable -- not necessarily a complete guru, but he knew his stuff, including the "why" behind the course material. He was able to present the material in a good way, and was good at answering questions. I think the decision to go to Red Hat directly was wise; unlike a third-party consultant, he had some idea of what was going on inside of Red Hat and of their potential future plans. For example, during the section on the printing subsystem, he mentioned that they're considering a replacement for LPR in future releases -- perhaps LPRng or even CUPS. It's unlikely that someone from a different company would have had access to that kind of information.

Other Students

The other students in the course had a wide range of skills and backgrounds. I think that everyone probably met the listed better than pico. However, I could tell that some people were struggling. The instructor mentioned that the pass rate for the exam is about 65%, and I wouldn't be surprised if our class came out at that level or worse. It's not that anyone was stupid -- just that some people were out of their depth. On the other end of the spectrum, there were some people who were over-qualified: a few highly experienced sysadmins, and some folks from IBM taking the class because they are soon going to teach it.

The Course

The course was generally similar to the outline found on Red Hat's site, although I think the online information is a bit out of date. (Notice that the Web page makes reference to ipfwadm instead of ipchains or netfilter.) The eight units had slightly different names, and covered slightly different information. In the most drastic example, Unit 8, listed on the Web site as "Systems Administration and Security II", has turned into "Routers, Firewalls, Clusters and Troubleshooting". Some of the information listed in the online Unit 8 was moved into Unit 7, and some of it (cops, for instance) wasn't talked about at all. Hopefully, the online info will be updated soon.

Overall, the class went into less depth than I was hoping. Some of this was due to limitations of the lab setup -- it's a bit difficult to experiment with RAID in any meaningful way when you've only got one IDE hard drive, and obviously impossible to set up a cluster on one machine (short of running VMware). Other things where just plain introductory -- the section on the kernel, for example, focused on the steps required to build and install a new kernel, rather than being an in-depth discussion of tunable parameters. The part about Apache was similar; I was hoping to hear "You've all configured Apache before; here's things you should be aware of when you need it to do such-and-such", but the most advanced we got was setting up a virtual host. Building RPMs from source was mentioned briefly, but there was no information given on important and largely undocumented topics like --buildpolicy.

That's not to say I didn't learn anything -- the section on LVS / Piranha was enlightening even without hands-on experience, and I appreciated the part about quotas, which isn't something I've worked with much. And, I learned a large number of tiny things which add up to making the experience worthwhile to me. RPM can now do globbing over ftp! Portmap uses tcp_wrappers, but doesn't do reverse name lookups, so be sure to use IP addresses instead of names. RH Linux provides a little script called "service" that lets one avoid the tedium of typing /etc/rc.d/init.d/servicename all the time. And so on....

The "300" designation is a bit misleading. This isn't really what I'd consider an upper-level course -- it's more along the lines of SysAdmin 101. Overall, I think this class is probably worthwhile to someone with a good RH Linux background who hasn't done any systems administration. In fact, I'd even recommend it to people in that situation. On the other hand, if you've been a Linux sysadmin for a while, you'll probably be bored most of the time. It might be valuable to experienced Unix sysadmins who haven't dealt with Linux much (or even Linux admins who haven't used Red Hat Linux), but the course wasn't particularly taught from that angle and there are probably better options.

The Exam

Since I signed a confidentiality agreement, I can't talk about specific details of the test, but I will address the exam in general terms. It's a day-long three part process, with each part being worth 1/3 of the total. To pass, your overall score must be at least 80%, and you can't do worse than 50% on any one part.

One of the sections is a typical multiple-choice test, but the other two are lab based. I was quite impressed with the hands-on tests -- they are certainly what makes the RHCE meaningful. I'm not aware of any other sysadmin certifications that work this way.

For one of the lab tests, students are given a several-page specification, and must install and configure Red Hat Linux and several network services. This wasn't particularly difficult, and shouldn't be for anyone with much experience. For me, the hardest part was resisting the temptation to go beyond the spec -- since I finished the given requirements with plenty of spare time, I considered installing and setting up additional services in a way that would fit in with the listed goals. But, I decided that it'd be better to leave well-enough alone -- there's no concept of extra credit.

The other hands-on test is the cool and exciting one. Students are given preconfigured setups which are broken in some way, and given a task that must be completed. The system's problem doesn't necessarily relate directly to the task, but does interfere with it. The test-taker must find out what's wrong and correct the error. (Reinstalling packages is not allowed.) Being able to list the steps taken and to repeat the fix is important, but ultimately the test is scored on a works / doesn't work basis. One the examiner verifies that the problem is fixed, he or she wipes the system and provides another broken config.

This problem-solving section directly tests skills important to being a sysadmin in the real world; if someone has trouble with these, they're probably not ready for a systems administration job. Of course, just passing this test doesn't guarantee good problem solving skills (let alone all the other needed abilities), but it does seem a genuinely valuable indicator.

I've only two complaints with this part of the test. First, I'd make it a much larger section -- at least 50% -- and I'd increase the number of problems given so that there'd be a better sample size. The various challenges are assigned at random, and some are easier than others, and each tests knowledge of different parts of the system. The way it's done isn't bad, but it wouldn't hurt to have a lot more of it. Second, I'd give each student two computers, and make more of the problems network-related. This has logistical and cost issues (especially in places other than Red Hat's own training centers), but since many of the problems faced in the real world have to do with the way systems interact, I feel it'd be worth it.

The Exam Separated From The Course

You may have noticed that I seem a lot more excited by the exam than by the course itself. I think both are valuable, but they seemed aimed at slightly different levels. The course definitely can serve as a good review for the exam, but if you need the course, you won't do well on the test. If you're tight on cash and the certification seems valuable to you or to your employer, going straight to the exam would be reasonable. (Make sure you take a look at Red Hat's test prep page.) On the other hand, if you need to be quickly brought up to speed on the basic knowledge required of a RH Linux sysadmin, it might make sense to take this course without worrying about the test. Since RH300 is equivalent to RH033 + RH133 + RH253, this could be a much more intensive and time-efficient option.

Red Hat-Specificness

It's probably obvious, but bears mentioning anyway: this is a Red Hat Linux course and certification, not a general Linux one. I found this to be true both explicitly and implicitly. The instructor was good about saying "This is the Red Hat way of doing things -- it's possibly different on other distributions." (I found the increase-the-whole-pie attitude to be common to all of the RH employees I talked to.) There were also quite a few things that were just assumed. If you take the exam without knowing a lot about Red Hat Linux in particular, you're likely to have trouble.

This doesn't make the certification meaningless for organizations running other distributions -- many of the skills and knowledge required for the test (especially the problem solving part) are generally applicable anywhere. In fact, due to the lab-based testing process, I have more respect for this exam than I might for a multiple-choice test covering more distributions. I think this issue is a one-way sort of thing: the RHCE exam requires knowledge of Red Hat Linux, but anyone who can pass it shouldn't have much trouble picking up other flavors.

Stuff

Ok, the Web page promises that they'll give Red Hat promotional items to course participants. Yeah, well, they can do better on this front. Not even a t-shirt! C'mon, everyone gives t-shirts. Vendor shirts are a staple of my wardrobe! All we got was a mousepad, some stickers, and a baseball cap. (No chance of getting a red fedora.) Oh, and of course an official copy of the CD (with the 180 days of support). Many people in the class were surprised to learn that Red Hat doesn't sell anything from their offices -- you can't buy copies of the distro or additional merchandise. They've got a lot of students coming through there, so it seems like this could be a decent (even if relatively small) revenue stream.

A Bit About Study Guides

Before I went, I flipped through RHCE Exam Cram , the sole study guide I found at the local bookstore. Someone in the class actually purchased it and brought it with them, and I got a chance to read more of it then. I wasn't really impressed. The book was especially concerned with what it called "trick questions", and indeed its sample questions were sometimes a bit confusing -- and often poorly worded. After taking the test, I can say that this seems mostly to be a problem with the book, not something encountered on the actual exam, which was mostly straightforward and fair.

There are RHCE study guides, but I wouldn't recommend spending any money on any of them. As the course instructor told us: if you're going to pass, you'll do so even if you don't have a guide. And if you're going to fail, the guide won't be much help.

Conclusion

I think the RH300 course and RHCE certification can be valuable to both employers and individuals. The course provides a nice quick overview of the basics needed to move, for example, from being a systems operator to being an admin. I wouldn't think of it as either a requirement for the test or as something that can make someone not ready suddenly have the skills required for the exam. Since the exam is hands-on and lab based, those abilities can only come from real world experience. Looking at that from the other direction: this is exactly what makes the RHCE worth anything. While it's not a total statement on someone's talent, being able to pass is a strong indicator that they have the basic skills for a systems administration job. If I were making hiring decisions, I wouldn't make the RHCE a requirement, but I would have more confidence in applicants who have it.

Matthew Miller recently went through the RH300 training course, as well as the RHCE Certification Exam. He was kind enough to write an overview and give us his opinions on both of them, as well as his opinions on the relevance and quality of the training and the exam. Certification has been discussed extensively with regards to Linux, and here's a big scoop of food for thought.

The following was written by Slashdot Reader Matthew Miller

I'm fortunate enough to work at a place that realizes the importance of keeping employees educated and up-to-date. Since my largest current project is Linux-related, and based on Red Hat's distribution in specific, we thought it'd be worthwhile to send me to Red Hat for their RH300 course. I'm pretty familiar with Linux, but I'm a long way from knowing everything, and it's always interesting to learn what the vendor thinks are the most important parts of their product. We chose RH300 because it's the highest-level systems administration class currently offered. It's also the one linked to the RHCE exam, which was an added bonus, but learning was my main goal, not getting the certification. This is my report on the experience -- hopefully, it will help you decide if this is a good choice for you, either as a sysadmin or as an employer.

The Training Center

This course is not only available directly from Red Hat, but also from various partner organizations, including Global Knowledge, which has a training center here in Boston. However, we decided that if we were going to go to the expense of sending me, I might as well go directly to Red Hat, to increase the chances of getting a good instructor, and to insure adequate access to resources. We've had experiences in the past with third-party instructors who didn't know much beyond what was written in the materials. Of course, I don't know that this would be the case with Global Knowledge's version of RH300 -- perhaps someone else can comment on any experience they've had there.

So, it was off to the Red Hat headquarters in Durham, NC. Incidentally, I stayed in the Residence Inn there -- it was on Red Hat's site as being nearby. They didn't mention that it was on the other side of a major highway, with no provision for pedestrians to get across. Moral: stay at one of the closer hotels, or else get a car. Anyway, the RH building is very nice -- much bigger than I expected. (I suppose the IPO cash is going to good use.) Of course, as students, we weren't shown much of it -- no tour, and we weren't introduced to any of the celebrity employees. (Fair enough -- with several classes coming through every week, they'd never get anything done.) The people I did meet seemed pretty cool, and in general I got the impression that it's a fun place to work.

The classroom was about as I expected -- projection screen up front, rows of decent-enough small-brand Celeron-based systems (one per student). The machines were on a private network -- reasonable for the course, but unfortunately there was no provision for Internet access, which at the least would have been nice to have when I finished labs early.

We did have access to a breakroom with free soft drinks / juice and various snack items. This is also where the lunches were served -- to my surprise, these were quite good, and there were even decent non-meat choices.

The Teacher

The instructor was very knowledgeable -- not necessarily a complete guru, but he knew his stuff, including the "why" behind the course material. He was able to present the material in a good way, and was good at answering questions. I think the decision to go to Red Hat directly was wise; unlike a third-party consultant, he had some idea of what was going on inside of Red Hat and of their potential future plans. For example, during the section on the printing subsystem, he mentioned that they're considering a replacement for LPR in future releases -- perhaps LPRng or even CUPS. It's unlikely that someone from a different company would have had access to that kind of information.

Other Students

The other students in the course had a wide range of skills and backgrounds. I think that everyone probably met the listed better than pico. However, I could tell that some people were struggling. The instructor mentioned that the pass rate for the exam is about 65%, and I wouldn't be surprised if our class came out at that level or worse. It's not that anyone was stupid -- just that some people were out of their depth. On the other end of the spectrum, there were some people who were over-qualified: a few highly experienced sysadmins, and some folks from IBM taking the class because they are soon going to teach it.

The Course

The course was generally similar to the outline found on Red Hat's site, although I think the online information is a bit out of date. (Notice that the Web page makes reference to ipfwadm instead of ipchains or netfilter.) The eight units had slightly different names, and covered slightly different information. In the most drastic example, Unit 8, listed on the Web site as "Systems Administration and Security II", has turned into "Routers, Firewalls, Clusters and Troubleshooting". Some of the information listed in the online Unit 8 was moved into Unit 7, and some of it (cops, for instance) wasn't talked about at all. Hopefully, the online info will be updated soon.

Overall, the class went into less depth than I was hoping. Some of this was due to limitations of the lab setup -- it's a bit difficult to experiment with RAID in any meaningful way when you've only got one IDE hard drive, and obviously impossible to set up a cluster on one machine (short of running VMware). Other things where just plain introductory -- the section on the kernel, for example, focused on the steps required to build and install a new kernel, rather than being an in-depth discussion of tunable parameters. The part about Apache was similar; I was hoping to hear "You've all configured Apache before; here's things you should be aware of when you need it to do such-and-such", but the most advanced we got was setting up a virtual host. Building RPMs from source was mentioned briefly, but there was no information given on important and largely undocumented topics like --buildpolicy.

That's not to say I didn't learn anything -- the section on LVS / Piranha was enlightening even without hands-on experience, and I appreciated the part about quotas, which isn't something I've worked with much. And, I learned a large number of tiny things which add up to making the experience worthwhile to me. RPM can now do globbing over ftp! Portmap uses tcp_wrappers, but doesn't do reverse name lookups, so be sure to use IP addresses instead of names. RH Linux provides a little script called "service" that lets one avoid the tedium of typing /etc/rc.d/init.d/servicename all the time. And so on....

The "300" designation is a bit misleading. This isn't really what I'd consider an upper-level course -- it's more along the lines of SysAdmin 101. Overall, I think this class is probably worthwhile to someone with a good RH Linux background who hasn't done any systems administration. In fact, I'd even recommend it to people in that situation. On the other hand, if you've been a Linux sysadmin for a while, you'll probably be bored most of the time. It might be valuable to experienced Unix sysadmins who haven't dealt with Linux much (or even Linux admins who haven't used Red Hat Linux), but the course wasn't particularly taught from that angle and there are probably better options.

The Exam

Since I signed a confidentiality agreement, I can't talk about specific details of the test, but I will address the exam in general terms. It's a day-long three part process, with each part being worth 1/3 of the total. To pass, your overall score must be at least 80%, and you can't do worse than 50% on any one part.

One of the sections is a typical multiple-choice test, but the other two are lab based. I was quite impressed with the hands-on tests -- they are certainly what makes the RHCE meaningful. I'm not aware of any other sysadmin certifications that work this way.

For one of the lab tests, students are given a several-page specification, and must install and configure Red Hat Linux and several network services. This wasn't particularly difficult, and shouldn't be for anyone with much experience. For me, the hardest part was resisting the temptation to go beyond the spec -- since I finished the given requirements with plenty of spare time, I considered installing and setting up additional services in a way that would fit in with the listed goals. But, I decided that it'd be better to leave well-enough alone -- there's no concept of extra credit.

The other hands-on test is the cool and exciting one. Students are given preconfigured setups which are broken in some way, and given a task that must be completed. The system's problem doesn't necessarily relate directly to the task, but does interfere with it. The test-taker must find out what's wrong and correct the error. (Reinstalling packages is not allowed.) Being able to list the steps taken and to repeat the fix is important, but ultimately the test is scored on a works / doesn't work basis. One the examiner verifies that the problem is fixed, he or she wipes the system and provides another broken config.

This problem-solving section directly tests skills important to being a sysadmin in the real world; if someone has trouble with these, they're probably not ready for a systems administration job. Of course, just passing this test doesn't guarantee good problem solving skills (let alone all the other needed abilities), but it does seem a genuinely valuable indicator.

I've only two complaints with this part of the test. First, I'd make it a much larger section -- at least 50% -- and I'd increase the number of problems given so that there'd be a better sample size. The various challenges are assigned at random, and some are easier than others, and each tests knowledge of different parts of the system. The way it's done isn't bad, but it wouldn't hurt to have a lot more of it. Second, I'd give each student two computers, and make more of the problems network-related. This has logistical and cost issues (especially in places other than Red Hat's own training centers), but since many of the problems faced in the real world have to do with the way systems interact, I feel it'd be worth it.

The Exam Separated From The Course

You may have noticed that I seem a lot more excited by the exam than by the course itself. I think both are valuable, but they seemed aimed at slightly different levels. The course definitely can serve as a good review for the exam, but if you need the course, you won't do well on the test. If you're tight on cash and the certification seems valuable to you or to your employer, going straight to the exam would be reasonable. (Make sure you take a look at Red Hat's test prep page.) On the other hand, if you need to be quickly brought up to speed on the basic knowledge required of a RH Linux sysadmin, it might make sense to take this course without worrying about the test. Since RH300 is equivalent to RH033 + RH133 + RH253, this could be a much more intensive and time-efficient option.

Red Hat-Specificness

It's probably obvious, but bears mentioning anyway: this is a Red Hat Linux course and certification, not a general Linux one. I found this to be true both explicitly and implicitly. The instructor was good about saying "This is the Red Hat way of doing things -- it's possibly different on other distributions." (I found the increase-the-whole-pie attitude to be common to all of the RH employees I talked to.) There were also quite a few things that were just assumed. If you take the exam without knowing a lot about Red Hat Linux in particular, you're likely to have trouble.

This doesn't make the certification meaningless for organizations running other distributions -- many of the skills and knowledge required for the test (especially the problem solving part) are generally applicable anywhere. In fact, due to the lab-based testing process, I have more respect for this exam than I might for a multiple-choice test covering more distributions. I think this issue is a one-way sort of thing: the RHCE exam requires knowledge of Red Hat Linux, but anyone who can pass it shouldn't have much trouble picking up other flavors.

Stuff

Ok, the Web page promises that they'll give Red Hat promotional items to course participants. Yeah, well, they can do better on this front. Not even a t-shirt! C'mon, everyone gives t-shirts. Vendor shirts are a staple of my wardrobe! All we got was a mousepad, some stickers, and a baseball cap. (No chance of getting a red fedora.) Oh, and of course an official copy of the CD (with the 180 days of support). Many people in the class were surprised to learn that Red Hat doesn't sell anything from their offices -- you can't buy copies of the distro or additional merchandise. They've got a lot of students coming through there, so it seems like this could be a decent (even if relatively small) revenue stream.

A Bit About Study Guides

Before I went, I flipped through RHCE Exam Cram , the sole study guide I found at the local bookstore. Someone in the class actually purchased it and brought it with them, and I got a chance to read more of it then. I wasn't really impressed. The book was especially concerned with what it called "trick questions", and indeed its sample questions were sometimes a bit confusing -- and often poorly worded. After taking the test, I can say that this seems mostly to be a problem with the book, not something encountered on the actual exam, which was mostly straightforward and fair.

There are RHCE study guides, but I wouldn't recommend spending any money on any of them. As the course instructor told us: if you're going to pass, you'll do so even if you don't have a guide. And if you're going to fail, the guide won't be much help.

Conclusion

I think the RH300 course and RHCE certification can be valuable to both employers and individuals. The course provides a nice quick overview of the basics needed to move, for example, from being a systems operator to being an admin. I wouldn't think of it as either a requirement for the test or as something that can make someone not ready suddenly have the skills required for the exam. Since the exam is hands-on and lab based, those abilities can only come from real world experience. Looking at that from the other direction: this is exactly what makes the RHCE worth anything. While it's not a total statement on someone's talent, being able to pass is a strong indicator that they have the basic skills for a systems administration job. If I were making hiring decisions, I wouldn't make the RHCE a requirement, but I would have more confidence in applicants who have it.

Matthew Miller recently went through the RH300 training course, as well as the RHCE Certification Exam. He was kind enough to write an overview and give us his opinions on both of them, as well as his opinions on the relevance and quality of the training and the exam. Certification has been discussed extensively with regards to Linux, and here's a big scoop of food for thought.

The following was written by Slashdot Reader Matthew Miller

I'm fortunate enough to work at a place that realizes the importance of keeping employees educated and up-to-date. Since my largest current project is Linux-related, and based on Red Hat's distribution in specific, we thought it'd be worthwhile to send me to Red Hat for their RH300 course. I'm pretty familiar with Linux, but I'm a long way from knowing everything, and it's always interesting to learn what the vendor thinks are the most important parts of their product. We chose RH300 because it's the highest-level systems administration class currently offered. It's also the one linked to the RHCE exam, which was an added bonus, but learning was my main goal, not getting the certification. This is my report on the experience -- hopefully, it will help you decide if this is a good choice for you, either as a sysadmin or as an employer.

The Training Center

This course is not only available directly from Red Hat, but also from various partner organizations, including Global Knowledge, which has a training center here in Boston. However, we decided that if we were going to go to the expense of sending me, I might as well go directly to Red Hat, to increase the chances of getting a good instructor, and to insure adequate access to resources. We've had experiences in the past with third-party instructors who didn't know much beyond what was written in the materials. Of course, I don't know that this would be the case with Global Knowledge's version of RH300 -- perhaps someone else can comment on any experience they've had there.

So, it was off to the Red Hat headquarters in Durham, NC. Incidentally, I stayed in the Residence Inn there -- it was on Red Hat's site as being nearby. They didn't mention that it was on the other side of a major highway, with no provision for pedestrians to get across. Moral: stay at one of the closer hotels, or else get a car. Anyway, the RH building is very nice -- much bigger than I expected. (I suppose the IPO cash is going to good use.) Of course, as students, we weren't shown much of it -- no tour, and we weren't introduced to any of the celebrity employees. (Fair enough -- with several classes coming through every week, they'd never get anything done.) The people I did meet seemed pretty cool, and in general I got the impression that it's a fun place to work.

The classroom was about as I expected -- projection screen up front, rows of decent-enough small-brand Celeron-based systems (one per student). The machines were on a private network -- reasonable for the course, but unfortunately there was no provision for Internet access, which at the least would have been nice to have when I finished labs early.

We did have access to a breakroom with free soft drinks / juice and various snack items. This is also where the lunches were served -- to my surprise, these were quite good, and there were even decent non-meat choices.

The Teacher

The instructor was very knowledgeable -- not necessarily a complete guru, but he knew his stuff, including the "why" behind the course material. He was able to present the material in a good way, and was good at answering questions. I think the decision to go to Red Hat directly was wise; unlike a third-party consultant, he had some idea of what was going on inside of Red Hat and of their potential future plans. For example, during the section on the printing subsystem, he mentioned that they're considering a replacement for LPR in future releases -- perhaps LPRng or even CUPS. It's unlikely that someone from a different company would have had access to that kind of information.

Other Students

The other students in the course had a wide range of skills and backgrounds. I think that everyone probably met the listed better than pico. However, I could tell that some people were struggling. The instructor mentioned that the pass rate for the exam is about 65%, and I wouldn't be surprised if our class came out at that level or worse. It's not that anyone was stupid -- just that some people were out of their depth. On the other end of the spectrum, there were some people who were over-qualified: a few highly experienced sysadmins, and some folks from IBM taking the class because they are soon going to teach it.

The Course

The course was generally similar to the outline found on Red Hat's site, although I think the online information is a bit out of date. (Notice that the Web page makes reference to ipfwadm instead of ipchains or netfilter.) The eight units had slightly different names, and covered slightly different information. In the most drastic example, Unit 8, listed on the Web site as "Systems Administration and Security II", has turned into "Routers, Firewalls, Clusters and Troubleshooting". Some of the information listed in the online Unit 8 was moved into Unit 7, and some of it (cops, for instance) wasn't talked about at all. Hopefully, the online info will be updated soon.

Overall, the class went into less depth than I was hoping. Some of this was due to limitations of the lab setup -- it's a bit difficult to experiment with RAID in any meaningful way when you've only got one IDE hard drive, and obviously impossible to set up a cluster on one machine (short of running VMware). Other things where just plain introductory -- the section on the kernel, for example, focused on the steps required to build and install a new kernel, rather than being an in-depth discussion of tunable parameters. The part about Apache was similar; I was hoping to hear "You've all configured Apache before; here's things you should be aware of when you need it to do such-and-such", but the most advanced we got was setting up a virtual host. Building RPMs from source was mentioned briefly, but there was no information given on important and largely undocumented topics like --buildpolicy.

That's not to say I didn't learn anything -- the section on LVS / Piranha was enlightening even without hands-on experience, and I appreciated the part about quotas, which isn't something I've worked with much. And, I learned a large number of tiny things which add up to making the experience worthwhile to me. RPM can now do globbing over ftp! Portmap uses tcp_wrappers, but doesn't do reverse name lookups, so be sure to use IP addresses instead of names. RH Linux provides a little script called "service" that lets one avoid the tedium of typing /etc/rc.d/init.d/servicename all the time. And so on....

The "300" designation is a bit misleading. This isn't really what I'd consider an upper-level course -- it's more along the lines of SysAdmin 101. Overall, I think this class is probably worthwhile to someone with a good RH Linux background who hasn't done any systems administration. In fact, I'd even recommend it to people in that situation. On the other hand, if you've been a Linux sysadmin for a while, you'll probably be bored most of the time. It might be valuable to experienced Unix sysadmins who haven't dealt with Linux much (or even Linux admins who haven't used Red Hat Linux), but the course wasn't particularly taught from that angle and there are probably better options.

The Exam

Since I signed a confidentiality agreement, I can't talk about specific details of the test, but I will address the exam in general terms. It's a day-long three part process, with each part being worth 1/3 of the total. To pass, your overall score must be at least 80%, and you can't do worse than 50% on any one part.

One of the sections is a typical multiple-choice test, but the other two are lab based. I was quite impressed with the hands-on tests -- they are certainly what makes the RHCE meaningful. I'm not aware of any other sysadmin certifications that work this way.

For one of the lab tests, students are given a several-page specification, and must install and configure Red Hat Linux and several network services. This wasn't particularly difficult, and shouldn't be for anyone with much experience. For me, the hardest part was resisting the temptation to go beyond the spec -- since I finished the given requirements with plenty of spare time, I considered installing and setting up additional services in a way that would fit in with the listed goals. But, I decided that it'd be better to leave well-enough alone -- there's no concept of extra credit.

The other hands-on test is the cool and exciting one. Students are given preconfigured setups which are broken in some way, and given a task that must be completed. The system's problem doesn't necessarily relate directly to the task, but does interfere with it. The test-taker must find out what's wrong and correct the error. (Reinstalling packages is not allowed.) Being able to list the steps taken and to repeat the fix is important, but ultimately the test is scored on a works / doesn't work basis. One the examiner verifies that the problem is fixed, he or she wipes the system and provides another broken config.

This problem-solving section directly tests skills important to being a sysadmin in the real world; if someone has trouble with these, they're probably not ready for a systems administration job. Of course, just passing this test doesn't guarantee good problem solving skills (let alone all the other needed abilities), but it does seem a genuinely valuable indicator.

I've only two complaints with this part of the test. First, I'd make it a much larger section -- at least 50% -- and I'd increase the number of problems given so that there'd be a better sample size. The various challenges are assigned at random, and some are easier than others, and each tests knowledge of different parts of the system. The way it's done isn't bad, but it wouldn't hurt to have a lot more of it. Second, I'd give each student two computers, and make more of the problems network-related. This has logistical and cost issues (especially in places other than Red Hat's own training centers), but since many of the problems faced in the real world have to do with the way systems interact, I feel it'd be worth it.

The Exam Separated From The Course

You may have noticed that I seem a lot more excited by the exam than by the course itself. I think both are valuable, but they seemed aimed at slightly different levels. The course definitely can serve as a good review for the exam, but if you need the course, you won't do well on the test. If you're tight on cash and the certification seems valuable to you or to your employer, going straight to the exam would be reasonable. (Make sure you take a look at Red Hat's test prep page.) On the other hand, if you need to be quickly brought up to speed on the basic knowledge required of a RH Linux sysadmin, it might make sense to take this course without worrying about the test. Since RH300 is equivalent to RH033 + RH133 + RH253, this could be a much more intensive and time-efficient option.

Red Hat-Specificness

It's probably obvious, but bears mentioning anyway: this is a Red Hat Linux course and certification, not a general Linux one. I found this to be true both explicitly and implicitly. The instructor was good about saying "This is the Red Hat way of doing things -- it's possibly different on other distributions." (I found the increase-the-whole-pie attitude to be common to all of the RH employees I talked to.) There were also quite a few things that were just assumed. If you take the exam without knowing a lot about Red Hat Linux in particular, you're likely to have trouble.

This doesn't make the certification meaningless for organizations running other distributions -- many of the skills and knowledge required for the test (especially the problem solving part) are generally applicable anywhere. In fact, due to the lab-based testing process, I have more respect for this exam than I might for a multiple-choice test covering more distributions. I think this issue is a one-way sort of thing: the RHCE exam requires knowledge of Red Hat Linux, but anyone who can pass it shouldn't have much trouble picking up other flavors.

Stuff

Ok, the Web page promises that they'll give Red Hat promotional items to course participants. Yeah, well, they can do better on this front. Not even a t-shirt! C'mon, everyone gives t-shirts. Vendor shirts are a staple of my wardrobe! All we got was a mousepad, some stickers, and a baseball cap. (No chance of getting a red fedora.) Oh, and of course an official copy of the CD (with the 180 days of support). Many people in the class were surprised to learn that Red Hat doesn't sell anything from their offices -- you can't buy copies of the distro or additional merchandise. They've got a lot of students coming through there, so it seems like this could be a decent (even if relatively small) revenue stream.

A Bit About Study Guides

Before I went, I flipped through RHCE Exam Cram , the sole study guide I found at the local bookstore. Someone in the class actually purchased it and brought it with them, and I got a chance to read more of it then. I wasn't really impressed. The book was especially concerned with what it called "trick questions", and indeed its sample questions were sometimes a bit confusing -- and often poorly worded. After taking the test, I can say that this seems mostly to be a problem with the book, not something encountered on the actual exam, which was mostly straightforward and fair.

There are RHCE study guides, but I wouldn't recommend spending any money on any of them. As the course instructor told us: if you're going to pass, you'll do so even if you don't have a guide. And if you're going to fail, the guide won't be much help.

Conclusion

I think the RH300 course and RHCE certification can be valuable to both employers and individuals. The course provides a nice quick overview of the basics needed to move, for example, from being a systems operator to being an admin. I wouldn't think of it as either a requirement for the test or as something that can make someone not ready suddenly have the skills required for the exam. Since the exam is hands-on and lab based, those abilities can only come from real world experience. Looking at that from the other direction: this is exactly what makes the RHCE worth anything. While it's not a total statement on someone's talent, being able to pass is a strong indicator that they have the basic skills for a systems administration job. If I were making hiring decisions, I wouldn't make the RHCE a requirement, but I would have more confidence in applicants who have it.

nicktook writes: "ZDNet has this story that RedHat has released a full alpha distribution for the 64-bit Itanium processor. Can Itanium hardware even be bought yet?" Not by jes folks, that's for sure. cnoe also sent a link to the official announcement from Red Hat as well. Coupled with SGI's release of Pro64 compilers for Itanium, it seems like Intel's Next Big Chip keeps whispering "penguins." Stunning news from MS on this front is long overdue.

nicktook writes: "ZDNet has this story that RedHat has released a full alpha distribution for the 64-bit Itanium processor. Can Itanium hardware even be bought yet?" Not by jes folks, that's for sure. cnoe also sent a link to the official announcement from Red Hat as well. Coupled with SGI's release of Pro64 compilers for Itanium, it seems like Intel's Next Big Chip keeps whispering "penguins." Stunning news from MS on this front is long overdue.

Jon347 writes: "KDE 2.0 beta has been released, and looking very slick, regular files are on kde.org's servers and rpms on people.redhat.com. This one has been delayed for a while but looks worth it. This is the first of three beta releases. "

buzzcutbuddha writes "Bob Young from Red Hat issued his rebuttal to John Taschek's Anti-Open Source Article on ZDNet. Well written and articulate, and to the point ... He shoots, he scores!" Check out the original article blasting the open-source idea. Good rebuttal, Bob.

As this release explains, BSDI are about to start providing support contracts for FreeBSD. Support options will include options from per-incident to 24x7 support. Linux already has a number of high profile companies providing support contracts, such as Red Hat, and IBM. It'll be interesting to see how much of an issue the availability of BSD support turns out to be.

patrixmyth writes "A default password of "Q" in the standard Red Hat 6.2 installation of the Piranha module opens a Web server to intrusion, according to Internet Security Systems. The problem was discovered during a review of Open Source code, and the fix is already available. Another victory for Open Source! The MSNBC article is here. The fix is here, or you could just reset the password yourself for the Piranha module."

patrixmyth writes "A default password of "Q" in the standard Red Hat 6.2 installation of the Piranha module opens a Web server to intrusion, according to Internet Security Systems. The problem was discovered during a review of Open Source code, and the fix is already available. Another victory for Open Source! The MSNBC article is here. The fix is here, or you could just reset the password yourself for the Piranha module."

bero-rh writes: "We have just launched the UFO (Unmaintained Free software Open source) project -- an attempt to fix one of the very few problems with open source: People who used to maintain a package and can't do it anymore can leave it to the project where at least basic maintenance work is done. We'll also try to find a new real maintainer for the packages."

Brian Ginter writes "I noticed on www.redhat.com today that 6.2 is officially released. It'll be shipping April 10. "

EricWright writes "Although this hasn't (apparently) hit major news outlets yet, the Raleigh News & Observer is running a story in the business section today that Red Hat has dropped plans to hold the Linux Expo 2000. Although I haven't seen anything at Red Hat's website, they have confirmed the cancellation, saying 'The Expo won't happen this year. This doesn't mean that the project is abandoned; it just means that it won't happen this year. We are very much interested in keeping it alive.'"