Slashdot Mirror
HURD For 'Big Iron'?
Julian Stoev wrote in with this query: "Recently I've seen quite a lot of conversation about Linux on 'big iron' and talk about a possible fork due to the fact that the Powers That Be do not want to include the necessary features that are necessary. A guy from IBM in an interview sounds a little desperate. He sounds like IBM is not very
happy with Linux's direction to small devices. The guy is cautious not to make kernel people angry and does not speak directly about kernel fork. But why don't they (IBM, SGI, et al) grab HURD and add to it all the things they find important for 'big iron' support?"
"HURD is not good for smaller machines because it needs more memory, but for big machines/clusters it can be very promising design. Why don't these vendors who moan about the lack of 'big iron' features start a 'HURD foundation' to help uplift the sleepy project? They could add compatibility for Linux binaries for compatibility, even which would be great as there would then be a powerful kernel (HURD) for 'big iron' and Linux for smaller and embedded systems.
Yes, HURD needs quite a lot of work to become stable, but large vendors have the resources to give it a push. In this way they will prove that they really appreciate open source by helping a project which is going through some hard times right now."
Some competition would be a good thing; and since all the code is GPL, they can re-merge later. Linux has gone through fork-and-merge several times already. A lot of developers keep their own little fork going while in development. XFS, for instance, ships as a patched kernel.
It would be good for SGI, IBM, HP, and other big players to create an Advanced Linux Kernel Project. I would even host it (I'm a director at a hosting company) and contribute code (filesystem, device drivers, unicode functionality).
Even if the Linus-headed effort is the One True Way, it doesn't have to be the Only Way. They're not infallible. Let's get a second project going! It won't be a threat to Official Linux -- just a development track for enterprise situations!
Email me if you're interested!
________________________________________
Napster-to-go says "Fill and refill your compatible MP3 player", which is a lie. It's not MP3. It's WMA with DRM.
- massive momentum
- probable emerging standard
- huge (unpaid by them) developer community
- existing high-quality implementation
- existing widespread hardware support
- easy to find techies who know it
- non-techies are comfortable with it
- liberal licensing
Now which of these does HURD offer?The question should be: Now which of these does linux offer?
Are you really telling me that linux offers *any* of:
- probable emerging standard (don't make me laugh - linux is anarchy - BSD is more standard since it is committe based)
- existing high-quality implementation (have you ever looked at linux code? And it changes every two minutes)
- existing widespread hardware support (DVD? Hardware RAID even?
....)
- non-techies are comfortable with it (please
...) - Liberal licencing (GPL!!!)
And as for stuff like "massive momentum", you must remember that the momentum behind linux is its use as an application not as an OS. Linux boys use linux as their primary app, not as an OS to run another app. Their app *IS* linux. It *IS* spending the whole evening downloading the latest Hungarian font patches and recompiling the kernel just so their system is "complete". Ten minutes later they're downloading the next patch that is issued, for hardware they don't have"Making linux GPL was the best thing I ever did" - Torvalds. I'd hate to see the worst thing...
Also, I would guess that the boys at IBM know about the '#ifdef IBM' statement.
"Making linux GPL was the best thing I ever did" - Torvalds. I'd hate to see the worst thing...
I work on the linux for s390 project & have worked with the mach microkernel but not with hurd. The mach micro kernel is a fantastic peice of code with fantastic ideas let down by the rpc abilities of C. The guys who developed the mach microkernel I think realised the limitations of C & sent some guy into the bushes for a few years & he invented objective C, this unfortunately runs like a dog with all the message passing & is why Open Step is so slow, I even considered writing my own language ( which would probably have taken me decades so I gave up on the brainfart ) which could while running bind as to local objects while sending messages to remote object rather than using messages everywhere. I even suggested the mach microkernel when I first joined in the early days of the project & personally am glad I was completely ignored for the following reasons. 1) Linux is pretty much bog standard unix no learing curves for most developers, the mach microkernel is a complex poorly documented beast with API's documented by doctorates & which is not exactly easy reading. By the time we'd have got as far as we have with this project I'd be about half way through reading about & fully getting to grips with Mach. 2) If we started on Hurd I'd be dead by the time we got it running as good as we have Linux running now, my life as most peoples is too short. ( we initially had only 5 developers on a skunkworks team ) & the Hurd project most likely will never grab the mindshare or momentum of Linux, Linus is a great marketer & great at rallying new support & it is getting better every day rather than every decade ( IBM couldn't buy this stuff, it didn't with OS/2 ). 3) The linux project is very well supported by documentation & web sites it isn't like you need to know someone or be on an inside track to become expert in it just web access . 4) There are a few places where the the basic Mach kernel is weak for instance no support for drivers as modules, lack of driver support, messaging is pretty slow & used for everything, great for building clusters but overkill otherwise. As mach uses seperate address spaces for everything this improves protection but decreases performance. 5) Getting new code ( e.g. Firewire or s390 support ) is pretty easy to get into the standard Linux kernel, admittedly they are pigs for accepting replacment code for stuff which works in most cases, even if it improves things. Plenty of IPV4, scheduler & filesystem improvments are posted regularly but are seldom accepted ( I'd suspect Hans Reiser has a lot of grey hairs at this stage ). From what I hear the a lot of developers left the BSD project because their patches simply weren't being accepted by the chief maintainers who liked their names over 98% of the source code.
WHAT?????
The whole _POINT_ of Crusoe is that you _DON'T_ run native code. EVER. That ruins the whole beauty of the architecture. That would be like taking the eiffel tower and removing two of the legs...
The whole point of the code morphing stuff is that they can change the native instruction set when ever they want w/o having to worry about it affecting in production proggy's/OS's. Thus they can move to the latest, greatest ideas in microchip design w/o losing customers. This is clearly stated on their website and in numerous articles about transmeta.
Linus isn't working on this because the idea goes counter to everything Transmeta is working on.
According to one article I read... maybe it was on linuxtoday.com.... I disremember. These patches are not being accepted simply because the memory management patches to support these large machines doesn't scale back down to desktops, laptops or handhelds. In other words, making the ness. changes in the kernel to roll this stuff in would affect performance on your server, desktop, or PDA. Considering that 99%+ of Linux users are running on these platforms it doesn't make sense to apply these patches.
---
RobK
Myddrin
As it happens, I got Debian GNU/Hurd up and running just yesterday. It took about two hours, counting the time it took to install Debian GNU/Linux and the time it took to download the installation .DEB files over a 115K ISDN line.
Of course, I can't actually do anything with it because the kernel is only about 1/3 done and there are essentially no applications available for it. Perhaps if the FSF decided to put their efforts into the Hurd instead of yet another major version of GCC, GNU EMACS, and the GNU LIBC, they would be able to actually finish it to the point where Linux was at kernel V0.10, the version I first used. Some additional stability would be nice, too.
I think I'll reformat the disk and see if I can't get OpenBSD on it.
AC also wrote:
Ummm, that's not factually accurate. I don't know if the code for the Hurd (the name for the collection of services running under Mach) antedates the arrival of the Linux kernel or not, but the project as a whole has been going on since the early 80's and the overall design of the system has been frozen for at least that long. In other words, the Hurd was not created in response to the success of Linux. It couldn't possibly have been.
However, I do share your doubts as to whether or not they're ever going to finish. They definitely bit off more than they could chew with the Hurd.
Here's a short answer to why IBM doesn't fork the Kernel.
1) Public Opinion / Community perspective
Linux is built on popularity. It gets marketing's attention because it's a popular choice among Admins. Therefore, IBM has to jump on the bandwagon and support the thing in its many incarnations. IF IBM FORKED the Kernel, it would be a marketing nightmare as Slashdot and others would do a 180 from "IBM is doing Linux solely for PR" (untrue) to "IBM wants to take our baby and twist it for its own evil purposes!" (also untrue.)
Don't believe Slashdot would be this unkind? Just look in the archives for RedHat 7.0 and see how supportive the community was at that time.
2) IBM has a huge investment in Linux.
IBM has a Linux Dev Center newly established in India. IBM has a Linux Compatibility Org to test and ensure that every IBM application for Linux will work with the 8 standard NLS Languages and doesn't break with standard libraries, and will function on most all recent distibutions.
IBM has invested huge amounts of resources in the IBM Journaled File System for Linux. IBM took part in developing Linux for the S/390. It also runs on the AS/400, and there's work on the RS/6000 distribution as well.
The investment is too great to move to HURD.
The HURD community and installed user-base is tiny. Linux has name recognition.
Currently, most new IBM Web Application Server / WebSphere type products support NT/2000/AIX/Solaris/Linux.
Moving to BSD would be smarter, but BSD hasn't got name recognition, even if it is more widely installed as a serverOS.
Moving to HURD would be suicidal.
Yes, there's AIX for Big Iron, but AIX is a commercial server OS and doesn't get the community's collective hearts pumping.
There's mindshare, which is what we logically know. AIX exists for big iron, why do we need a linux for it?
and then there's *HEARTSHARE* which is the emotional based decision. We know in our hearts nothing makes us growl with manly pride at having Linux, the Free-Little-Operating-System-That-Could-TM, running on the biggest Iron made.
Heartshare will keep IBM focused on Linux as long as there's a community of people running Linux.
AFAIK, Debian is the only big-name that's bothered to propose a HURD distro. http://www.debian.org/ports/hurd/
I'll believe it when I can install it.
(Tho, it uses Mach and linux hung around it... if it was Mach and BSD, it'd be a relative of Darwin-x86! hmmmm....)
In Summary
IBM has 20 top-level links to Linux related dev sites on its Intranet. These snowball into deeper levels. IBM has a huge investment in Linux, and in having it run on their boxen, big iron or not.
And you want them to drop it all and move to HURD?
A host is a host from coast to coast, but no one uses a host that's close
Linux grew from humble beginnings, small i386 machines with little memory and scant resources. unfortunately, it's kept a lot of baggage from those days even though it's come a long way and been ported to many architectures.
That's simply not true. Take a look at the latest kernel. It can support up to 64 Gigabytes of memory, and it can scale quite well up to 4 and 8 way SMP boxes. People have booted Linux on 32-way ccNUMA machines. Yes, it's not optimized for ccNUMA yet, and it probably doesn't scale all that well for > 8 way SMP yet, at least for many workloads, but it's currently quite a very long way from "small i386 machines". A "small i386 machine" is orders of magnitude than the original i386 back in 1991, and a "small i386 machine" today is comporable with the vast quantity of Unix machines which are used as servers today.A lot of the "we scale to 64 nodes" is I think more machismo more than anything else. Sure, for system vendors they have better margins than the smaller machines. But you don't sell that many of them. One of the reasons why Cray was getting killed was that they only focused on the high-end, and they got their lunch eaten by the "low-end" machines moving upwards and killing off all of their market, so that they only had one or two customers. (Or should I say "One Agency". :-)
So personally, yes, people are interested in making Linux scale to the bigger machines, but some of this I susect is either (a) because of the technical challenge (for the Linux Kernel developers) or (b) as a marketing exercise (for the non-technical folks who are interested.)
There are many things in the kernel which do things the x86 way and force the other architectures to munge the way the native system does things so that they look like the x86 way. When I last looked at the SPARC port, the memory management system had to jump through hoops to change the way the SPARC processors do VM so that it looked to the rest of the kernel like the way the x86 architecture does it.. it was very inefficient. No doubt these problems haunt the other architectures too.
That's simply not true. There is an awful lot of that "complexity" which is optimized out by the compiler. So you should take a look at the assembly language before you make these sorts of complaints. Secondly, on the UltraSparc particularly, all of the virtual memory translations have to be done in software, since the hardware basically only provides a TLB which is manually programmed by the OS. This may be the source of some of the complexity which you saw, and which is required by all operating systems for that platform. This is actually a good thing, though, since the OS can do a much better job of managing the TLB than a typical hardware platform, and there are hooks in Linux that are especially designed for the capabilities of the UltraSparc VM architecture. So to say that Linux is only optimized for the x86 architecture is very much overstating the case.
One of the problems is that for many mainframe systems, massively parallel processors are a common feature. Linux's SMP support isn't really that great. There are some massive improvements in the 2.4 kernel, but some fundamental design decisions get in the way.
For example, Linus is a big proponent of a non-preemptable kernel. A preemptable kernel is one that can allow tasks within the kernel to be preempted by other tasks. The Linux kernel does not allow tasks to be preempted by anything other than interrupts.
As an aside, interrupts in the Linux kernel typically run a small bit of code to set up some state information about the interrupt, flag a "bottom half" to run later, and then get the hell out of dodge. The bulk of the work is in the "bottom half," which is a bit of code which is enqueued to run when the kernel gets the time to get around to it. For example, the timer interrupt increments the jiffies (an internal measure of sub-second time), lost timer ticks (ticks that haven't been handled by the bottom half), and flags the bottom half for later execution. The bottom half will later update time-related statistics and service kernel timers.
Basically, though, once the "top half" of an interrupt is handled, the kernel must immediately go back to what it was doing. It is guaranteed that kernel code will not be preempted. This makes the kernel cleaner, easier to understand and maintain, and faster on uniprocessor systems or systems with few processors. However, there's a reason that the Solaris kernel is preemptable. Preemptability is a serious performance enhancer for massively parallel systems, but Linus is not budging on this. There are a number of alternative solutions that the Linux kernel is following, such as its system of "bottom halves," some of which can now be run on other processors in the 2.4 kernel (if I understand correctly). Essentially, a preemptable kernel requires that fine-grained kernel locks be scattered all over the kernel, and that more code in the kernel be considered "critical sections." This is somewhat of a hassle.
Solaris scales extremely well partially due to its kernel's preemptability and heavy use of threads. Solaris's kernel can be preempted by another task and can dispatch kernel threads to work on other processors. This keeps system call hungry processes on other processors happy and well fed. However, it does incur some overhead which isn't justifiable on desktop machines. I'm reminded of a funny quote from the 2.2 kernel mailing list I found once while looking for info on this once: "MVS spends a lot of time running OS algorithms to allow full preemption that Linux wastes on running user applications."
Basically, this is just one of a variety of wide-sweeping changes to the kernel that go many levels beyond a simple patch that "Big Iron" vendors are pushing for. Linus is against it happening in the first place. While Linus and Co. are coming up with a lot of innovative alternative solutions to the problems, "Big Iron" wants to go with proven solutions that they know will work for their systems. In addition, the changes would impact performance on desktop uniprocessor and SMP machines. In essence, these are political issues.
RTLinux has already forked the code to handle this. Basically, RTLinux runs a hard-RT capable microkernel that runs the Linux kernel as its idle process. Basically, when a RT process needs something at exactly a certain time, it will get it and all normal processes will be run under Linux when the system isn't busy doing more important things. There's also the uLinux fork to attempt to make something Linux-like on systems without hardware memory-management units, like the 286. Code forks already exist, often for good reasons.
There's a great article kernel design and impact on real-time systems here:
TradeSpeak.com - White Paper Library: Linux for Real-Time Systems: Strategies
If it's for-profit but free, you're not the customer -- you're the product (e.g., the Slashdot Beta's "audience").
"What about the Alan Cox series?"
Ok. Apart from the Alan Cox series, Linux has never had a Code Fork in all of it's long existance.
"There are the L4Linux, L4KALinux and MkLinux microkernels."
Ok, ok! Apart from the L4Linux, L4KALinux and MkLinux microkernels, and the Alan Cox series, Linux has never had a code fork in all if it's long existance.
"There are patches from SGI (XFS, kernel profiling), IBM (JFS), and various other developers (timing patches, nanosecond clock patches, reiserfs, ext3fs, gfs, the international patches, freeswan, etc)"
OK!! So, apart from the patches, the microkernels, the Alan Cox series, Linux has never had a code fork in all of it's long existance.
"What about the UserLand Kernel?"
SHUT UP!
(Apologies to the Monty Python crew for this horrible bastardization of their excellent Roman sketch in Life of Brian)
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
The HURD is based upon the MicroKernel technology which was in vogue in the Computer Science community about 10 years ago. Time has moved on, people have taken the ideas from this technology and incorporated the best bits into the semi-monolithic kernels of today, Microkernels are looked upon as a relic.
MicroKernels are theoretically so much cleaner and efficient. They are built on the idea that every part of the system has its own thread of execution and that the "kernel" contains nothing more than a facility to pass messages between the threads. In some extremes of the idea even the scheduler is just another thread. This design means that everything is compartmentalised, clean, organised.
The problem with this approach is that in the real worl this deisng has a massive performance hit. Every thread context switch, every message passed needs CPU and/or memory overhead. If a processor was designed with internal massively parallel instruction streams and internal message passing this wouldn't be a problem.. with current processors it is.
What's wrong with Linux
Linux grew from humble beginnings, small i386 machines with little memory and scant resources. unfortunately, it's kept a lot of baggage from those days even though it's come a long way and been ported to many architectures.
There are many things in the kernel which do things the x86 way and force the other architectures to munge the way the native system does things so that they look like the x86 way. When I last looked at the SPARC port, the memory management system had to jump through hoops to change the way the SPARC processors do VM so that it looked to the rest of the kernel like the way the x86 architecture does it.. it was very inefficient. No doubt these problems haunt the other architectures too.
There is also the problem of the "one size fits all" mentality.. it doesn't work!
Sure, you can have a family of kernels, each aimed at a different niche with what common code they can share shared, but don't try to force the same kernel onto everything otherwise everyone will loose out.
Before anyone tries to label me as a supporter of some other system saying that I'm only bashing their favourite because I have an axe to grind, I don't see any of the free or commercial operating systems today being able to be all things to all men.. and I don't see any being so in the future either.
Agrajag: "Oh no, not again!"
You have to admit RMS is rather wacky though. I subscribe to NT Emacs mailing list. Not so long ago there was a discussion about the new version of the NT Emacs FAQ. It seems that RMS wanted the section about using Emacs with Opera pulled. Why? Because Opera isn't free. Fortunately there weren't too many upset people as demand for the information is low, and it can induced by by reading the work-arounds for other products. Most of the information in that FAQ concerns using Emacs with non-free products. In fact, it's all about using Emacs under *NT*, so perhaps there shouldn't be an FAQ at all. RMS is cracked.
imagine big vendors trying to get along with RMS...
Vendor: "Hey Ricky darling! We're writing a killer App which will make everyone want to use Hurd.
RMS: "Are you going to let everyone copy it for free?"
Vendor: "No, but we're making the file format public domain. We're also explaining how the algorithm works in detail so that the GNU can make a free version if they want. We're supplying the source to anyone who wants to write a patch for it, or reverse engineer it. We just aren't allowing people to pirate copies of it."
RMS: "You're all evil scum. You have no right to make a profit from your work by restricting others from making a profit"
Vendor: "But they can make a profit. They just have to write their own version"
RMS: "You should give it away for free"
Vendor: "But we spent time and money developing this. We deserve a profit"
RMS: "You don't deserve anything"
Vendor: "Oh, sod this. We've changed our mind. We're going to make it secret closed source, with a restrictive license. Anyone who tries to compete we'll sue into oblivion"
I think the real question is whether one Linux Kernel can - or even, should, fit all systems. Obviously the current scalability and flexibility of Linux to run on an enviably diverse range of platforms is impressive - but will it ever be the best OS for everyone?
I guess you can please some of the people all of the time and all of the people some of the time, but no more - and frankly this makes sense.
Big Iron is bound to have requirements that differ greatly from handheld computing. I'm pretty sure that this will continue. My fear is that code forks will reduce the impact that the Linux community is having in attracting applications developers. You want to make it as easy as possible for people to write once and run anywhere.
And, fundamentally, isn't that really the point? After all the end user won't know/care whether there's a special patch on the kernel powering their system, as long as it's possible(hopefully easy) to run the apps they like. That's when you get real flexibility and power out of having a common denominator.
I think the stakeholders in this discussion (Linus, IBM etc) should be encouraged not to take their eyes off this goal.
The closest big-iron manufacturer's management gets to appropriate technical decision-making is in counting headlines.
All these say "Linux good". *BSD is *possibly* just about on the bottom of their radar-screens, but probably not. You can guarantee that no-one in that chain has even heard of the HURD.
Sadly, this applies to the techies too, although to a lesser extent. Never underestimate the herding mentality, especially within corporate organisational structures.
The other side of this is that the technical decision makers - the architects if you will - have generally come up through the techie trenches but have left all that behind. They won't necessarily be current with the minutia of OS design or implementation, but will have a strong background and understand the basic facts.
Key to this thinking is that industrial-strength OSes require years of development. Linux has had years of development (though not as many as big-iron OSes, by a factor of 3 or more), and has only recently passed this magic threshold where it can be treated seriously because it's been under live-use development for X years.
These are all points in Linux's favour. HURD on the other hand, regardless of any architectural or technical merit, hasn't been discussed, hasn't been *in use* for X years, has a quiet/non-advocating user community (RMS notwithstanding), and is a break with everything these guys know about. HURD is therefore out of the running
Now, if this whole question had done s/HURD/*BSD/ (for some value of net|free|open), then it would be more appropriate. I still think the answer would be the same (rightly or wrongly, Linux is more popular and therefore better).
Never underestimate the impact of inertia...
--
I'd rather have a bottle in front of me than a frontal lobotomy
The Hurd rocks, but it's not "rock solid". And not yet ported to anything else but i386, according to what I've heard. I had troubles installing it; first my ne2000 card gave troubles, then something as yet unidentified went wrong.
Added to this, it's a microkernel system. Big Iron doesn't need microkernels. They're a little bit slower by nature, but you get the advance of a small, configurable "mostly user-space" kernel. That's pretty cool for palmtops and everyday PC's but not Big Iron, I'd say. That's why QNX doesn't run on mainframes.
If Linux can't do the "big iron" thing, I think BSD would be a more serious alternative for now. NetBSD runs on almost every platform, so its kernel should be portable. And I hear rumours that BSD can do some things that Linux can't as of yet. (And vice versa, of course; that's not the issue. The issue is, maybe BSD will just be able to do the trick.)
GNU's Not Unix: this sentence is more important than you may think.
The HURD microkernel can be set up to listen to POSIX if you wish, but it can also listen to other stories. So here's a possibility to move away from ugly ol' POSIX in a decent manner, without having to completely abandon the ship like e.g. BeOS does, providing only POSIX as a portability layer.
The HURD microkernel allows you to mount filesystems in a "new, radical" way which takes away the requirement of seperate "/usr" dirs, etc. (The mounted fs will be mapped on top of the current one, IIRC.) The HURD microkernel allows for user space device drivers and kernel modules, which I think can in the end make the "UNIX experience" a lot less harder.
So I think that when it's finished, the HURD will be there for the common people and will supply more ease of use and programming freedom than Linux or BSD do now, because it's very flexible and can be set up very friendly towards the user (in my imagination, anyway). It will slowly move to liberate us from the less nice UNIX inheritance, as well.
But I simply do not think that Big Iron is a target of the HURD.
But I'm not very afraid about Linux loosing Big Iron. All we need is an open source kernel (isn't Caldera opening UnixWare?) for it. I don't care if it's a fork, has a different name or implementation. That's what standards are good for. All the tools and gears are here already, so it would be "easy" for e.g. Debian to make a Big Iron port, just as they did a Hurd and FreeBSD port once.
It's... It's...
"We can confirm that Debian does *not* ship the version with the trojan horse. Our version predates it." [CA-2002-28]
Now which of these does HURD offer?
The evaluation of an action as 'practical' . . . depends on what it is that one wishes to practice.