Slashdot Mirror

Linux has corporate muscle. No, really -- you just don't see it.

While Linus decides what really goes in, a great many of the coders writing patches do it for pay. My employer, MontaVista Software, employs quite a few kernel hackers (particularly linux-ppc folks). While the specific things they work on are dictated by our customer's and clients' needs, it all (or the best of it, rather) goes back in the tree. We have customers. Big ones. Lots of them. Can't say more -- I'm NDA'd -- but trust me, there's profit to be made working on Linux.

Filesystem development is being corporately sponsored -- see Red Hat, Namesys and SGI for examples. MontaVista has been sponsoring work on making the kernel fully preemptible for some time, and we do a great deal of internal QAing (and fixes, the latter being submitted upstream to the "real" kernels) on otherwise rare platforms. There are a great many "big names" putting programmers into this thing, because it's cheaper to hire folks to improve Linux to do what you need than to write your own OS from the ground up (which a great many places building embedded systems used to do), and far, far easier than trying to get your closed-source OS vendor to make the changes you need for you.

The open source model is unique in that it has a wide variety of different agents each with their own private agenda. Rather than one entity trying to build the "perfect OS", each individual group focuses on the thing that's important to them. When the users and the developers are one and the same, it really does improve quality -- I've written apps for myself and apps for other people, and I know which ones work better.

As for the "limited scope" thing... what limited scope? What does Windows try to do that open source software does not?

In many respects, Linux has a much wider scope than Windows. Consider: the same Linux kernel and userspace works not only on x86 systems but also MIPS, PowerPC, PA-RISC, SuperHitachi, StrongARM, Alpha, the S/390, Sparc and elsewhere. While Windows has separate revisions for embedded space and high-end servers, the same Linux kernel is expected to work everywhere -- and does so.

I was much happier at university. It's what I've been doing since in the "real world" that seems like busywork.

Okay, so your job sucks. :p

Actually, I suspect that this is far from unique -- of the six or so jobs I've held (counting the consulting gigs), only one is really better than academia. However, it's really, way, way, insanely better -- I'm working with the most talented people I've ever met (not an exageration) on really sweet hardware (also true) and getting paid to implement software I both conceive of and design (yup, it's so!).

Nothing in the academic world was ever this fun, even the research projects (at least, the ones I was involved in; they were great in theory and I learned a great deal designing them, but not one ever picked up any users -- and part of my thrill of writing software is seeing it making people's lives better through use). Further, while my professors were roughly my equals (some a bit better, several much worse), my coworkers here are damn near godlike (and I hold this opinion after working with them for two and a half years).

You're right, though -- I haven't graduated, and would need three or so semesters of classes (almost all general ed, as I've finished the fun classes already) to do so. As for my school, Chico State, its CSCI program is neck and neck with that of Cal Poly San Luis -- but as good as it may be, it just doesn't touch the Real Thing.

You'll note that it is touted for use in "ATMs and slot machines". That is because it cannot be used for hard realtime, low latency applications like flight controls, robotics, or medical devices. It is also sure to be bloated and inefficient compared with something that is designed from the ground up to be an embedded OS.

I'd highly recommend QNX instead, it is POSIX and QNX sits on the realtime Linux committee. Hard Hat Linux and cousins are looking better and better as well. These solutions do support hard realtime scheduling, thus providing across the board solutions for all your embedded needs. QNX, in particular, is also very well engineered. It provides a highly modular architecture, allowing you to deploy only the functionality you need, minimizing system cost.

299,792,458 m/s...not just a good idea, its the law!

I work for MontaVista Software, an open source company. I'm not sure I'm free to discuss our financial situation, but suffice to say we're going through this "slump" just fine.

Really? It looks to me like your growth has stopped -- there's only one job opening for the whole company. Your end of 2000 announcement discussed ten-time revenue growth (from what? it's your first full year), and growth from twelve people to 160, but it was conspicuously quiet on the question of profit. Meanwhile, at that resource load, you're probably burning through the $30 million in investment fairly quickly, though by my back of the envelope it shouldn't be all gone yet.

Don't get me wrong -- you may have good prospects in the future, but it does not appear to me that you are now a profitable company, and the dearth of new job openings is a disturbing bellwether for future growth potential. If you're doing "fine" now, that may only mean that you haven't yet spent all your investor capital.

Tim

Your company sucks if it's somewhere like that.

No, seriously.

Where I work, we (the engineering staff) don't want, need or worry about promotions, and the folks who are so unfortunate as to receive them get teased mercilessly. Why? We're good engineers. We all know we're good engineers. If we wanted to be managers, we'd be managers already. We don't resent our coworkers, because they're good engineers too. The managers might step on each other, but that's their own damn business.

(For that matter, several of us *do* get together out of work, watch movies, drink, play in a band, &c).

Pick a small(ish) company with a good work environment and ; it's well worth it. After being here for a few years, I personally just flat out won't work (long-term) anywhere less laid back.

(Oh, did I mention we're an open source company, too? This contributes to the atmosphere -- it means that many of us, while we don't mind money, don't have it as our sole reason of working).

One can, under the GPL, provide binary software and source only on request, and one can charge for that initial binary.

My company does just that with the "professional edition" of our linux-based embedded systems development environment, though it's based on Free software.

In some companies, unless you were told to write that automation software, you could be reprimanded, maybe even fired, for writing that. That's not to say that it isn't a good idea. I'm just saying that way too often, company management, bureaucracy, and politics have the effect of making an otherwise good job become a lousy one.

Yeah, I can see that happening -- I've friends who work at such places (HP, in particular).

I, however, don't work at places like that -- a policy which, IMHO, vastly improves my quality of life.

My present employer is relatively small but has highly clueful management, a fantastic engineering department (never in my life have I had such skilled coworkers) and is doing (very) suprisingly well in this whole downturn thingy. I'd like to think that there's a correlation between all three.

Every programmer, sys-admin, developer, designer, de-bugger, support-desk jockey, etc. that I know DESPISES their job and can't wait to get home.

Uhh... ya obviously don't know me. :)

Seriously, though, all the best tech folks I know love their work for a passion (and, being employed by MontaVista Software, I work with a great many of them). I (and probably most of the folks who are good at what they do) won't work somewhere I'm not happy, even if the money's better.

If you think coding is drudge work, then you Just Don't Get It; the joy of creation is lost on you. So be it -- some people don't have it -- but don't deny that some of us do. Hell, most free software wouldn't exist if creating it weren't fun.

I read User Friendly for the same reason I read Dilbert -- it's funny, and I can laugh about people whose jobs suck compared to mine (though we may both work in the same industry).

Basically, why is this not the default behavior?

Well, the traditional view on this is that reducing latency by whatever means tends to lower overall data throughput, which is just what you don't want for a server OS, which is still what accounts for most Linux installations.

However, it may be that the traditional view is wrong. It may well be that the increased usage efficiency that comes with kernel pre-emption may actually increase throughput - high-priority disk I/O for instance now never has to sit waiting for the CPU to complete a syscall. There were some interesting results posted linux-kernel regarding this, see here.

The linux scheduler ensures that no process is ever completely starved of CPU time, so no huge backlog of syscalls ever builds up.

The other reason that it's not the default behaviour is that it's an interesting and new approach to how to achieve a pre-emptible kernel. All other pre-emptible kernels have been designed as such from the ground up - Linux certainly hasn't. There are a couple of white papers, here and here from MontaVista (who kickstarted the pre-emptible kernel project) about the approach taken. They also detail a few other approaches to making Linux more responsive for real-time and interactive tasks.

Basically, why is this not the default behavior?

Well, the traditional view on this is that reducing latency by whatever means tends to lower overall data throughput, which is just what you don't want for a server OS, which is still what accounts for most Linux installations.

However, it may be that the traditional view is wrong. It may well be that the increased usage efficiency that comes with kernel pre-emption may actually increase throughput - high-priority disk I/O for instance now never has to sit waiting for the CPU to complete a syscall. There were some interesting results posted linux-kernel regarding this, see here.

The linux scheduler ensures that no process is ever completely starved of CPU time, so no huge backlog of syscalls ever builds up.

The other reason that it's not the default behaviour is that it's an interesting and new approach to how to achieve a pre-emptible kernel. All other pre-emptible kernels have been designed as such from the ground up - Linux certainly hasn't. There are a couple of white papers, here and here from MontaVista (who kickstarted the pre-emptible kernel project) about the approach taken. They also detail a few other approaches to making Linux more responsive for real-time and interactive tasks.

There is a lot of good stuff here. Some of the more useful and general-purpose patches -- such as VLAN, TUX, and Software Suspend -- should get a chance to become mainstream. The various IPC and speed improvements should make it in, too.

There's currently a debate over which real-time scheduler is the best. Personally, I'd like to see it resolved in the same way as the other options with choices: let all of them be integrated into the mainstream, and let the user select which one to use, either at compile or boot time! I'd like to see an option in the kernel configuration, asking what real-time scheduler you wanted: MontaVista, RTLinux, RTSched, Linux-SRT, RTAI, DWCS, something else, or simply the default.

Linux needs a real-time scheduler today. Currently, things become choppy whenever it decides to service the system in some way, such as syncing the disk. Playing movies, audio/video recording, burning CD's, even playing games would benefit from real-time support. I hope that this can become mainstream in 2.6!

Super eurobeat from Avex and Konami unite in your DANCE!

It doesn't make sense to make a "software company" and think that you're going to profit creating software and releasing it under the GPL. The reality is that the overwhelming majority of GPL fans are the leachers of society who between Napster sessions are busy ripping off cable TV and stealing the neighbours newspaper (or living in university labs).

This is not necessarily so in a case where a piece of software is entirely new; however, in a saturated market, use of an open license (providing customers with flexability between support vendors and an assurance that some level of support will be available even if the vendor goes out of business) is a big win. I say this not theoretically, but as a programmer at a company using just such a strategy. Here at MontaVista Software we support the use of Linux in building embedded systems. We have a great deal of internal build, testing and development framework which is very hard to replicate otherwise, and a high saturation of talented developers. As such, we make good money off companies looking for an operating system suitable for embedded use which they can use royalty-free and get excellent support for.

We don't just profit off those other developers who wrote the software we support, either. We fix bugs, add new board support and contribute new features quite regularly -- and our customers pay us to do it.

In our market, use of the GPL is really win-win. Our customers get an excellent embedded OS, and the ability to switch vendors if we disappoint them (something which, to the best of my knowledge, hasn't happened yet); the community which wrote the code we support gets bug fixes and new features; and we get paid. What's not to like?

I work for an embedded Linux company. It pays the bills quite nicely, and I get to write (and port, and bugfix) free software all day. WooHOO!

The 40us is nonsense: depending on the motherboard/processor it can go from 1 to 15. As for the submicrosecond claim
40us is correct for Montavista. I searched for RTLinux numbers but was unable to find any detailed report(surprise, surprise). I generally saw numbers ranging from 15-30us. Some QNX numbers are here. I understand VxWorks is a little better

I would guess the same is true of the realtime Linuxe" --- You guessed wrong.
Maybe you should read the analysis section above where it talks about the number of processes on the system affecting interrupt latency! Again, dumbass. I only wish RTLinux had some decent documentation so i could prove you wrong about them too.

being naive is no crime, I guess
but being an arrogant dumbass should be!

Why do I say this? Well it should be obvious now. RTLinux is not a Linux distribution, but rather a realtime executive that can ran a "Linux image". Similar approaches have been taken by several groups, like Radisys and Nematron, to make Windoze a "realtime OS". The results are generally all the same. Because of the special tricks that must be used, you end up with an OS that is less stable than the off-the-shelf product. And you really do not get many of the benefits of using an off-the-shelf OS, because anything you do that needs to be realtime has to be run by and programmed for the "realtime kernel" (not the OS kernel!)and its proprietary API.

As far as realtime performance is concerned, the last numbers I heard at the 2000 ISA show showed that RTLinux (actually it may have been Montavista) was well behind the major players (i.e. QNX, VXWorks) in terms of realtime performance. Worst case interrupt latencies were on the order of 40 microseconds, compared to sub-microsecond latencies for others. This is only to be expected with the overhead of running two OS's. In all fairness, it did beat Windows CE. :-) An interesting thing to note if you read about Nematron's HyperKernel (above) is that realtime latencies actually get worse when the Windows NT side is heavily utilized! I would guess the same is true of the realtime Linuxes. So don't play Quake or your reactor may meltdown... ;-)

you forgot OSE by Enea Systems. Since you forgot a vendor too, either:

a) You're as ignorant as he is, or:
b) He's not ignorant.

My hope is B. That said, the failure of poorly performing embedded Linux companies is just selective market forces in action. Windows resellers are going out of business too, does that mean Windows is doomed? What about the solid embedded Linux vendors who are making money in this space?

COUtrollGH COUGH

I got a chance to see hardhat linux at work. Was actually a pretty sweet little deal for embedded systems. Easy to setup and configure. If I actually did any serious embedded work I'd at least consider it.

Embedded applications have an extra requirement called "Real-Time". This means that many tasks must execute within a certain number of ms. Many commercial operating systems such as VxWorks have made a living out of providing just that service.

Linux, of course, does not have a real-time scheduler. But many linux vendors such as Mvista are providing linux distributions with a real-time kernel module to take care of those tasks with a real-time requirement, and allowing the standard linux scheduler to take care of the rest.

This provides the following advantages to embedded developers over commercial RTOS's:

1) It's WAY cheaper
2) The support for linux is actually much better, due to the open source nature
3) Closed source RTOS hardware integration bugs are nearly impossible to debug, because the source is CLOSED. You have to wait for a Field Application Engineer to get flow in, on your dollar, to write your stupid driver.

Linux is creating real fear in the embedded RTOS market space.

For true real-time, it's not as stable. But for complex network applications with a lower real-time requirement, it's a killer.

http://www.mvista.com

Yup, we used the Intel IQ80310.

The IQ80310 can be used as a PCI host with a passive PCI backplane, but it can also be used as a normal PCI card in a PC (or whatever else with a PCI bus).

Because we didn't have a passive PCI backplane in New York, MontaVista bought the cheapest PC available at a local computer store and we used that as a backplane. I have no idea what kind of PC it was, we just used it as backplane and power supply to get the XScale running. The PC was definitively worth its price :-)


Erik

No, that's not what it means. Nico has been working on this for quite some time (sponsored by MontaVista), but he still maintains the SA1100 Linux tree. I am still working on the LART.

Nico and I have been sharing a hotel room during the Linux World Expo in New York, and we wanted to have the XScale shown running on the expo. So we hacked on thursday night and showed the result on friday.

About the UCB1200 driver: there is a unified UCB1200 driver in Nico's latest SA1100 patch. I promised to review it, but still got no time to do it. I'll do it Real Soon Now [tm].


Erik

Because it makes a difference to developers, who want to leverage experience we already have. Further, all OSen are not created equal...how is Palm's sound and sound capture support for instance?

Where do you think software features and hardware support come from?!? The OS is an integral part of the mix.

Palm has the most handheld applications today. That says nothing about tomorrow. Some reasons developers might want to use Linux on a PDA:

I hope that helped! :-)

(BTW, is there a way to moderate the "Insightful" tag back off the parent post? ;-)

One last thought - I bet Linux is in a much better position to support the next PDA superCPU...it's called portability you know... :-)

They're using Hard Hat Linux under the hood.

http://www.kerbango.com/press/R_monta vista.html

Somebody moderate this rant up! I've been working with pSOS, VxWorks, etc. for some years now, and I can agree with what you say wholeheartedly.

One small item:

But if my target is some weird iron featuring the latest funky embedded microprossesor from motorola, there is zero chance of linux being ported to my target by somebody else, least of all motorola themselves.

This seems to be the whole point of MontaVista -- they take that "weird iron" and get Linux running on it. So far, they seem to have only tackled Pentium CompactPCI systems, but I expect that given time and incentive ($$$) they will branch out. Certainly Force isn't an Intel-only shop.

And if Mot were smart, they'd help push Linux onto their funky embedded chips, especially the 68360 and the PowerPC 860 series. But I've never been able to figure out Mot's OS "strategy" ... they seem to just want to make cool hardware and not to have to worry about what's running on top of it.