Review of Embedded Linux Book

An Anonymous Coward writes "LinuxDevices.com has just published a very detailed review by Jerry Epplin of the new book by Craig Hollabaugh, Embedded Linux -- Hardware, Software, and Interfacing, published by Addison Wesley Professional. Quoting briefly from the review, "A system developer planning to use Linux for an embedded design is faced with a number of decisions, not the least of which is whether to use a packaged commercial Embedded Linux distribution or to devise a homebrew solution from the available free tools and components. The custom approach has much appeal because of its low cost and radical flexibility, allowing one to choose any approach or tool rather than those chosen by the toolkit vendor. But with this flexibility and low cost comes the chaotic documentation typical of Linux. Thus, books like [this one] fill a significant void . . .""

Comments

Interesting but

[RTFA Man]

The only disagreement I have with this is Hollabaugh's philosophy regarding non-x86 architectures. He contends, "This book's examples show that working with the [ARM-based] MediaEngine and the [PowerPC-based] RPX-CLLF was as easy as working with the [x86-based] MZ104."

This assertion is hard to credit -- even with the author's well-written scripts, acquiring and building cross-development GNU toolchains certainly takes more time than simply using those already provided on your desktop computer.

Hollabaugh had to modify the netcat source in order to compile it for ARM, show that even the author himself has had his share of trouble working with non-x86 architectures. The lesson is certainly not that x86 should be the only architecture considered, but it is fair to say that a slight bias toward x86-based devices is prudent when choosing embedded Linux target platforms because of the maturity of Linux on that architecture.

Re:Interesting but

[JKR]

I agree with you 100%. The GNU tool chain is not a no-brainer on non-x86 platforms, presumably because the projects rely on testing feedback to find and fix bugs.

For my first (PPC-based) project where I tried to build a toolchain from scratch I had real problems finding a mutually compatible set of binutils, gcc & glibc that could successfully compile QT Embedded (i.e. C++).

I think this presents a real problem for business. The source code is freely available, but some feature or other doesn't work on your chosen platform without extra patches (gcc in particular, but also glibc). The appropriate set patches is hard to find - Redhat and Montavista know about them, but they ain't telling because their business model effectively revolves around knowing what you need to do to make the software work. So, "Open Source" becomes "Closed Knowledge" because at the end of the day, everyone needs to make money and if the source is free, then charge for the knowledge / expertise.

This makes support an interesting proposition - you get companies who will help you, but only by doling out the information a piece at a time - because in their marketplace, knowledge is power.

Now, Montevista supply (excellent, patched and working) toolchains for all their supported platforms for "free" (or rather the cost of downloading 3 ISO images). By doing so, they effectively try to lock you in to their support model (which is around $10,000 a year for a single point-of-contact), especially when you discover that the range of BSPs they ship is pretty small, and expensive to add to - you're on your own if you're platform isn't on the list.

In the end, it's no better than proprietry solutions - just different.

Jon

Re:Interesting but

[alext]

The depressing thing is that books such as this encourage the myth that Linux is a viable cross-platform solution. Well, it's just about viable now, but woefully inadequate compared to where Dotnet will be next year. Either embedded Linux needs to start embracing Java, or the 101 different alternate VM efforts (Parrot, TCL, Mono, DotGNU...) need to start consolidating.

Re:Interesting but

[johnnyb]

It _is_ a cross-platform solution. In what way do you believe it is not?

Re:Interesting but

[alext]

This is unfortunately just wishful thinking, as others have pointed out.

1) The development tool chain simply isn't there for non-x86 developments such as ARM, unless you pay for it.

2) The deployment problems are crippling because all apps have to be targeted. And not just ARM vs. x86, but often ARM 9 vs ARM 7, PIII vs. AMD vs. VIA etc.

PDA users simply can't and won't build apps for themselves. The Linux model breaks down, resulting in a gift to MS because vendors are trying to avoid using Java, the only viable VM currently available.

Re:Interesting but

[johnnyb]

alext:

1) The development tool chain simply isn't there for non-x86 developments such as ARM, unless you pay for it

**********

Ummm... it's there for Sparc, PPC, Alpha, and others. More obscure architectures such as ARM do not require payment, just extra legwork. Have you seen .NET on ARM lately? I don't think so.

***********

alext:

2) The deployment problems are crippling because all apps have to be targeted. And not just ARM vs. x86, but often ARM 9 vs ARM 7, PIII vs. AMD vs. VIA etc.

************

What does this have to do with anything? There are many, _many_ sites that are dedicated repositories of binaries for particular architectures.

************

alext:

PDA users simply can't and won't build apps for themselves.

*************

Same for most Linux users. You know, it only takes one person to build it.

Having a VM is a waste of time. Why? Because why not just have a processor that does that VM. Any architecture can be virtualized, and the only benefit Java gives is the availability of VMs for it. You can VM x86 if you like.

However, having to target specific devices isn't a problem if you have a portable architecture. As I said, it only takes one.

For PDA users, there are only going to be so many PDA systems. I imagine those selling the PDAs will work on porting whatever needs to be ported. For most other devices, you don't add software. It just a box that does whatever, and any binary interoperability problems simply don't affect them.

Re:Interesting but

[dbarclay10]

Use Debian.

Not only is Woody (the next Debian/stable release, due Real Soon Now) being released for alpha, arm, m68k, i386, sparc, powerpc, mips, mipsel, hppa, ia64, and s390 (that's 11 architectures, friends), but in order for an architecture to actually be officially released, the massive bulk of all packages have to be compiled and ready on that platform - with _the same version as every other architecture_. You won't be getting XFree 3.3.6 on PPC and XFree 4.2.0 on i386. You'll be getting the same.

Oh, right, "not only" :) Most complex packages, which include extensive patches, are organised in a very nice way. 'apt-get source libc6', go into the new directory, and poke around in debian/, there should be a directory there with all the patches that are applied during the build process.

Nice and easy to seperate, pick 'n choose, what have you.

Re:Interesting but

[norwoodites]

This should have been modded as a flamebot.

This shows how people do not get away from x86.
IBM and MOT put together makes more embedded chips than intel does. The PPC series sells.
Cisco uses them and will use them for a while because they are fast and low power consumption something that x86 does not have.

Re:Interesting but

[alext]

Hmmm, you don't work in this field, do you?

...obscure architectures such as ARM do not require payment, just extra legwork. Have you seen .NET on ARM lately? I don't think so.

ARM will be the single most widely deployed PDA and phone CPU. Period.

.NET on ARM? Try here on the Microsoft site (of all places). To quote:

Windows CE .NET allows developers to select the optimal hardware architecture for their projects by supporting a broad range of processor families, including x86, Xscale, ARM, MIPS and the SH series. Device bring-up is now faster than ever with Windows CE .NET providing numerous board support packages and features such as the Platform Wizard, which provides preconfigured design templates for embedded devices such as handhelds, smart phones, Web pads, retail point-of-sale devices and set-top boxes.

Your devastating critique continues:

What does this [deployment problems] have to do with anything? There are many, _many_ sites that are dedicated repositories of binaries for particular architectures.

There are? I think you'll find that most applications (as opposed to platform builds) are offered in a few builds that the authors know about, and finding any others is down to luck.

Let's try searching for 'linux tetris' in Google, shall we? OK, here's the first one. Oh dear! There's one build, and it's for x86 Mandrake. Care to point to one of your "many, many binary sites" that has it built for a PPC, for example?

And, lastly, our special bonus, the extra-insightful:

Having a VM is a waste of time... Any architecture can be virtualized

Riiiigghhtt. Well, I hope you've taken the opportunity to tell Microsoft, Sun and all those phone vendors that they're wasting their time pushing VMs. I bet they'll be kicking themselves soon! Perhaps you'd like to point to a phone or PDA that runs a foreign CPU instruction set, just as an example?

Re:Interesting but

[johnnyb]

Have you tried looking on penguinppc.org? Or maybe one of the PPC distributions? Or even typing in Tetris PPC into the google search? You can find most Linux programs ported to PA-Risc at http://hpux.cs.utah.edu/. Pretty much, if there is an architecture, someone has a site full of ports. The point is not that each project owns the ports, but usually there is a single clearing house of ports to a particular platform, like the FreeBSD ports collection.

.NET on ARM was a bad example. However, what about .NET on other architectures? Which has more ports, Linux or .NET?

As for the VM, my point was that the specific Virtual Machine is not much of an issue, as much as everyone standardizes on one. But what's the difference between standardizing on a virtual machine and a real machine? You pick your platform and you write to it. Either it's popular or not. What I'm asking is what does the fact that you are writing to a virtual platform benefit you rather than writing to a stable, widely-used hardware platform? If someone wants a different platform, they can virtualize. Witness Apple switching from 68K to PPC.

Has anybody done a project like the scenario?

[Sheetrock]

One of the things that seems to come up consistently in this review is the concept that employing Linux as part of an embedded system solution will be more difficult than buying a commercial product and tailoring it. Does anybody have any real-world example where this was (or wasn't) the case?

Looking at the scenario the book presents, I'd expect to invest a good deal of time in customizing anything to fit the requirements, and it just seemed to me like Linux is far enough along that there wouldn't be a huge difference between trying to make that fit against trying to do, say, a WinCE solution.

Re:Has anybody done a project like the scenario?

[emag]

I wish I could say that I had. The trouble being, I got the itch to move out of defense work at probably the exact wrong time. Had I been more patient, I would have probably been one of the lead developers on a proof of concept to examine switching from VxWorks on PC104+ to RT/Linux on the same platform. I'd ask some of my old coworkers, but I've lost touch with them, and the one or two I still keep in touch with periodically are no longer working there either.

<speculation>
Chances are, the project is either going along well (assuming it was approved), or has already completed what it was set out to look at. Since VxWorks provides essentially a GNU suite for development and cross compiling (we were targetting x86, building on Solaris), I can't imagine that switching to RT/Linux would have been much more difficult. I'd like to think that by now, the project I was working on is no longer locked into a single-source supplier for the OS (Wind River Systems), and that there's now a heterogeneous environment for development. I guess I'll never know...
</speculation>

Re:Has anybody done a project like the scenario?

[lenski]

Yes, I have done it / am doing it now. It is (already noted to be) not a no-brainer! If your organization is small or has limited heavy-lifting build/makefile skills, it might be worth considering one of the commercial kits.

On the other hand, if your organization is large enough to dedicate some time, or needs to maintain a toolchain to meet long-term (ISO9000...) requirements for building "old" products from source, then it is probably worth the hassle of establishing and maintaining the toolchain. Of course, given that many of us are control freaks, we all just love having access to and control over our sources.

Our organization has "invested" :-) the effort in establishing our own toolchain in order to increase the likelihood that we'll be able to reproduce our compiled product long after the current generation of workstations has gone the way of the dodo. We keep backup sources for everything including GCC, binutils, glibc, etc...

Having built the toolchain, we can manage portability of our product code among several processor architectures locally.

Re:Has anybody done a project like the scenario?

[jguthrie]

Sure. The company I work for deployed Linux in an embedded product and it was a no-brainer. It certainly was easier than the Embedded-NT that was chosen for another part of the application. The secret to success was the fact that we use all PC-compatable parts.

The fact is, the farther you get from a desktop computer, (and, at least with Linux, the farther you get from X86 PC's, or so I've been told,) the more trouble you're going to run into trying to use a desktop operating system in an embedded device. Most embedded devices don't look very much like the general-purpose computers most people are used to.

Of course, the other experiences I've had with commercial RTOSes is that they require considerable fiddling just to get them to start and start your application. Also, remember that WinCE isn't really considered an embedded operating system by many embedded people. Something like psOS or Lynx or QNX would be more like what's considered a "real" embedded OS.

Re:Man this stuff is amazing...

[ClimberTech]

Imagine how far we've come in a hundred years. In a hundred more they'll have virtual reality games in computers inside sunglasses. Amazing stuff...

Hmm... interesting how this relates to Embedded Linux Books... :P

Odd comments about the GPL in this book

[00_NOP]

My copy tuned up yesterday and I started reading it in the bath this morning - looks interesting but has a rather odd line on the GPL when it says, in effect, if you write a device driver for the kernel it will not find itself covered by the GPL.

Now, I know that major wars rage on this issue in the Linux world, but this book leaves no room for doubt.

Anybody else read it and care to comment?

Re:Odd comments about the GPL in this book

[JKR]

It's your choice - as developer you have the option of released a non-GPL kernel driver (assuming you don't use GPL code, of course!) as a module. See the desktop NVidia graphics card drivers, for example.

Jon.

Re:Odd comments about the GPL in this book

[tzanger]

My copy tuned up yesterday and I started reading it in the bath this morning

The reviews and the sample pages didn't get into it -- how is it for depth? As an example, its chapter on USB interfacing seems rather... thin. How does it fare? Are there schematics? Is it an "Embedded interfacing for Dummies" type of book or is there actually some meat to it?

Why Linux ?

[MosesJones]

Why not Symbian, QNX/a> or any of these ? This unhealthy obsession of "one size fits all" that abounds in the Linux world is exactly the sort of thing that people on Slashdot complain about the Microsoft world. You can't shoehorn these things without getting a poorer product as a result.

Wouldn't it be a better open source project if someone did what Linus did when he wanted to build an Open Source Unix and do the same for a proper RTOS ? By viewing Linux as the "only" solution it turns into the old "everything is a nail if you only have a hammer" discussion.

News for Nerds would be detailing what is happening in the RTOS and embedded world, rather than just being "News about Linux" to the detriment of better technologies. I know it sounds like a rant, but people like Wind River really do know what they are doing, this isn't a crappy Microsoft driven arena, this is where people really do know their shit, and the customer will not accept failure as part of the package.

Re:Why Linux ?

[Lumpy]

brecause it has ZERO cost, ZERO royalties, ZERO license fees, it takes a total of 20 minutes to whip up an embedded distibution for an X86,Arm,SH,or other processor platform. and you have 100% control over every aspect of the software. something no other solution can offer.

If two engineers came to the boss with the same project, one was QNX, the other was Linux... the Linux solution will win as it will have larger margins for profits due to the significant reduction in costs, greater flexibility, and in today's day and age.... Increased profits or lower cost of manufacture will always win.

Let's not even mention the fact that I can even stuff a GUI on the roll-your-own embedded linux that uses 1/10th the resources of any other solution available by simply using either PicoGUI or Microwindows (Formerly known as NanoX) with PicoGUI looking better and faster to develop for than any other possible solution.. (I can put together an entire user interface under picoGUI in less than 10 minutes and less than 100 lines of code.. I can also use C, C++ PERL or python if I desire... making it even more flexible... the same GUI in WindowsCE will take 1300 lines of code. QNX is close to that also in bloat of code versus GUI design)

Re:Why Linux ?

[nevets]

It really is the openness of the code that attracts many to Linux. As to create a new RTOS from scratch, well that's a lot of work and not many companies will fund it. Someone did mention eCos from RedHat, but it's still not Linux.

What I mean is that Linux has a lot of drivers. To use another OS, you will not get as many drivers that come with linux.

Some say that Linux is "too big" for an embedded system. But today's embedded systems are not you daddy's embedded systems. They have more power and more memory.

Also you have the buzz word of Linux. We get a lot of reaction when we mention useing Linux for a device. There are a lot of managers out there that have heard the benefits of Linux, real or otherwise, and want to jump on it if they can.

I don't have much to say about Wind River. I use to work for Lockheed Martin, and had to deal with them quite a bit. The vxWorks we had had no support for virtual memory, and only supported FAT filesystems, which gave us a problem with a database server that had thousands of files. We had found a bug in their code and since our department wasn't a big customer, the support we got was to modify the source code ourselves (we had an NDA). This was pretty much the same as an open-source project answer, but we had to pay for the code, not to mention the "support"! Also, I might add that I got the impression that Wind River was pretty at ease with their monopoly on the embedded market that they didn't progress as much as they can/should. They seemed stuck in there ways of doing things, being a monopoly, and unless you were a GM, or maybe another department of LM, you really didn't get much out of them.

Sorry for the rant, and a little disclosure: I work for TimeSys.

Re:Why Linux ?

[thorsen]

There are three things that makes Linux stand out as a very good choice:

Free Software - to the company involved in robots that I used to work for, this is simply *the* selling point. It is a point that no other embedded system can do

*Very* good driver support. There simply is nowhere near the level of driver maturity in other embedded OSes

Since it's coming from a full-fledged system, you have the option of taking all the stuff off the full OS shelf. This means you can decide if you want to use TCP/IP, firewalling, X/QtEmbedded, java... The flexibility offered here is extremely good

A very good toolchain. Yes, gcc can loose very much in benchmarks against Intels or Microsofts compilers (though it still beats them in others), but compared to other RTOS compilers, it is simply so much better. The compilers are usually obscenely expensive, very buggy, and support is non-existant or equally obscenely expensive

To my former employer these points meant a lot more value than any other OS could offer.

Bo Thorsen,
SuSE Labs.

Re:Why Linux ?

[Lumpy]

Sure, no documentation, no one to fix your kernel/glibc/driver, no one accountable to your deadlines, no guarantee to the quality of code, ownership of the IP is frequently debatable...

Ok I'll nail this one at a time...

no documentation... Only a complete idiot and moron would say this... Linux is the most documented system on the planet, I can with only 20 minutes of searchin on the web thousands of texts with detailed technical documentation.. I can fill your office with reams of paper full of linux documentation... you know this so shut up with your lies...

noone to fix your driver... What the hell are you talking about? are you saying that the NE2000 driver that had trouble in 1992 is still broken because NOONE is fixing them? and that the Kernel is still broken? are you a complete idiot? because it sounds like you are. if you roll your own for an embedded solution you can choose from ANY driver you want.. hell you want to use a 1.x Linux kernel? GO FOR IT!, Me? I'd choose 2.4.17 and write the special hardware drivers in house and NON GPL'd to continue to be a prick corperation and to make the idiots in the management wing happy. if you use standard hardware, the drivers in linux get fixed faster than QNX and microsoft.. hell QNX has horrible networking drivers BTW.. I have a QNX development box that contantly loses a critical networking connection because the driver RESETS!... They confirmed that hhe problem is there and responded with "we are working on that" wow where's my fixed QNX driver?

No one accountable to your deadlines... Ummm yes, it's called the morons in R&D or the people working on the project.. Linux is great this way, the idiots that are doing the programming dont have a scapegoat to blame for their own inabilities and laziness... Yes, there is someone responsible... YOU.. and if your company hires people incapable of taking responsibility then that's their own problem...

No guarentee of quality of code... This is true, same with QNX, microsoft, everything else... SHOW ME one piece of paper from any of these companies that guarentee quality of code... they all EXPLICITLY make you waive that in their license agreements... again, you either are clueless or one of the above types of developers, people who have no abilities to take ownership of a task and do NOT fully research things.

Finally... Ownership of the IP... Hey, do YOU own QNX? no... Wow,, just like YOU dont own Linux. it's the same here... the only thing you need to be concerned with OWNERSHIP of is the application that is written for your product. If it's GPL that you are using, YOU DONT HAVE TO GPL YOUR APPLICATION.. Hear that? you ALSO DONT HAVE TO GPL ANY DRIVERS YOU WRITE... Wow amazing eh?

the only companies that whine about not owning the linux on it are the ones ran by incompetent boobs. who gives a rats ass who owns the underlying OS, if we can use it and ship it for free then profits are that much higher. If your company is based on a idiot's business plan that will make the company fail because some hacker out ther used your product as a toaster instead of a toilet disinfecter and you arent getting royalties from the toilet association and drain-o sales... then it all comes back around to my first point...

They ARE choosing linux for embedded by the way.. many more products that you know are using it, BIG companies are using it and developing weith it.

but they have smart developers and engineers... they dont need to have their hand held, so they dont BUY a linux solution, they make their own. something that QNX people are incapable of. (yes that was a direct and Cheap shot.. I used QNX for a platform for 8 months, it's unstable out of the box, overly limited, and needs serious help along with it's significant cost per unit. It is purely unacceptable for any sucessful product.)

Your companies "eval" .... please give details.. did your company sucessfully get a test system running in under 4meg of flash WITH the QNX gui? how about network communication with GUI interaction on something as slow as a 200mhz StrongArm.. did you get around the problems with the PhotonGUI hiccuping the communication system? how about the instablility in the QNX networking drivers?

let alone the fact that your "company" hasn't even heard of picoGUI until me (which shows that your reaearch team are bungling boobs) I have the sourcecode to an mp3 player for both PicoGUI and QNX.. the code for mp3 playing and file manip is the same... the GUI code is over 10 times the size in the QNX side...

there's the extra code sir... QNX has the same problems with almost everything other than picogui does in code size devoted to GUI. if you would have actually READ my post you would have seen I mentioned that. (and for your pleasure, I include X11R6 in that. along with gnome,kde,pde,gde,wee-dee-i-eee, and whatever else you want... I'm talking EMBEDDED SYSTEMS HERE not the slow and bloated world of desktop computing.

I should write a book.

[SHEENmaster]

Then all of you could tell me that it lacks "passion" and debate what type of license the book should be released under, whethere copyright law is legal, etc.

Re:I should write a book.

[johnnyb]

I _am_ writing a book. You can see it's current version at http://www.eskimo.com/~johnnyb/computers/Programmi ngGroundUp/

If anyone is interested, please read what's there so far and let me know if it is helpful, where it needs work, and so forth.

I'm especially looking forward to critical comments.

Re:I should write a book.

[JKR]

Well, your PDF breaks Acrobat Reader 5.01 - page down from the title page gives "There was an error processing a page. The page contents object has the wrong type." The table of contents doesn't seem to start until page 5. Corrupt PDF?

Jon

Re:I should write a book.

[tzanger]

Well, your PDF breaks Acrobat Reader 5.01 - page down from the title page gives "There was an error processing a page. The page contents object has the wrong type." The table of contents doesn't seem to start until page 5.

Weird... It works beautifully on Acrobat Reader 4 for Linux.

Re:I should write a book.

[johnnyb]

The problem is the interpretation of the PDF spec by GhostScript of blank pages. Acrobat 5 started being more strict about following the spec, but there are some vagaries, specifically with blank pages. GhostScript tries to make the Contents Key have an empty array, while the spec says that the Contents key should have an array of streams. Technically, an empty array could be considered to be an array of streams of length 0, but apparently Acrobat 5 disagrees.

See http://www.ghostscript.com/pipermail/bug-gs/2001-J uly/000535.html

for details.

NetBSD

[cpeterso]

When I tried installing Debian Woody on my Sun Sparcstation IPX firewall, it would freeze at the same point during every installation attempt. I emailed Debian's Sparc list for help. One person said they had the same problem, but no answers. Another person said the standard boot floppies were broken and to try some random Sparc boot floppies on some guy's random FTP server, but those had the same problem.

I gave up on Debian and installed NetBSD 1.5 without any problems. Even with Debian's cross-platform "support", I would still be very wary of using Linux on non-x86 platforms. You will still need to jump through many hoops. Is Linux only free if your time is worthless? Admittedly, NetBSD is not as sexy as Linux, but it supports many hardware platforms, has a "business friendly" license, and has commercial support.

Re:NetBSD

[dbarclay10]

Woody is not yet released.

Don't install Woody. :)

Install Potato. Or know what you're doing :) It's one, or the other.

If you're not inredibly familiar with Debian ingeneral, you really shouldn't even consider installing something other than the RELEASED Debian...

You were installing the equivalent of an early Red Hat beta release :)

Re:NetBSD

[cpeterso]

you are right, but this was after the supposed Woody release date of May 1. I figured that if Debian is THIS close to finally releasing Woody, then simple fundamental features like installation should probably be working. I guessed wrong, I suppose.