Slashdot Mirror
Porting Open Source to Minor Platforms is Harmful
Tamerlan writes "Ulrich Drepper posted a blog entry titled "Dictatorship of Minorities". He argues that open source projects' attempts to support non-mainstream (read "non-Linux") operating systems slow down development and testing. While Ulrich may be biased (he is a RedHat employee) he has the point: if you ever read mailing list of any large open source project, you know that significant piece of traffic is about platform-specific bugs or a new release broken on some exotic platform."
But I agree with the opinion if, as a developer and/or integrator, you are attempting to reach a mass market. Red Hat, then SuSE and others have abandoned the Sparc platform awhile ago. I bought an old Ultra 1 and am now limited to a few distros to keep my Sparc machine up to date.
But it was my understanding that the idea behind open source was to roll your own if your machine was not covered. If I want to keep the Sparc chugging along, I guess I'd better learn to port it myself.
"Rocky Rococo, at your cervix!"
The development of netBSD to over 40 different platforms has brought a lot of good development to many different platforms that would have been dominated by mono-operating systems. A good instance is the handheld devices platforms (HPC,Palm PC, etc.), which would otherwise be dominated by Windows CE (except for the few Linux Palm PC's, but majority are WinCE). The development of netBSD for the majority of platforms has reached great maturity, and it is still developing well.
This post was surely inspired from this message to libc-alpha.
He's curt to the point of being rude, and I'm surprised anyone wants to develop on anything he's involved with. I wonder if the more social glibc developers like Roland agree with his position?
Doesn't that sound like something MS would say? Don't waste your resources developing for small-time operating system like Linux or Mac when only the large market platform, ie. MS, is feasible for reaching your feature goals in a timely manner?
Come on. While this is an excuse for a proprietary for-profic product with a small team, I don't see this as flying as well in open source land. If some guy wants to port Firefox or OpenOffice to something off the wall like AROS or some other nearly unknown platform, let him. He wouldn't be working on the Windows or Linux ports regardless, so why prevent him from doing something so crazy with his own time and money?
If you write your code to be portable in the first place, fixing platform-specific issues should be quick and easy.
And, of course, you write your code to be portable because you make sure it runs on the big three: Windows, Mac OSX, and Linux.
Right?
Actually, I think a much larger problem is just that: Many OSS developers don't even try to support Windows. Yes, I know you hate the OS and don't want to support Microsoft, etc., etc.. But, how can you complain about major software not supporting Linux when you're writing your own software that doesn't support Windows? Isn't that entirely hypocritical?
My take: Port your software to every platform you can, especially Windows. This gives freedom of OS to your users. And if you're a Linux user yourself, you should understand just how valuable and important this freedom is.
Okay, so the AIX guys are asshats in the way they treat GCC, fine
I've watched the GCC list; David Edelsohn has consistently worked with the other people on the list. He takes the time to work out a solution to the AIX problems. OTOH, when Ulrich Drepper thinks something is broken on GNU/Linux, he will fight tooth and nail for the solution he thinks is right, as long as it doesn't mean that he has to clearly explain why his solution is right.
Very good point. Supporting minority platforms is only troublesome if your code is riddled with i386isms and lazy coding practices. If you do it right the first time with always portability in mind, not only will you get cleaner and more maintainable code, you also get the ability to run just anywhere as a bonus. My prime example as usual is NetBSD.
/x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(85): warning: pointer casts may be troublesome [247] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(94): warning: pointer casts may be troublesome [247] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(111): warning: bitwise operation on signed value possibly nonportable [117] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(111): warning: possible pointer alignment problem [135] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(111): warning: pointer casts may be troublesome [247] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(111): warning: possible pointer alignment problem [135] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(111): warning: pointer casts may be troublesome [247] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(111): warning: possible pointer alignment problem [135] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(111): warning: pointer casts may be troublesome [247] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(111): warning: possible pointer alignment problem [135] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(111): warning: pointer casts may be troublesome [247] /x/s.NetBSD.netbsd-3/crypto/dist/openssl/crypto/de s/fcrypt_b.c(111): warning: bitwise operation on signed value possibly nonportable [117]
It's sad that whenever you build some GNU sources or the latest Linux app du jour you get tons of warnings. Some projects even compile their files with 2>&1 >/dev/null. How sad is that?
It's not just the Linux folks. OpenSSL is actually worrying me:
etc. etc.
And this in a project that's driving quite an amount of sites, authentication mechanism and what not?
Most if not all of the sources that are native to NetBSD, i.e. not imported like OpenSSL and GCC compile without any warning. You automatically get a good feeling about using it.
Something needs to change in coding land.
</rant>
Of course it runs NetBSD. BTC: 1NT7QvbetmANwaMzhpVL6
After I read "AIX is irrelavant," I read a little further, hoping to find the necessary justification for such a statement. When I didn't find it, I stopped reading. I appreciate the point he's trying to make, but randomly firing shots across IBM's bow doesn't so much aid his argument as reveal a gross personal bias. That said, perhaps GCC doesn't need to support AIX, when much better alternatives (VAC/XLC) exist. Anyone who can pony up the dough for a nice POWER box can probably ought to pitch for a good compiler too. :) Even in Linux, XLC will run circles around GCC. But then, that all goes back to the question of GCC's purpose in life . . . broad support, or good performance. I'm not convinced you can have both, and I don't think there's any real middleground between them.
I tried linux, backed away once. I tried firefox and thunderbird and loved them. I know use Mepis with firefox/abiword/thunderbird because I became comfortable with them. It's a "taste" that gets you. Why do you think they give free samples of cheese at supermarkets?
I don't get it.
Well, by his logic, you should only develop for Microsoft, because Unix/Linux/Mac and everything else has only a minor overall deployment rate.
Just sounds to me like someone's a little greedy, because it only hampers the main platform (if it even does that). But, in the process, it raises these other platforms. This gives more variety, choice, options... all really obvious things.
I got into OSS as a consumer because, while a lifelong Windoze luser, I was also broke. I must have tried just about anything they gave away for free if there wasn't a download.com message board that said it made your mobo burst into flames. Over time, I noticed that I got all my best stuff from sourceforge.net and stopped going anywhere else.
When my wife told that her company's thrift store could set me up with a PII 300 Mhz for $20, my eyes lit up, because by then I knew that OSS had an OS up to the job of making that old clunker the same usefool tool it was when it was put together. After actually getting my Bittorrent/FTP server running (and skirting a nasty argument with the wife about P2P on her/our machine), I know there's no way I'm going to get another machine that puts money into Micros**t's hands.
If keeping Xine or XMMS running on Win32 slows them down, screw it! Media Player Classic has saved me the evils of Real Player and WMP, and confirmed for me with personal experience what everyone says about OSS. I don't need to be enticed with keeping it when I run my real OS on a real computer, the point's been made. As far as Windows (not Cygwin or Mingw, I'm not throwing my ignorance around in that debate) is concerned, you can say the port should be its own project, and there's a lot we won't be left out on.
That's what this whole argument is about. To incorporate the platform-specific switches in the main gcc code, for example, would slow down every future gcc release to test on symbian/nokia. The alternative is to keep gcc simple(r) and let the minor platform (symbian in this case) use their own branch. There's something to be said for either approach.
If you slow down development to cover a broader set of platforms then you will be late with the latest new buzzword features (hyperthreaded keyboard support!). On the other hand, who gets to decide what counts as a minor platform? Do you decide based on age? (sorry, Amiga is too old) or based on number of units? (sorry, not enough IBM mainframes out there) or just interest level of the developers? Anything that you abandon is going to cause big problems to someone.
primetest.cpp
primetest.java
primetest.pl (for fun, because I like perl)
Using Java Blackdown-1.4.2-01, gcc 3.3.5-20050130 and perl 5.8.5 on a 1.3 Ghz Pentium-M, I get these results:
neil@t40-n Documents $ javac primetest.java
neil@t40-n Documents $ time java primetest
real 0m7.809s ./primetest
user 0m7.700s
sys 0m0.033s
neil@t40-n Documents $ g++ primetest.cpp -o primetest
neil@t40-n Documents $ time
real 0m2.699s ./primetest.pl
user 0m2.676s
sys 0m0.007s
neil@t40-n Documents $ time
real 0m47.928s
user 0m46.207s
sys 0m0.138s
neil@t40-n Documents $
How about you? Sure it's a trivial benchmark, but it definitely shows Java way behind (by a factor of three!) C++ for number crunching. Of course we see Perl well behind both, but it's definitely not meant for number crunching, so no surprise really.
Game! - Where the stick is mightier than the sword!
I've actually done some fairly hefty optimizations (real-time array processing on an original Sparcstation, for a nuclear research facility) and know the sort of problem you are talking about. The best thing to do, once you have a working, solid, implementation is profile it and find the areas that - if accelerated - would offer the best improvement in performance.
The problem with having so little time - in your case a month - is that you can't optimize cleanly. You've got to hit the code in the best places first and dig your way through as far as you can in the time.
Typically, slow areas will involve moving lots of data, number-crunching and I/O. You can't do much with graphics I/O, unless you've some good specs on the hardware and know assembly. (That's one reason I learned assembly.) Number-crunching is tough, too - just do as little as possible, do what you can when the system is idling, try to merge operations, and stick to integer arithmetic as it is so much faster. Moving data is easy - don't, unless it'll absolutely kill you. Do everything in-situ, if you can. (I wrote a Mandelbrot generator that did ALL arithmetic inside the 80x87 coprocessor - nothing loaded, nothing saved, which meant no slow transfers. It was still slower than fractint, but it was a lot faster than any other floating-point fractal generator out there.)
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
I have seen this phenomenon but it certainly doesn't have to be that way. One of the best kept secrets of having both cleanliness and portability is to eliminate #ifdefs from sources as far as possible. The obvious ideal is to have identical source and abstract platform differences by other means (e.g. macros instead of #ifdefs, and using identical APIs on different platforms, supplying only platform dependent implementations).
Keeping it clean probably also requires occasional refactoring and I guess some projects shun this as a matter of policy. The projects I would cite that achieve a high level of portability typically have no machine dependent code at all (TeX, for example). Not all code has that luxury but it's not as difficult as many people think - apparently including our friend Drepper.
you had me at #!
Have you ever developed on Linux in C and then tried to switch to Unix type system? I suspect not, since you are saying such things.
I learned most of what I knew about POSIX from GNU manpages... Then when I first tried to code for OpenBSD, Solaris, Irix, what have you, I was in for quite a few shocks.
I've learned a lot about writing portable Unix software since then. Still, porting between Unices throws lots of curveballs, especially if your development platform is Linux. Linuxisms (or glibcisms) are a trap all too easy to fall into. It's much easier to make something portable when you're not using glibc.
I track development of The GIMP - it just compiles fine in all "strange minor platforms", and a recent chain of Bugs in Irix compiling was resolved overnight - a matter of the Irix user reporting the bugs, and the core developers commiting the fixes.
However, there is a non minor and weird platform which actually does generate a lot of trafic on the list, and is strange for most developers. Anyoen checking The GIMP bugizlla will find a lot of open bugs for Microsoft Windows Plataform. That however, doesn't slow the project either. It simply goes on, and the developers who work on Compiling and making the windows installer do what they can for the work arounds.
-><- no
Ulrich Drepper certainly is an important contributer of OSS by being the GLIBC maintainer. However specifically the glibc project is often a huge source of problems because of all the "magic" glibc tries to do in its headers. For the last few version of GCC, all this "magic" actually causes the compiler to produce worse code compared to if the header files would just restrict themselves to define the appropriate macros and prototypes. More than once, the GCC folks have to strugly to get newer versions of GCC to work with older versions glibc because of all the cruft in it.
I think before criticizing maintainers for other platforms where GCC is ported, he should look at how much special work his own project causes for GCC and see if he can do his own contributions to making GCC development easier by simplifying the glibc headers.
Marcel
With a project as big and important as GCC, you'd think they'd have a server for each platform set up for all their developers to play with. Gentoo has Sparc, MIPS, PPC, etc. boxes for their developers to use for porting software.
It seems to me that a smart idea would be to have some kind of system where a developer could submit a patch, which would then be sent out to a server farm, where each server would try to compile GCC with the patch, then run a test suite. Doesn't Mozilla do something like this nightly?
"I don't have access to a [foo] box" should never be a valid excuse with larger projects.
LOAD "SIG",8,1
More specifically, the VIRTUAL MACHINE architecture addresses this problem. The .NET/Mono framework is another stab.
I suspect that we have not yet seen the VM architecture in its full maturity.
Surely in vain the net is spread in the sight of any bird -- Proverbs 1:17
I too have found endian specific bugs in code I maintain. It's why I keep an Ultrasparc III in my closet for compile testing.
I can easilly port between Linux and *BSD but the problem comes when users ask me to port my code to non Posix OS or claim to be Posix but really aren't Posix(win32). Or my latest peeve: sort of Posix but non ELF non GNU (OS X).
and boy is it ever ugly.
it has a lot of ugly nextstep-isms though, including the heavy leanings toward objc and bundles.
and for a long time osx was missing even basic freebsd mechanisms like dlopen() (yes, i know someone wrote a wrapper, but it took a long time for apple to appropriate it as an official api).
People seem to be repeating a lot of "folk wisdom" about portability. Oh it's just bugs. Make your damn software clean you damn coder. Etc.
I can guarantee you that anybody who says this has never actually read the sources to glibc, binutils or gcc. Hell, they probably never even read the mailing lists! I have, and when Ulrich says an enormous amount of effort goes on supporting minority platforms he is totally correct. Hello, binutils isn't the GIMP people! Other platforms have totally different architectures and often need huge amounts of platform specific code from these projects. This isn't a case of sloppy coding, it's a case of massive amounts of work being done to support edge cases. Go read the sources to bfd sometimes. Adding support for one platform that uses different assumptions about basic things like memory layout can require huge reworkings of the code.
Essentially there are a lot of people spouting off here based on their experiences of compiling FooApp on FreeBSD or whatever. Have you written a C library? A threading library? No ... then you are probably not really qualified to judge how much work this generates for Ulrich.
Oh, and for the guy later on in this thread who says "AIX is not a minority platform, WTF?" - I say to him WTF. AIX most definitely is a minority platform. Maybe not in the world he lives in, but in the real world Windows is dominant, sometimes people think about Linux/Mac or even FreeBSD and everything else barely registers at all unless you administer high end servers or work on embedded software (most people do not).
I've been bitten by this mentality before. Back when exec-shield was first developed, it broke Wine (which I work on). So I set out developing a fix. Eventually, I wrote a GNU linker script that arranged virtual memory such that things would work correctly even when exec-shield was active. But it didn't work, because of a simple bug in the kernel. No problem, right? Just fix the bug, right? Well, actually, somebody did. A patch was written, submitted, got into Andrew Mortons tree ... and it didn't compile on Itanium. The original author didn't have access to such a machine, neither did I, and the person who reported the failure (who worked for Intel, IIRC) was overloaded with other work and couldn't fix it. So the patch was dropped.
In the end, a few months later, we had a different solution that was about a million times more complicated. Largely because a simple bugfix patch didn't compile for unspecified reasons on a platform nobody uses and this was grounds for it to be dropped. That mentality of "all computers are born equal" is why Debian has become a laughing stock and it cuts both ways.