SourceForge Announces Compile Farm

HeUnique sent us the NewsAlert press release regarding SourceForge's new compile farm. For the projects hosted there, it means that they will be able to do test compiles on both Linux and *BSD systems.

Comments

Nice to hear

[Brian Feldman]

Encouraging cross-platform compatibility (even though it's all just Unix ferchrissakes! *G*) is a good thing, especially with all of the people using SourceForge :) They could probably take some tips from the FreeBSD Ports port-builder cluster that Satoshi Asami runs; he's been running this for quite a while, and has the same situation of doing it for multiple platforms (and yes, FreeBSD 3.4-STABLE, 4.0-CURRENT, etc. are quite different platforms, if mostly just for the ABI). I would suggest that the person who's going to be running this at SourceForge should contact him and kindly ask for some tips on it =)

--

Re:Nice to hear

[Brian Feldman]

Sorry for forgetting this link:
http://bento.FreeBSD.org is the moniker for the actual port-building cluster. Satoshi's documented some very good ideas there, including trying each port under its own clean chroot to make sure dependencies are correct.

It seems you've already gotten the idea of putting multiple distros in chroots on the machine, so it could be trivially extended to have a tarball containing that environment that could be unpacked and chrooted to for testing :) That's how the building for 3.X-STABLE is done, on top of a 4.0-CURRENT base.

It might be really nice to have a ports-type tree to do this, with more simplistic Makefiles which have targets for configuring, compiling, installation, and cleaning. Full SourgeForge "tree" building could be done that way =)

--

Re:Other Unices?

[Brian Feldman]

You either seem to have some delusions about the real world here, or it seems that you're trying to write flamebait.

Increasingly, the quality of applications can be linked to the number of supported platforms and the number of geeky libraries required. The more platforms and fewer libraries, the better.

I'm sorry, but that's simply untrue. Libraries are there so everyone and their mother doesn't have to include the same freaking code in their programs. Libraries are to simplify things and allow code sharing. Maybe you're thinking about libraries from hell which enforce a particular coding style on the users (you know what I'm talking about.) I'm sorry, but I like to see my programs that use JPEG imaging linking with libjpeg, just as I like to see programs using compression streams linking with zlib. You cannot say anything about a program's quality because the author linked it with important support libraries.

1. The GNU version of tar
2. The GNU version of make

I think you meant to say here that programs shouldn't be using GNU extensions of anything. I can agree with that for the most part, but...

3. Any particular kind of C compiler beyond specifying ANSI compliance or other standards compliance

The only real standards for inline assembly usage are in GCC. You're also kidding yourself if you think that most C compilers are nearly as closely standard-compliant as GCC is with ANSI. When something requires machdep code, it has to be done reasonably.

Also, you really need to note that many things won't compile on some extremely esoteric operating systems because they are simply not a good enough facsimile of the Unix API. Don't pretend for one minute that anyone's going to stop using the Unix API for portable software.

--

no, of course not

[hawk]

It's a place where they do eieio . . .

:)

The GNU configure and build system

[kip3f]

Here is an introduction to automake/autoconf.
--
Whether you think that you can, or that you can't, you are usually right.

Re:Sparc / Alpha ?

[Precision]

Accually we are working on getting some other platforms (sparc, alpha, ppc, etc..)

Re:Other Unices?

[Jon Peterson]

This is VERY true.

When we had 'free software' it compiled on pretty much everything that smelt of Unix and often on things that didn't.

Now, much of the latest and greatest 'open source' software seems to think that cross platform means RPMS for Red Hat AND SuSE. It really annoys me. I see shit like "This was designed as a cross platform project, so it should work on Linux and *BSD".

Increasingly, the quality of applications can be linked to the number of supported platforms and the number of geeky libraries required. The more platforms and fewer libraries, the better.

My definition of cross platform does NOT include requirements such as:

1. The GNU version of tar
2. The GNU version of make
3. Any particular kind of C compiler beyond specifying ANSI compliance or other standards compliance

At work I have a very well looked after Sparc-solaris 2.7 box. Free software that doesn't build on it, and whose readmes only talk about Linux goes straight in the bin. Yes, I do email the authors so they can fix it. No I don't immediately sign up as the official Solaris porter for the project and fix it myself. Yes I do like to point out that if they'd written it in Perl* they wouldn't have these problems :-)

The traditional argument has been that the developers don't have access to the wide range of platforms to test on. Cynics like me might point out that SHURELY wonderful amazing open source software will never suffer from such problems because it has 2.6 billion potential developers who have every known flavour of every hardware to test it on. Sure...

So, it would be nice to see a place with some HP-UX machines, some RS/6000 gear, and some Irix boxes so that people can actually write cross platform code. Or, to be more accurate, so that people would no longer have an excuse when they didn't write cross platform code.

*Substitute Java for Perl if you feel that way inclined :-)

Writing Portable Software

[howardjp]

This leads me to an interesting problem. I am developing a program. I am using FreeBSD as my development enviroment but my target environments are BSD/OS, OpenBSD, FreeBSD, SunOS, and possibly UnixWare. Obviously, anything else would be great on top of this. But how does one write code that will compile without problems across so many different architectures and platforms? Is autoconf the way to go? Is there a decent manual on autoconf? The standard documentation that comes with it is a bit lacking in how to make it work with a new project. Ideas?

Re:Writing Portable Software

[Ed Avis]

I believe Mozilla has its own portability layer, the Netscape Portable Runtime or NSPR. Maybe you could use that for other programs if the licensing is acceptable.

Re:Writing Portable Software

[Arandir]

"Perhaps a testing enviornment could be written that emulates the various configurations above"

A most excellent suggestion. Myself for one would find such a tool to be invaluable. Most developers who don't work on more than one machine would be surpised at some the assumptions they make. At home I work on a Linux PC. At work I have a Solaris Sparc5. When I compile my code at work I often discover where I have made my bad assumptions.

This is a great way to get rid of "linuxisms", but it requires a range of hardware to test on. The SourceForge idea is great, but it needs to be expanded as opportunity and cash allows. Add some Solaris and IRIX boxes. Include a larger variety of hardware.

Re:Writing Portable Software

[Arandir]

Autoconf is only a small part of the problem. So small that I'm tempted to say it's not even necessary. Autoconf will detect what's available for you on a system, but it won't write your code.

The best way to write a program that will work on any Unix is simply not to write one for Linux only :-) No, I'm not being flippant. Write according to the Unix standards. Use POSIX and avoid Linux-only kernel calls. Use standard C and avoid the extensions in glibc.

And finally, don't make the same mistake millions of other developers make every day, don't assume that the user has exactly the same system and setup that you do. Don't assume that the user has a large monitor or lots of memory. Don't assume that they have an active connection to the net. Don't assume that home directories are under /home.

Re:Writing Portable Software

[Jon Trowbridge]

The combination of autoconf and automake is definitely the way to go.

The autoconf and automake docs can be a bit obscure; most people seem to learn how to use them by studying the configure.in and Makefile.am files from other projects.

There is also a brief discussion of autoconf and automake in Havoc Pennington's Gtk+/Gnome Application Development that might help you get started. This fine book is available on-line: here is the relevant section.

Re:Writing Portable Software

[LetterRip]

"And finally, don't make the same mistake millions of other developers make every day, don't assume that the user has exactly the same system and setup that you do. Don't assume that the user has a large monitor or lots of memory. Don't assume that they have an active connection to the net. Don't assume that home directories are under /home."

Perhaps a testing enviornment could be written that emulates the various configurations above - ie small and large amounts of memory, small vs large screens, active and innactive net connections, as well as other areas that are often overlooked by programers - millions of colors vs four bit graphics...

Just a thought,

LetterRip

Re:Writing Portable Software

[Cramer]

I heartily agree... glibc is an evil creation; libc isn't supposed to be portable.

What you need is an abstraction layer to allow the same set of code (read: your code) to be compiled without alteration or other system dependance on various platforms -- this means more than ten versions of Linux/FreeBSD. The Cosm CPU/OS layer is designed to do just that. With a Cosm library for a given platform, your code will compile without modification.

Cosm is still in development (slow, but what do you want for free.) The Cosm API source is available via http, ftp, and CVS. Most of the functionality one would need is there. We would welcome your input and feedback.

Re:Writing Portable Software

[espensk]

I can see your concernt with mingling GPL'ed code with your BSD licensed program. The configure scripts however (as generated by autoconf) are not GPL'ed. The last time I did an autoconf, the script that was produced contained the following text in its header:

This configure script is free software; the Free Software Foundation gives unlimited permission to copy, distribute and modify it.

I would say that this enabled you to do most anything you could imagine with the script. It is afterall ``compiled'' out of your configure.in files that you miy very well put under the BSD license.

Use a portable language

[tilly]

Depending on how important performance is, you may want to just write it in a portable language, like Java, Perl, or Python. Possibly using C at some key places to speed things up...

Cheers,
Ben

Security concerns

[XNormal]

This means that anyone can execute arbitrary code on their compilefarm servers by including it in the build scripts. They need to make sure they have really good local security.

Another problem is that some software packages assume that "make install" will always be executed by root and include things like "install -o root" which will fail for non-root users. This is the most common reason for source RPMs which cannot be built by a non-root user.

----

Re:Security concerns

[Straker Skunk]

Another problem is that some software packages assume that "make install" will always be executed by root and include things like "install -o root" which will fail for non-root users. This is the most common reason for source RPMs which cannot be built by a non-root user.

Anything that uses -o root in the install is broken. That means I can't install things to e.g. a home directory. (There might be exceptions to this rule, e.g. dpkg, but those are very few and far between)

The usual problem with SRPM's, however, are install -o user commands written directly into the .spec file (not the Makefile). I keep a nice, big stick labeled "%defattr" for folks who write those :P

Re:Security concerns

[QuMa]

The arbitrary code thing shouldn't be a problem, sourceforge already gives you a shell account iirc, and you can get shell accounts at lot's of places nowadays.

As for the root thingy... BSD has jail()... As for linux I'm not sure, chroot is not root-resistant.

Re:Security concerns

[goemon]

It should be pointed out that Debian, at least, has a fakeroot command, which through LDPRELOAD makes it look like the user is root; as a result, tar etc. will set permissions correctly. This is commonly used when building Debian packages, and an equivalent could easily be written for all the other OSes, I believe (although I don't know BSD well enough to tell you whether the same trick will work).

Security? No problemo

[Straker Skunk]

X can be sent through an encrypted ssh tunnel. Security is no more a concern than for the shell access (terminal line) itself.

What would pose some problems, however, is bandwidth. I don't doubt VA has reams of it, but several hundred simulataneous X sessions (particularly of fancy Gnome/KDE apps with heavy high-color graphics) could bring even a .com's netlink to its knees.

There will probably have to be a bandwidth policy, to keep things working smoothly. It would be completely reasonable.

(And as an aside, to VA: THANKS!!! The Compile Farm is an excellent idea! I look forward to testing and building my apps there. HP/UX and Alpha binaries, here I come!)

Re:Features, Current and Future

[Straker Skunk]

You forgot to say, "Thank you."

Because even if VA is doing this to help move more hardware, they still deserve the kudos. The compile farm (even if all x86) will allow developers to catch subtle bugs brought on by distribution quirks, as well as allow them to create RPM's *and* DEB's easily from one spot. It's a win-win situation, and one of those wins is ours.

But I definitely agree that other architectures in the farm, including a plethora of non-free Unices, will be a boon to everyone. Each non-GNU system has its own eccentricities and bits of brain-deadness; testing code in such enviroments provides first-rate insight into its robustness and portability. The disadvantage to Linux/gcc is simply the sheer quality and capability of the platform. When you write for it, you're writing for a better Unix than "Unix" itself. (Hence you see all these apps that won't build on commercial Unices, and-- in the worse cases-- even *BSD).

I know when I compile stuff under IRIX (for example), the compiler gives me warnings I don't see under gcc, some header files (libintl.h for one) are nowhere to be found, the paths for X binaries and libraries are completely different, etc. You have to be aware of things like that to write solid configure scripts, and design the code accordingly. Myself, I'm lucky enough to have access to computer labs lined with Sun and SGI workstations. But many developers don't. And even I can't build/test anything on HP/UX, Tru64, AIX, etc. MIT doesn't support those {:-(

What you're saying, however-- having each and every Linux/BSD/etc. on each and every platform that they runs on-- is not worthwhile. Sure, that covers a lot of possible end-user setups, but keep in mind:

• Do you want to test and build for 30+ slightly different OS/platform combos?
  
• Heck, forget using; who here wants to administrate that??
  
• Is it even necessary? Sure, Linux distros have some incompatibilities, but IT'S NOT THAT BAD!!! This isn't UNIX, after all...
  
• In the end, when you're building something with gcc, the platform doesn't make all that much of a difference anyway. The GNU libraries are well-designed, and work around a lot of those issues. (Endianness isn't even that much of a problem, if you code things the right way)
  

IMHO, it would be much more sensible to add in a group of machines widely spread across the spectrum of platforms. Corner cases, if you will. If your app can build on three weird platforms, it'll probably build on a fourth. But you can write an app that will compile on every Linux distro in existence, yet fail horribly on any commercial Unix system you can name.

And let me emphasis the need to shun gcc and test using vendor-native compilers. If you're on a non-x86 platform, and gcc isn't the only compiler available, you have to cater to people who don't have gcc installed. And more important, the compiler warning messages tend to be a lot more interesting, simply because they're different from gcc's (and gcc's are almost always the same)

But putting an S/390 into the farm is overkill. Far and beyond. Those machines are waaaaay too expensive for something like this. (Seriously; the only thing to top it would be a Cray supercomputer!)

You can't always count on standards

[Straker Skunk]

Write according to the Unix standards. Use POSIX and avoid Linux-only kernel calls. Use standard C and avoid the extensions in glibc.

Agreed. Definitely worthwhile advice. But keep in mind that a lot of systems don't quite measure up to the standard. And that's where autoconf comes in.

E.g. older libc5 Linux systems don't have strtok_r() -- autoconf can detect this, tell automake to build strtok_r.c, and add strtok_r.o to @LIBOBJS@. Some systems don't quite agree on what the argument types to select() are -- autoconf can determine those, and put them into handy macros you can use for casting. Etc., etc.

The point being: Yes, write your code The Right Way(tm). Anything less is a hack. But use autoconf so that folks with less-than-perfect operating systems can compile it.

Re:Other OS's

[Straker Skunk]

Nahhh . . . Windows systems are easy to come by, and if you really wanted to build Win32 binaries, you can always grab Mingw32/Lccwin/etc. and use that. I don't think there are too many folks interested in supporting Win32, however. (We'll have to see-- things might change when the Windows-capable GTK+ 1.4 comes out)

As for Microsoft contributing anything: you have got to be kidding. Those boys not only don't have to encourage anyone to write stuff for their platform, they happily charge an arm and a leg for VC++ and all the MSDN program junk....

/. sing along...

[Otto]

Ol' Sourceforge had a compiler farm..
E- I/O I/O.
And on this farm they had a cluster..
E- I/O I/O.
With a make make here,
And a make make there..
Make make make make everywhere!
Ol' Sourceforge had a compiler farm..
E- I/O I/O.

---

Re:Shell or Web?

[tweek]

I think this is why you would still keep a local copy of your code and have md5 checksums for the packages on your personal distribution site. This is more along the same lines as the compaq testdrive program. If you just want to test a compile on another platform and do some debugging.

Re:VA, top linux company

[fusion94]

Well..I'm going to lose a few karma points for this but here goes:

Dude, You need to quit hitting the crack pipe.
We did not steal any ideas from any group of volunteers. The planning and development of SourceForge was done by 4 people. Those 4 were Tim Perdue, Drew Streib, Uriah Welcome, and myself. When planning and developing SourceForge we received no input from the community until after the site launch on Nov. 17th, 1999. We also received no input from VA. We worked on SourceForge in a "black ops" environment. By this I mean that we were kept separate and shielded from normal channels within VA. I know this because I set up the structure to be this way.

As far as giving back to the community we have tried since we launched the site to listen to the direction the community has wanted to take SourceForge. This is the community at work. Period.
We also released the code under the GPL so that others could help improve our code and if they were so inclined they could use the code to setup their own site.
We are also currently in the process of allowing other developers to work on the "live" code per se to help improve SourceForge.

Other Unices?

[Stephen]

This is a good start, but it really needs other Unices than the free ones to be included. Have you tried compiling a C program under SunOS 4 recently? Or HP-UX 9? These are where the real compatibility issues are revealed, IME.

(Of course, compiling under Windows/Mac/BeOS/etc. too would be even better, for programs which are intended to be used there.)

Re:Telnet Access to Different *x flavors?

[maan]

It's a service by Compaq which you'll find right here http://testdrive.compaq.com/
BTW, it wasn't posted a few years back, just 1-1.5 year ago...

Enjoy

What architectures?

[Oskuro]

I didn't have a look at SF.net yet to read about this, but I'm already wondering if it will be a i386-only farm or Sparc, Alpha, PPC, etc. will be available.
Many bugs in the Debian Bugtracking system are "fail to compile under foo arch" bugs, so it would be very cool to be able to test the code under various archs to avoid this. That would make SourceForge a unique environment to squash these types of problems, as many programmers can be aware their code does not compile on Sparc but they can't do anything about it as they don't have access to a Sparc machine.

About being able to execute binaries, and X and all... that sounds a bit more difficult here, imagine what hardware you need to allow that number of X sessions?

Long Overdue

[Lazaru5]

It's about time an organized effort was made to keep source code portable across the free OSs.

This is why C was invented in the first place, but too many people forget it.

Many programs already compile on multiple OSs, and kudos to their respective authors for writing good code. Other programs only require minor changes, which is where autoconf makes it easy. Then there's the occasional code that assumes Linux only (kernel or /proc) code, and won't build anywhere but on Linux (maybe even specific distros?)

</RANT>
Then there's the whole "Linux" software phenomenon. It's sad when people don't realize that the programs they use in Linux aren't "Linux" programs, they're Unix programs. I don't just mean grep, etc. I mean programs like XMMS, GNOME, KDE, etc. This stuff compiles, unmodified, on the *BSDs as well. Even some authors call their software "Linux" software when it's portable *as is*. This is misleading, both towards the Media, who pick up on it, but to new free Unix users as well, who might be deciding between Linux and a BSD. This new offering from SourceForge will not only provide a place for authors to test their code, it will help educate people as to the true nature of Open Source as well.

`./configure && make install` shall set you free.
<RANT>

Farms? Hmm...

[thomasj]

Is a compiler farm a place they do egcs?

URL?

[dlc]

anyone have the sourceforge or va linux url for this? I can't seem to locate it on either site.

Cthulhu for President!

Re:Shell or Web?

[Kujo_42]

They could easily put down a plethora of checkboxes and have a make button. You could probably upload all of your makefiles too, which would be easier. Anyone out there used it and could let us other pundits know? :)

What may be more interesting, and have serious ramifications, is the fact that you're storing your data off site on SourceForge's servers. A coordinated DOS attack to keep you from your source? Crackers going in and modifying your files perhaps?

Re:Shell or Web?

[dtype]

There is full shell access to the machines. Linux distributions can be switched on the fly with canned chroot scripts.

---
SourceForge Programmer Type - http://sourceforge.net

Features, Current and Future

[dtype]

I see a lot of questions regarding what is available on the Compile Farm, and a lot of great ideas as well. I'd like to dispel a couple of myths and offer some insight as to where this is going as well.

First, this is not a web-only service. We do like to provide web interfaces to as much as possible, but we do realize that for some things, program compliation and testing included, nothing can substitute for shell access.

A lot of people are asking about other hardware architectures and OS's. For now, the Compile Farm is i386 based, and contains several Linux distributions and FreeBSD. This does not mean that we have ruled out other possibilities. This is just another step in what we hope can be an expanding feature set for Open Source developers on SourceForge.

There is a lot of setup involved in something like this Compile Farm, not the least of which is having thousands of skilled Open Source developers with shell accounts on a set of boxes. We're attempting to keep things as secure as possible while also offering enough features to make this thing useful. One reason for the limited number of distributions/architectures/OS's now is the limitation of variables in a very complex system. Hopefully, we can work out the kinks in this system soon so that it can become a valuable resource to developers who might not otherwise have the capability of getting their hands on so many different machines.

We're also working on giving users the advantages ot having a cluster of machines available. Uriah Welcome worked very hard to provide parallel make capability to projects, and this is being tested now. (Parallel makes will allow you to take advantage of multiple dual-processor machines simultaneously in your compiles.)

Please be patient as we test this new system. We're definately open to criticism, but please also be constructive with it so that we can continue to improve these services. Thanks to all of the SourceForge users who have contributed patches, criticism, and helpful suggestions. Every day my confidence in the Open Source model increases...

---
SourceForge Programmer Type - http://sourceforge.net

Telnet Access to Different *x flavors?

[zpengo]

Does anyone remember an article posted on here a few years back about a server that let you telnet in and mess around with different flavors of Linux?

If anyone could remind me where that was, I'd be much obliged!

ICQ: 49636524
snowphoton@mindspring.com

Shell or Web?

[RevAaron]

I wonder if you have shell access, or a web based interface which excutes a set of commands like configure, make, and so forth. Hmm! Interesting stuff!

Sparc / Alpha ?

[SuperJ]

As an owner of a Sun Sparc ELC, I think it would be more useful to provide different architectures rather than different distros. This is a great move for VA.