Alternatives To SF.net's CompileFarm?

cronie writes "Not long ago, SourceForge.net announced the shutdown of the Compile Farm — a collection of computers running a wide variety of OSes, available for compiling and testing open source projects. SF.net stated their resources 'are best used at this time in improving other parts' of the service. I consider this sad news for the OSS community, because portability is one of the strengths of OSS, and not many of us have access to such a variety of platforms to compile and test our software on. As a consequence, I expect many projects dropping support for some of the platforms they can't get access to. Are there any sound alternatives with at least some popular OS/hardware combinations? Any plans to create one? (Perhaps Google or IBM might come up with something?)"

Comments

not to be a jerk but...

[User 956]

I consider this sad news for the OSS community, because portability is one of the strengths of OSS, and not many of us have access to such a variety of platforms to compile and test our software on.

Maybe the project has ended because that's not where the future of computing is headed. Maybe the future is something more like "write once, run anywhere".

Re:not to be a jerk but...

[Anonymous Coward]

I think thats kind of the point. Its more like write once, debug everywhere.

Re:not to be a jerk but...

[remahl]

Hardly. The future is and has long been one of "write once, test anywhere". And that's the need the compile farm filled. Writing once and expecting it to automatically run everywhere without modifications is a pipe dream.

Re:not to be a jerk but...

[Doppler00]

How did he get marked interesting? I'm pretty sure he was trying to be funny by quoting a popular Java phrase. Truth is, Java is pretty popular, but never got the market share they wanted. Still way to many platforms out there.

Re:not to be a jerk but...

[Vexorian]

I think the project ended because it wasn't enough. Compiling some C++ program in some platform is not as hard as making it work correctly in that platform...

And on that insight you have: Not even Java or .NET really work that way, so we are kind of far, far away from that.

Re:not to be a jerk but...

[rucs_hack]

sourceforge has been having increasing numbers of problems recently. Their shell service for instance was down for weeks not too long back. That's happened many times over the last few years, and it's been a source of real problems, since its the only way to get access to update projects.

Their entire service was off-line for a while last week, not fun.

I've moved my project to google code project hosting. Their service is simpler, but reliable. The addition of a wiki is really helpful, and uploading new releases is trivially easy.

google could offer a compile farm with ease. I expect it won't be long now that sourceforge have removed theirs.

When I first started using sourceforge four years ago I liked the service, but when they moved to having paying customers, everything started to decline for the free hosted projects. They said it wouldn't but it still occurred.

I'm of the opinion that sourceforge got too complex, and now they can't manage all the aspects they wanted to include. No doubt if everyone paid it would be easier, but not many open source developers have free funds for such things. If people had to pay then small incomplete projects might not even get off the ground. Mine certainly wouldn't have, since I was a student, and financially limited.

Vendor support...

[Lord Prox]

Naaahhhhh. Nice thought, but no computational utopia yet.
How about vendors supply compile farm gateways linked from SF.NET for use by SF members. Great way for hardware vendors to show off their new stuff to folks that might be inclined to buy or have influence in the purchase decision.

Kinda like a hands-on remote(?!) demo.

SciTechPulse. Geek News Netcast. Hot Polynesian Geek Chick Host Silulu.

Re:Vendor support...

[TeraCo]

The only problem is that the people compiling aren't the same as the people who are buying.

Re:Vendor support...

[Peet42]

The only problem is that the people compiling aren't the same as the people who are buying.

True, but remember that the more software that eventually runs on your platform, the more people who are likely to adopt it.

Re:Vendor support...

[imemyself]

I don't know if it would be suitable for the sort of thing you're talking about, but HP has (or atleast they had it a year or so ago), a thing where you can telnet into a variety of different systems they had. Mostly OpenVMS and HP-UX running a a few different architectures. I know that you didn't have network access from the box that you telnetted into, but I don't know what other restrictions there were. It might be something to check out if you're interested in making software for some of HP's higher-end stuff, but don't have the hardware to run OpenVMS or HP-UX.

Re:Vendor support...

[Anonymous Coward]

I've heard this argument before. From a vendor's point of view, it is so divorced from reality as to be laughable. Let's face it, if you, as a developer, have enough pull to direct a couple million US dollars toward a hardware contract (or even a few hundred thousand), you probably aren't sucking around for free equipment and resources. If you don't have the ability to throw that much cash at a vendor, there isn't a lot of incentive to talk, is there? Workstation/server vendors have different cost structures and markets than PC vendors.

Re:Vendor support...

[Atzanteol]

You're thinking of testdrive. My friend used to run that site. They have lots of machines you can telnet into and compile on.

Emulation?

[headkase]

I wonder if emulation of other hardware architectures would allow developers to try things out on their commodity machines? VMWare and Virtual PC do a good job for x86 emulation and there are many emulators for obsolete machines available so the question comes down to the time and effort required to implement new architectures. Maybe what could be practical is something along the lines of Transmeta's morphable instruction sets technology but with an extra layer of associated hardware (video, sound) emulation/translation.

VMs

[krakass]

With the availability of VMWare, Xen, etc. you can have your own CompileFarm. Obviously it's not a good choice if you're trying to render an animated movie or similar, but for testing or compiling it should fulfill most of your needs.

Re:VMs

[cbreaker]

That would only work on all x86 platforms.. so like, four.

QEMU won't do POWER, and it certainly won't run anything other then the normal OS configurations.

VMware is excellent for development, but has nothing to do with a render farm.

Re:VMs

[Curtman]

The SF.net CompileFarm was not there to provide 'power'.

I believe he meant this kind of power. ;)

Re:VMs

[that this is not und]

True, but you can test 68000 compilations on an X86 machine with Basilisk. I bet you could even run NetBSD/68k on it, if you had a LOT of free time.

Basilisk on a NetBSD/i386 box, running NetBSD/68k. Hmm. I suppose you could run it another layer deeper by running the NetBSD/i386 on bochs on a NetBSD/sparc box. Make it a SparcStation IPC just for fun.

Virtualisation negates the need for a compile farm

[Jailbrekr]

I use VMWare Workstation and Virtual PC to do testing and whatnot, negating the need for multiple systems in my home office. I have, for example, Windows XP Pro, Windows 2000 Pro, OpenBSD, FreeBSD 5.5 and FreeBSD 6.2 all set up as seperate virtual systems on a single computer.

Who needs a compile farm when most of what we need can be run from a single moderately decent workstation?

They announced this AFTER the shutdown?

[Animats]

Posted By: wdavison
Date: 2007-02-16 00:13
Summary: Compile Farm News

As of 2007-02-08, SourceForge.net Compile Farm service has been officially discontinued.

Shutdown on Feb. 8, announcement on Feb. 16th?

With behavior like that, SourceForge can't be considered a safe location for important code. I'd suggest that it's time to get projects off SourceForge. Make offsite backups of anything important now.

Latest announcement from VA Software, which owns SourceForge:

VA Software Corp., whose software and online media are targeted for the open-source software community, said Thursday it named Scott E. Howe to its board of directors.

Howe is president of a division of digital marketing company aQuantive Inc.

"Scott's extensive knowledge of the media markets will be invaluable as we continue to focus on our core media assets and strive to secure alliances in the global competitive landscape," VA Software President and Chief Executive Ali Jenab said in a statement.

VA Software slipped a penny to close at $4.24 on the Nasdaq Stock Market.

If VA Software thinks they're now a "media company", it's time to get off SourceForge.

Re:They announced this AFTER the shutdown?

[Animats]

It was announced afterwards for a reason. They're not really taking it down because nobody wants it or anything, it's because they lack manpower to keep it working. It basically needs a lot of work to get it back in a usable state, and it's not widely used, so they're just dropping it.

This is the classic downside of "software as a service".

Re:Virtualisation negates the need for a compile f

[Anonymous Coward]

That's not much use for testing compiling on Solaris on SPARC64, or Tru64 on Alpha, etc...

Dummy - Slashdot IS VA Software

[Anonymous Coward]

Animats wrote:

If VA Software thinks they're now a "media company", it's time to get off SourceForge.

VA Software owns Slashdot:

http://www.ostg.com/about/index.htm:

OSTG (Open Source Technology Group), formerly Open Source Development Network (OSDN), has had its roots in the technology community since its early days as the ground-breaking tech network Andover.net. Founded in 1996 with the mission to provide unbiased content, community, and commerce for the Linux and Open Source communities, Andover.net grew in community relevance and popularity by adding the provocative community-centric sites Slashdot and freshmeat.net to its technology group, and ThinkGeek and AnimationFactory.com to its e-commerce division. After its acquisition by VA Software Corp. (NASDAQ: LNUX) in early 2000 and the introduction of SourceForge.net and Linux.com, the network cemented its position as the Internet's leading destination for the Linux and Open Source community.

Ergo, VA Software is a media company.

Time to get off Slashdot.

It's no big deal

[EmbeddedJanitor]

Compiling is only half the fun. The compile farm cannot test most of the applications. Thus the compile farm only does half the job needed to release a package.

Most projects are staffed by people using multiple platforms anyway and anyone coming along with a requirement to support some odd-ball OS might just get pulled in to do compiles and tests. For example, the SF project I work on is mainly staffed by Linux people with a few Windows and this project does not use the compile farm. Those using OSX just need to recompile and it works for them.

Industry moving forward

[CaptainTux]

Personally, I see less and less need for compiled and distributed software as broadband internet becomes ubiquitous and rich internet applications become more sophisticated. As it stands now, there is very little that traditional software does that can't be replicated on the web using the right technology. Software as a service is slowly becoming a reality and compiled software is soon to go the way of the dinosaurs.

Re:Industry moving forward

[ZenShadow]

Yes, because all those rich web applications will run on... rich web applications?

--S

Re:Industry moving forward

[Excelcia]

And what is the client running? A web browser running on machine with an OS. So, you need compiler, programming, and testing infrastructure for:

• The application provider's OS
• The application provider's network services
• The application
• The client's OS
• The client's network client

And this is supposed to be a less complicated system to write, distribute, and debug than traditional systems that you can do away with traditional compile-farms? Software is a service, no need to install anything. Unless, of course, you want to print something. Or is that a service too? Burning a DVD is a service? Put your DVD-R in the drive, connect to your favourite DVD authoring service, and... go to sleep. Maybe tomorrow your disc will be done. Unless DVD or HD-DVD quality video is something you expect to get solely off broadband.

There are so many exceptions to what software-as-a-service can reasonably do that the majority of people who are reading this do on a daily basis that I just have to laugh when people bring this up. Beyond a wet dream for Microsoft where they lovingly sit back and watch the monthly subscription dollars roll in, this is never going to happen.

Re:Industry moving forward

[rbarreira]

Implement a media player on the "software as a service" model.

Now implement a cryptography library on the "software as a service" model. Oops, you're sending plain text data through the cables...

Now implement a real time application on the "software as a service" model.

Now implement an application which requires near-100% availability on the "software as a service" model.

Now implement a high-end game on the "software as a service" model.

Are you done? Do you like the results?

Can we start a replacement project

[the100rabh]

Can we start a community driven project similar to Compile Farm where people with systems contribute their system time in an anonymous fashion. Something like a p2p compilation.

Re:Can we start a replacement project

[Antique Geekmeister]

It's expensive: power, cooling, rent on the building with the rackspace, and bandwidth all add up to a considerable chunk of change. And the professional skills to run such a farm are unusual and expensive to hire, or to contribute. Even a modest Q/A testing and evaluation farm can cost a few hundred thousand dollars a year when you add up all the costs.

Debian build daemons

[Josh Triplett]

Get your software packaged by Debian (which you probably want to do anyway), and it will get built on (currently) 15 architectures of GNU/Linux, along with 3 non-Linux architectures (kfreebsd-i386, kfreebsd-amd64, hurd-i386), with more popping up occasionally.

Re:Obvious

[tiocsti]

It's rarely about getting stuff going on a platform, but rather making sure nothing regresses. Compile farms are useful for doing the following:

- compiling the software on all platforms

- running automated test suite

- automatically building packages periodically

- determining what percentage of the code your test suite covers

- verifying the built package works

Patches from users cant reproduce all of these things, and this is where compile farms come in handy. Whether it makes sense for something like sourceforge is another matter.

Solution: Rent some zombies from some hax0rz!

[Anonymous Coward]

Finally a legit use......

http://www.testdrive.hp.com/

[Harry8]

http://www.testdrive.hp.com/
HP dude Bdale Garbee has said HP is delighted if people use testdrive to test their code on different architecture and OS combinations.

I'll do it.

[delirium of disorder]

I can donate hardware and sysadmin man-hours, but I need either space, electricity, and bandwidth or money (which can obviously get me space, power, and bandwidth). I have lots of platforms just sitting in storage, and I plan to ebay most of it unless someone can get help for an interesting and useful project like this. The architectures I can provide are as follows:

4x Sgi o2 (MIPS both R10k and r5k) currently running IRIX, but I could install Linux, NetBSD or OpenBSD
Compaq with Xeons (eight way SMP 4GB RAM) Debian or FreeBSD
Sun (four way SPARC64 SMP 2GB RAM) running Solaris, but I could install Linux
Sgi octane2 (MIPS R14k 1GB RAM) IRIX
HP visualize J6700 (dual SMP PA-RISC64 4GB RAM) running Debian, could install HP-UX
HP precision book (PA-RISC32) running HP-UX, could install Linux or OpenBSD
Sun (SPARC64) running OpenBSD, could install Linux or Solaris
Plenty of boring x86 machines, some older PA-RISC32 junk, and probably other RISC boxen that I forgot about....

Send an email to
unixclan
REMOVE THIS IF YOU ARE NOT A BOT
@
gmail.com
If you think you can help me host an alternative compile farm.

Re:Virtualisation negates the need for a compile f

[Haeleth]

This is true.. how many operating systems are in wide use for most applications these days?

We have... Windows, MacOS, Linux, and BSD.

All of them in numerous different versions, and in the case of OS X, Linux, and BSD, running on a variety of hardware. (There's still PLENTY of PPC-based Macs around, for one.)

I spose there's still people working with Sun/Solaris and HP/UX and AIX

Damn right. More than you'd think, in fact.

but for the most part, open source devs care that it works on their stuff, and to heck with whatever else.

Do you consider this an attitude to be encouraged?

And even if you don't see a problem with it, what about those OS devs who do actually kind of like the idea of testing on a variety of hardware? There aren't many hobbyists who can afford to buy servers from HP and IBM.

The openSUSE Build Service

[apokryphos]

The openSUSE Build Service: http://opensuse.org/Build_Service (supporting Mandriva, Debian, openSUSE, SLED, SLES, Ubuntu, Fedora...).

Re:Virtualisation negates the need for a compile f

[vrai]

I spose there's still people working with Sun/Solaris

Yes, such as the entire banking industry and almost all it's associated software vendors. Admittedly there's been a move towards Solaris/x86 but there's still a huge market for UltraSparc machines; not all jobs can efficiently distributed across multiple machines and Intel architecture can't provide more than 16 cores. The Cell processor is attracting a lot of attention as a potential replacement for Sparc and requires specialist development machines. You can't really test your new Cell optimised uber-parallel pricing model on a four core Intel.

For most open source software you're completely correct - it'll never run on anything more exotic than a Core Duo. But if you're developing something other than desktop applications (e.g. programming languages, libraries, frameworks, etc) and you want your software to be used by the widest possible audience; you need to test it on as many architectures and operating systems as possible.

Usage stats?

[Orlando]

As a consequence, I expect many projects dropping support for some of the platforms they can't get access to.

Do we have any actual data on how popular the service was? I think this was a neat idea, but if it wasn't being used it won't be missed...

Power != POWER; Sparc != SPARC

[tepples]

QEMU won't do POWER The SF.net CompileFarm was not there to provide 'power'. It was there to provide access to different systems for compilation of your project. Anyone using it for 'power' was abusing it. cbreaker said POWER; you said "power". There is a difference between "power" and the POWER architecture, and QEMU doesn't emulate the 64-bit POWER architecture. There is also a difference between "spark" and the SPARC architecture, and QEMU's support for SPARC is still very immmature. (Source: QEMU Status)

To sum it all up: alternatives for SF Compile Farm

[sick_soul]

To sum it up, there are no complete alternatives for SF Compile Farm
at the moment, and it will be missed a lot.

The suggested alternatives can partially alleviate the problem:
http://www.testdrive.hp.com/
[FreeBSD, HP-UX, HP OpenVMS, HP Tru64 Unix,
Mandriva, Debian, RedHat]

http://www.blastwave.org/ [Solaris]

But a lot of stuff is left out (at least NetBSD, OpenBSD, Darwin,
Linux on POWER, AIX).

Please prove me wrong and provide links for alternatives to the CF for those
systems.

Re:To sum it all up: alternatives for SF Compile F

[tchuladdiass]

Even SF's compile farm wasn't all that great -- they mostly provided various x86 environments, or Linux on non-x86 hosts. The main thing I want to see in a compile farm is systems that I can't easily get ahold of -- a couple of Sun Sparc boxes, a couple flavors of IBM powerpc AIX systems, hpux (on both pa-risc and itanium), etc. Along with a good job control environment -- you supply the build and validation scripts, then the build gets run, tested and packaged on all the architectures.

Re:Virtualisation negates the need for a compile f

[Wdomburg]

Intel architecture can't provide more than 16 cores.

IBM sells a 64 core Intel based system.

The Cell processor is attracting a lot of attention as a potential replacement for Sparc and requires specialist development machines.

Unlikely. The Cell is PPC, not Sparc. And Sun already has their own highly parallel designs - Niagara (eights cores) and Rock (four cores with four processing engines each).

As much talk as there is about Cell's potential, I'm not convinced. It's not a particularly good general CPU - most of the die space is dedicated with SIMD instructions, which are only useful for a certain class of application. The most obvious market outside real-time video processing would be scientific applications, but the Cell throughput drops from a claimed 218 gigaflops to about 26 gigaflops when you put it in double percision mode (which also enables IEEE standard rounding). Still fairly impressive but you'll only reach that number if you're doing strictly vector math.

GCC Compile Farm

[guerby]

If you want to test your free (as in speech) software with recent GCC, there's a little farm (9 bi Pentium 3 1GHz) I help maintain:

http://gcc.gnu.org/wiki/CompileFarm

See "How to get involved" chapter to get an account.

Re:To sum it all up: alternatives for SF Compile F

[The Wicked Priest]

For NetBSD/Alpha, you might consider getting an account at freeshell.org.