Slashdot Mirror
Alternatives To SF.net's CompileFarm?
cronie writes "Not long ago, SourceForge.net announced the shutdown of the Compile Farm — a collection of computers running a wide variety of OSes, available for compiling and testing open source projects. SF.net stated their resources 'are best used at this time in improving other parts' of the service. I consider this sad news for the OSS community, because portability is one of the strengths of OSS, and not many of us have access to such a variety of platforms to compile and test our software on. As a consequence, I expect many projects dropping support for some of the platforms they can't get access to. Are there any sound alternatives with at least some popular OS/hardware combinations? Any plans to create one? (Perhaps Google or IBM might come up with something?)"
I consider this sad news for the OSS community, because portability is one of the strengths of OSS, and not many of us have access to such a variety of platforms to compile and test our software on.
Maybe the project has ended because that's not where the future of computing is headed. Maybe the future is something more like "write once, run anywhere".
The theory of relativity doesn't work right in Arkansas.
Gee, you'd think if a project was interesting/useful enough to be valuable on a second or nth platform, the users of that platform could supply some patches.
I've been porting things between win32 and linux for years with some BeOS and MacOS ports along the way. Not exactly rocket science, get a grip.
Naaahhhhh. Nice thought, but no computational utopia yet.
How about vendors supply compile farm gateways linked from SF.NET for use by SF members. Great way for hardware vendors to show off their new stuff to folks that might be inclined to buy or have influence in the purchase decision.
Kinda like a hands-on remote(?!) demo.
SciTechPulse. Geek News Netcast. Hot Polynesian Geek Chick Host Silulu.
I wonder if emulation of other hardware architectures would allow developers to try things out on their commodity machines? VMWare and Virtual PC do a good job for x86 emulation and there are many emulators for obsolete machines available so the question comes down to the time and effort required to implement new architectures. Maybe what could be practical is something along the lines of Transmeta's morphable instruction sets technology but with an extra layer of associated hardware (video, sound) emulation/translation.
Shh.
With the availability of VMWare, Xen, etc. you can have your own CompileFarm. Obviously it's not a good choice if you're trying to render an animated movie or similar, but for testing or compiling it should fulfill most of your needs.
I use VMWare Workstation and Virtual PC to do testing and whatnot, negating the need for multiple systems in my home office. I have, for example, Windows XP Pro, Windows 2000 Pro, OpenBSD, FreeBSD 5.5 and FreeBSD 6.2 all set up as seperate virtual systems on a single computer.
Who needs a compile farm when most of what we need can be run from a single moderately decent workstation?
Feed the need: Digitaladdiction.net
Posted By: wdavison
Date: 2007-02-16 00:13
Summary: Compile Farm News
As of 2007-02-08, SourceForge.net Compile Farm service has been officially discontinued.
Shutdown on Feb. 8, announcement on Feb. 16th?
With behavior like that, SourceForge can't be considered a safe location for important code. I'd suggest that it's time to get projects off SourceForge. Make offsite backups of anything important now.
Latest announcement from VA Software, which owns SourceForge:
VA Software Corp., whose software and online media are targeted for the open-source software community, said Thursday it named Scott E. Howe to its board of directors.
Howe is president of a division of digital marketing company aQuantive Inc.
"Scott's extensive knowledge of the media markets will be invaluable as we continue to focus on our core media assets and strive to secure alliances in the global competitive landscape," VA Software President and Chief Executive Ali Jenab said in a statement.
VA Software slipped a penny to close at $4.24 on the Nasdaq Stock Market.
If VA Software thinks they're now a "media company", it's time to get off SourceForge.
That's not much use for testing compiling on Solaris on SPARC64, or Tru64 on Alpha, etc...
VA Software owns Slashdot:
http://www.ostg.com/about/index.htm:
Ergo, VA Software is a media company.
Time to get off Slashdot.
Most projects are staffed by people using multiple platforms anyway and anyone coming along with a requirement to support some odd-ball OS might just get pulled in to do compiles and tests. For example, the SF project I work on is mainly staffed by Linux people with a few Windows and this project does not use the compile farm. Those using OSX just need to recompile and it works for them.
Engineering is the art of compromise.
Personally, I see less and less need for compiled and distributed software as broadband internet becomes ubiquitous and rich internet applications become more sophisticated. As it stands now, there is very little that traditional software does that can't be replicated on the web using the right technology. Software as a service is slowly becoming a reality and compiled software is soon to go the way of the dinosaurs.
Anthony Papillion
Advanced Data Concepts, Inc.
"Quality Custom Software and IT Services"
Can we start a community driven project similar to Compile Farm where people with systems contribute their system time in an anonymous fashion. Something like a p2p compilation.
Get your software packaged by Debian (which you probably want to do anyway), and it will get built on (currently) 15 architectures of GNU/Linux, along with 3 non-Linux architectures (kfreebsd-i386, kfreebsd-amd64, hurd-i386), with more popping up occasionally.
Finally a legit use......
Just upload your code to CPAN, and the CPAN Testers will test it for you. Easy!
My other car is first.
Well, I am happy to test my compilations on ppc-linux, ppc-darwin, i386-darwin, alpha-linux, hppa-hpux, mips-irix, ppc-aix, just to name of few. I can't do that with virtualization. (and yes, for large projects, the amount of porting to different platforms is non-trivial, and regular testing is required)
Anyway, the Compile Farm SF service was crappy and never realiably worked, so it's not a real loss.
Virtualisation can help you with different OSs on one platform. If you want to do multiplatform-compilation on software, than you have to emulate, and that will be a real pain, slow, and it will be the project developer who has to maintain all the target platforms. So, should every project admin on sourceforge get qemu, set up 15 diskimages and install operating systems for x86, ppc, sparc, cell, alpha, mips, ...? I think people will be quite busy installing operating systems and working around emulation trouble, but not develop any more than.
You can say, ok, Apple moved away from PPC, so we do not need anything but x86. Great. But wasn't the right to choose between alternatives one of Linux' key ideas? And wasn't the multiplatform-approach the main reason that the big money came to open-software from companies like e.g. IBM, who develop more than one platform and have been searching for a way to get their solutions work on all of them? I think if OSS looses its multiplatform idea, there won't be much left of the dynamic we saw during last years.
So, if sourceforge doesn't want, maybe a specialized project is needed? May be Debian, who is already bundling porting to all kinds of platforms? Maybe a Porting Group that hardware vendors could become member of, with the sole purpose to offer best possible build and testing environments to distribute clean binaries?
Lars.
http://www.testdrive.hp.com/
HP dude Bdale Garbee has said HP is delighted if people use testdrive to test their code on different architecture and OS combinations.
Hewlett-Packard provides a program called TestDrive, which allows you to access for free some interesting systems like HP-UX, Tru64 Unix, SuSE Linux, FreeBSD, and even OpenVMS! They also just integrated a Windows 2003 box ;-)
Link: http://www.testdrive.hp.com/
If you want to test/port code for (Open)Solaris, become a member of the Blastwave [blastwave.org] community.
Dennis maintains a farm of both i86pc and sparc platforms from Solaris 8 up to the latest Nevada build, and the membership is free to all interested parties.
In fact, I encourage you to do so at your earliest convenience.
Short of doing that, the best solution is to snag a 19" rack from ebay (they can be had for cheap these days), stick it in the basement if you have one, and load it up with cheap hardware from ebay (and Sun and hp systems are (*very* cheap nowdays, especially on ebay).
Buy a cheap PC in Walmart for your needs.
This is true.. how many operating systems are in wide use for most applications these days?
We have... Windows, MacOS, Linux, and BSD.
Anyone else?
I spose there's still people working with Sun/Solaris and HP/UX and AIX, but for the most part, open source devs care that it works on their stuff, and to heck with whatever else.
"Champagne for my real friends - and real pain for my sham friends!" http://ericblade.postalboard.com/
I can donate hardware and sysadmin man-hours, but I need either space, electricity, and bandwidth or money (which can obviously get me space, power, and bandwidth). I have lots of platforms just sitting in storage, and I plan to ebay most of it unless someone can get help for an interesting and useful project like this. The architectures I can provide are as follows:
4x Sgi o2 (MIPS both R10k and r5k) currently running IRIX, but I could install Linux, NetBSD or OpenBSD
Compaq with Xeons (eight way SMP 4GB RAM) Debian or FreeBSD
Sun (four way SPARC64 SMP 2GB RAM) running Solaris, but I could install Linux
Sgi octane2 (MIPS R14k 1GB RAM) IRIX
HP visualize J6700 (dual SMP PA-RISC64 4GB RAM) running Debian, could install HP-UX
HP precision book (PA-RISC32) running HP-UX, could install Linux or OpenBSD
Sun (SPARC64) running OpenBSD, could install Linux or Solaris
Plenty of boring x86 machines, some older PA-RISC32 junk, and probably other RISC boxen that I forgot about....
Send an email to
unixclan
REMOVE THIS IF YOU ARE NOT A BOT
@
gmail.com
If you think you can help me host an alternative compile farm.
------ Take away the right to say fuck and you take away the right to say fuck the government.
You talk about BSD as if there's only one BSD. And there are other Unixes apart from linux and BSD, such as the proprietory from big-name vendors such as Sun, IBM, and HP.
Also FatPhil on SoylentNews, id 863
And even if you don't see a problem with it, what about those OS devs who do actually kind of like the idea of testing on a variety of hardware? There aren't many hobbyists who can afford to buy servers from HP and IBM.
I used the Compile Farm for Apple Mac OS X.
Now where do I go?
There's a tool from RedHat (at least, available for fedora) called mock which makes the build environemnt completely separate from the local system (as long as the binaries are compatible of course) that might help considerably in separating the build environment from the local system.
http://fedoraproject.org/wiki/Projects/Mock
You might want to look into HP's TestDrive program. You get access to wide range of OS's, on x86, Itanium and PA-RISC. Sign up, log in, and play.
IIRC, it's not quite such a range of hardware as SF provided, but it is a wider range of OS's on the hardware they do provide.
This post will enter the public domain 70 years after my death, unless Disney buys another extension.
The openSUSE Build Service: http://opensuse.org/Build_Service (supporting Mandriva, Debian, openSUSE, SLED, SLES, Ubuntu, Fedora...).
For most open source software you're completely correct - it'll never run on anything more exotic than a Core Duo. But if you're developing something other than desktop applications (e.g. programming languages, libraries, frameworks, etc) and you want your software to be used by the widest possible audience; you need to test it on as many architectures and operating systems as possible.
Big deal. My advice to people is to start moving off those platforms ASAP and not bother trying to get new stuff to work on it.
;) ).
SPARC's on death row and HP buried the Alpha (poor thing was still kicking and screaming
People who pick "one vendor" platforms should be well used to paying lots of money for anything.
Anyway, the only one who should build an expensive SPARC compile farm should be Sun. It's crazy for anyone else to do so. Are people going to suggest some company/organization buys lots of _expensive_ Intel "Itanic" servers just so people making _free_ software can test their software for _free_?
That's almost like expecting someone to buy a bunch of Bugatti Veyrons so that people around the world can learn to make 3rd party add ons for it for free.
Face it, if you buy a "Bugatti Veyron", don't complain when you have to go back to the vendor and their bunch of "approved partners" for practically everything.
As a consequence, I expect many projects dropping support for some of the platforms they can't get access to.
Do we have any actual data on how popular the service was? I think this was a neat idea, but if it wasn't being used it won't be missed...
-= This is a self-referential sig =-
portability a strength in oss? don't make me laugh.. most of it is linux specific. :-/
it still sucks that that compile farm gets shutdown - i didn't even know it existed until now
Considering how much interest SourceForge's CompileFarm project had generated up until now, it seems likely that a larger company, like Google for instance, would very soon become aware of the wide ranging attractiveness of such a cross-platform testing service. Keeping in mind the broad readership Slashdot enjoys, I find myself wondering how far away we might now be from a larger company, with the processing power and bandwidth to spare, offering us a free cross-compatibility testing forum that obeys the majority of different OS coding factions' general acceptance.
you do realize, one nine allows 36.5 days of downtime a year?
every day http://en.wikipedia.org/wiki/Special:Random
This doesn't solve the problem facing developers of ports to Windows OS. For development of ports of free software to proprietary operating systems that are not freeware, performing a day job for several hours of overtime in order to buy an OS license for each developer does require a massive time investment. How many hours of flipping burgers is Windows Vista Ultimate (or an MSDN subscription) worth to you? Besides, a lot of proprietary operating systems have EULA restrictions against virtualization at all, such as Mac OS X. Furthermore, virtual machines for CPU architectures other than Intel x86 tend to run unacceptably slow on some machines because fewer Free or freeware emulators tend to use dynamic recompilation.
[1] Throughout this comment, "freeware" means "lawfully obtainable at marginal cost", including popular free software.
This malicious post should not be modded down but DELETED.
Bots do not care if the poster was a troll.
The anonymous coward also posted the email address in the title.
Now also my previous post should be deleted. Funny and sad.
To sum it up, there are no complete alternatives for SF Compile Farm
at the moment, and it will be missed a lot.
The suggested alternatives can partially alleviate the problem:
http://www.testdrive.hp.com/
[FreeBSD, HP-UX, HP OpenVMS, HP Tru64 Unix,
Mandriva, Debian, RedHat]
http://www.blastwave.org/ [Solaris]
But a lot of stuff is left out (at least NetBSD, OpenBSD, Darwin,
Linux on POWER, AIX).
Please prove me wrong and provide links for alternatives to the CF for those
systems.
Unix revenue was about five billion dollars of the fifteen billion dollar server market in the last quarter of 2006. Yes, Virginia, there are still people working with Solaris, HP/UX and AIX.
Even SF's compile farm wasn't all that great -- they mostly provided various x86 environments, or Linux on non-x86 hosts. The main thing I want to see in a compile farm is systems that I can't easily get ahold of -- a couple of Sun Sparc boxes, a couple flavors of IBM powerpc AIX systems, hpux (on both pa-risc and itanium), etc. Along with a good job control environment -- you supply the build and validation scripts, then the build gets run, tested and packaged on all the architectures.
Intel architecture can't provide more than 16 cores.
IBM sells a 64 core Intel based system.
The Cell processor is attracting a lot of attention as a potential replacement for Sparc and requires specialist development machines.
Unlikely. The Cell is PPC, not Sparc. And Sun already has their own highly parallel designs - Niagara (eights cores) and Rock (four cores with four processing engines each).
As much talk as there is about Cell's potential, I'm not convinced. It's not a particularly good general CPU - most of the die space is dedicated with SIMD instructions, which are only useful for a certain class of application. The most obvious market outside real-time video processing would be scientific applications, but the Cell throughput drops from a claimed 218 gigaflops to about 26 gigaflops when you put it in double percision mode (which also enables IEEE standard rounding). Still fairly impressive but you'll only reach that number if you're doing strictly vector math.
How does this help for commercial software or non-x86 platforms? Its a start but hardly a finish.
Troller cannot just be modded down. Because the user obviously did not want the address posted for security, the post should be deleted in entirety.
You know, a massive free source control system does sound like the sort of thing Google would tackle. I don't know about the multiplatform compile angle, but the other basic aspects of a system like SF seems to be right up their alley.
Slashdot quality declines as the number of hot grits posts decreases. - Provolt's Law, Apr-09-2005
If you want to test your free (as in speech) software with recent GCC, there's a little farm (9 bi Pentium 3 1GHz) I help maintain:
http://gcc.gnu.org/wiki/CompileFarm
See "How to get involved" chapter to get an account.
The single reason I have ever used SF's compilefarm is to test my code on 64 bit architecture. For testing on different distros, I indeed do it all locally (using chroot environments). But for testing x86_64, if you don't happen to have a 64 bit CPU, VMWare is not going to help.
Using the compile farm to do regular automatic compiles of the latest code (if there are any changes) was one of the most usefull features of SF. *sigh*
If you want to cater to 99.% of users, you target Win32, Linux, and OS X. Xcode does x86+PPC for you without any work so that doesnt matter. Windows and Linux are trivial to throw into a VM, which is handy.
So you setup an Intel Mac, VM Win/Linux, write Python, Perl, PHP, or C, to minimize the testing needed and you're done. Add another 3-5 flavors of Linux because the various distributions are complete assh*les and can't standardize on all the libraries of course, so your code won't run on Redhat and SUSE without tweaks. Or just forget Linux and be happy with 99%.
Back even 10 years ago that list would have been 20 systems, not 3.
- Adam L. Beberg - The Cosm Project - http://www.mithral.com/
I used to use SourceForge's Compile Farm (in addition to HP's Test Drive) to test sigsafe. I need to write assembly for the cross-product of supported processors and operating systems. Without the ability to log in, compile, run my automated tests, and use a debugger, I can't support a platform. This decision means I'll have to drop sparc support. It's a shame - I learned a lot from writing assembly for these different platforms.
For my purposes, I don't care how much market share a particular platform has. Having a diverse build farm is useful for testing. Solaris will flag errors that Linux won't, and vice-versa. MSVC++ and gcc generate warnings for different problems. SPARC, PPC, Alpha, and Itanium processors catch things (particularly alignment errors) that x86 won't.
Why not build your own? Old PPCs, SPARCs, Alphas, and Intels are cheap on eBay. The software to create your own compile farm is readily available (and in the case of Buildbot, decentralized).
Having your own farm means you don't compete for resources, and you're not beholden to some assh^W other organization's business case for shutting it down.
You bring in interesting point. I prefer having programs work on many architectures but I don't expect some company to build a server farm for open source programs. Frankly, the open source community should pool together and create a community farm (Compile@Home or some crap like that) and/or have the hardware companies create farms. If Sun want's more applications ported to their stuff, set up a farm for the opensource community. The thing with SF is that you only need to go to SF for all the platforms. A log on for each architecture is too much hassle. As for the dying platforms, no comment. Don't know enough about the supposed deaths of platforms to make comment.
I use it at work for the same reasons (development machines are not all release machines)
As a bonus, you can run both release and debug variants.
To make this really work, you have to make it easy for people to set up. This means installing and/or building the minimum of libraries.
Where will I leave my penguins when I'm away from home?
For NetBSD/Alpha, you might consider getting an account at freeshell.org.
Share and Enjoy: 09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
Really? I couldn't find any on their website and have never heard of IA-32/64 architecture being pushed that far. According to their products site only their POWER based machines are 64-way, their Intel/AMD units are four socket (16 cores at most).
The Cell is being looked at by large banks and they are sufficiently interested to set up specialist teams to see how the architecture can be best used. The Niagara is a toy with laughable floating point performance; the example unit we got sent ended up as a foot-rest. Rock is much more useful (each of the four cores having a dedicated FPU) but the IC design has only just been finalised and the first servers won't be shipping for another year. There's also the problem that it's an extension of the UltraSparc architecture rather than "cool" and new like the Cell. Sadly this is often the deciding factor with IT managers at the very large banks.
The Cell's designed, with one PPE and a number of SPEs is very suited to a number of pricing calculations. Many instruments are priced in an iterative manner or by use of monte-carlo. The lack of DMA for the SPEs is not really a problem in these cases as the inputs for each iteration/simulation don't change that much and (in most cases) could be wedged in to the 256K available to each element. You're assertion that Cell is best deployed only for a limited task set is correct, but banking contains such tasks and the Cell appears to be well suited to the role.
Amazon's ec2 is not free, but it's not expensive either. a compile farm alternative is the perfect use for ec2. you pay only for the cpu you need. so you don't have to maintain machines, virtual or otherwise, between milestones (for example).
in this age of communication i'm just not getting through
The people who would care if it runs on several platforms are probably not developing the software as a hobby, anyway. I'm not saying it should be encouraged, in fact, as portable as possible is always the best way to do things, and that way you don't have to change things so much when things on your own operating system change. However, even when it was an available and free resource from SourceForge, apparently not enough people used it to make it worth it to keep going. I'm not saying that that's how it should be, but that that's how it IS.
The general attitude is "it's not my platform, if you want it, you port it."
"Champagne for my real friends - and real pain for my sham friends!" http://ericblade.postalboard.com/
Yes, but the other issue is: How to tell when a bug is actually cause by the VM rather than the code itself. VM's do a pretty job, but they're not perfect.
Most open source software development is done on Linux and *BSD, which support the POSIX standard. There's no good reason why open source software couldn't run on almost every POSIX system under the sun, except that you need a testing ground to find and eliminate a few quirks resulting from unspecified behaviour.
http://www.xensource.com/xen/
I mod down anyone who says "I will be modded down for this", regardless of the rest of their comment
You said "I consider this sad news for the OSS community." Good then you write the check.
use xen or vmware.
I'm a Linux admin and a C++ dev. so given the choice between a free commercial product and an opensource alternative, I have to choose the free commercial product.
They're using their grammar skills there.
QEMU supports MIPS, Arm, and SPARC. You could at least test under Linux and probably BSD for all of them.
See my blog http://ilovecookes.blogspot.com/ for light hearted technical information.
The NMI Build and Test Facility at UW-Madison might be useful. They keep a fleet of Linux (x86, x86-64, IA64, and PPC, on most of the major distributions), Windows, MacOS X, HP-UX, Tru64, FreeBSD (only x86 for now) and SPARC Solaris running to do nightly build and test runs of a number of different projects.
http://nmi.cs.wisc.edu/files/nmi_lisa2006.pdf
Really? I couldn't find any on their website and have never heard of IA-32/64 architecture being pushed that far. According to their products site only their POWER based machines are 64-way, their Intel/AMD units are four socket (16 cores at most).
:)
:)
Try the IBM System x3950. See also the Unisys ES7000/one.
To be fair, the aging bus archiecture of the current Xeons is definitely a hinderance onec you move beyond commodity servers. In terms of widely available machines, you're still generally looking at 16 core (four quad core Xeon or eight dual core Opterons). Howver, Intel is set to debut CSI (Common System Interface) to replace the front side bus and AMD is set to transition to Hypertransport 3.0, both of which will support at least sixteen quad core CPUs.
The Cell is being looked at by large banks and they are sufficiently interested to set up specialist teams to see how the architecture can be best used.
I never said it wasn't being looked at. So was the Itanium.
The Niagara is a toy with laughable floating point performance; the example unit we got sent ended up as a foot-rest.
A product isn't suitable for applications it wasn't designed for? Shock.
Rock is much more useful (each of the four cores having a dedicated FPU) but the IC design has only just been finalised and the first servers won't be shipping for another year.
There's also a FPU per-core in the UltraSparc T2, which should be available soon. Neither of the Niagara CPUs are a good replacement for the mainstream UltraSparc line, though, since they suffer from the same "weakness" as the Cell - they're good for the subset of problems they're deigned to solve, but mediocre to bad at things they're not designed for.
There's also the problem that it's an extension of the UltraSparc architecture rather than "cool" and new like the Cell. Sadly this is often the deciding factor with IT managers at the very large banks.
And the Cell is just an extension of the PowerPC architecture (yes, I know the SPE implements a new ISA, but that sort of nuance is lost on pointy-hairs) rather than "cool" and new like the Niagara. I mean, come on, the UltraSparc T2 can handle 64 simultaneous threads and the Cell can handle a piddling eight. Big numbers sell just as well as novelty (often better, since novelty comes with free "new and scary").
The Cell's designed, with one PPE and a number of SPEs is very suited to a number of pricing calculations. Many instruments are priced in an iterative manner or by use of monte-carlo. The lack of DMA for the SPEs is not really a problem in these cases as the inputs for each iteration/simulation don't change that much and (in most cases) could be wedged in to the 256K available to each element. You're assertion that Cell is best deployed only for a limited task set is correct, but banking contains such tasks and the Cell appears to be well suited to the role.
Ah, I read "The Cell processor is attracting a lot of attention as a potential replacement for Sparc and requires specialist development machines." as more of a general statement rather than being applicable specifically to banking. Would explain some of the disagreement.
I'm not intimately familiar with financial software, but I was under the impression fixed point arithmetic is generally prefered. I have no idea how the Cell would perform for that makes the performance hit of IEEE-754 mode out of the picture.
Obviously this somewhat depends on your definition of providing. If you mean providing as in the limit of slapping cores on one board and killing your FSB, then yes 16 cores is possibly the current limit. But if you mean scaling with a NUMA architecure and still getting performance then no. http://www.sgi.com/products/servers/altix/4000/. Take a play with a real machine. Yes the latency is somewhat high (1000ns c.f. maybe 2-300ns) but you've still got 3200Mbytes/s of bandwidth (per direction per link).
jh