Slashdot Mirror
Does Linux Need Another Commercial Compiler?
Lurks asks: "My company, Codeplay, is set up to develop new and innovative compiler technology for the games industry. Our C compiler, VectorC, is a
cutting edge vectorizing compiler aimed at games and multimedia
applications that demand high performance generally through hand-optimized assembly. I'm writing to ask the
burning question on our minds, is it worth porting VectorC to Linux?
In fact, we're already targeting Linux as part of the PlayStation 2
version albeit not generating x86 code of course. A Linux port would see us converge this work with our Win32 compiler and such an
undertaking would certainly be popular with our Linux loving techies!
One caveat worth mentioning now is that the current version of VectorC is plain C only. 2.0 with full C++ compatibility is due early next year."
"Of course, commercial realities will rear their ugly head and we must consider that Linux isn't perhaps an obvious choice for a compiler aimed at games and multimedia applications. Given the certain degree of hostility towards commercial closed-source products of this nature, perhaps the idea should be consigned to the pet-project back burner for the future rather than rolled out as a commercial product?"
Do you mean converting your compiler to run under linux, or making it able to compile code FOR linux (and if so, under what platform).
IF it's a playstation/games developmetn compiler.... then the only reason to port it to linux is if you want to offer linux as a development environment for playstation games. Is it? What's the benefit?
For those who don't know much about it, VectorC is a compiler for targetting fairly specific types of system. Namely those with a bit of SIMD like your P3/P4/etc ,or small-scale parallelism like the PS2 (is there an XBox-specific version to help with shaders yet? dunno).
;o) Why? Well, I just don't see a huge market of Linux users with the type of processors which get the advantage from that kind of compiling. Cross-compiling for PS2 yeah, we do that in my office (though not from Linux). But one of the big selling points of Linux is that it works on lower-end processors, which don't have SIMD.
Anyway, I can see this not going down hugely well on Linux. Apart from the commercial thing, which is a big turnoff to many of the free-as-in-beer users
Anyway, its a good compiler, if you need it. Just think about whether you do or not...
Game dev and music blog
Today, there are 2 compilers that are well known - GCC which is the default and most used on Linux, and Intel's ICC which is a commercial, but there is a free non-commercial version available from Intel. So far I have heard mixed reports from people about ICC effiency in terms of code generated, speed of binaries, size of binaries etc (slashdot users who use ICC - please post your conclusions).
Now - the next big compiler that will come out (commercially) is from Borland. Early reports from various testers suggest their C/C++ compiler is kicking both ICC and GCC in the ass, but again - I belive it when I see the numbers, although Borland got a reputation of isssuing quite fast compilers..
So - if you decide to release a compiler, you'll need to think about 3 points:
1. GCC compatible - you'll need it if you want to be used by open source users OR to allow developers to move their apps which used GCC to your compiler.
2. A free version (free as a beer) - in order to be really accepted and widely used by Linux users, you'll need to issue a free version for the developers to use. Intel learned this quite since the beginning that if they want their compiler to be accepted by the Linux users - they'll need to release a free version. Borland is rumored to release a free version also with their upcoming C/C++ compiler (command line version, not the GUI)
3. Competition - well, not much to say here, but you got companies like: Borland, Intel, and the GNU GCC, along with the Portland group's compiler, code warrior (Metrowerks) - plenty of competition. do you really want to get in?
Hetz (Heunique)
Porting this to linux seems to be a good idea, but the inherent problem is your business model. Who is your target audience? Are you targeting corporations such as lokisoft who will use your compiler to port/create games for linux?
(side note- the lokisoft page is down, i dunno if that is a fluke, if i have the wrong URL, or if they've packed up and left)
The problem is that if you target corp.'s like loki, you may not be able to sell enough units, or whatever, to justify the cost of porting. These linux gaming companies seem to fold faster than omlettes at waffle house.
If on the other hand, you just ported it and released it at random into the linux/OSS community, you would be doing the community a favor, and independant cells of programmers could attempt to port/write games for linux.
The problem with this solution is also the cost: If you release it open source for linux, you would be somewhat of a hero, or philanthropist, to the OSS community; however, you may not be able to justify the cost of porting it, if your idea is to make money by porting to linux.
I guess it depends on what time frame you think you can port it to linux in - if it would take you and your team an extra two days of programming, it may be worth it, as both a PR move and a gift to the OSS community. However, if it will take extra months of coding, just bear in mind that philanthropy doesn't pay bills.
Don't mean to be cynical, but you have to consider each decision as it relates to the almighty dollar.
It sounds as if you're doing fantastic work, though, keep it up.
sig?
You're obviously quite right. We get a surprising amount of interest from the scientific/academic community. High tech compilers that vectorize are nothing new in this area, but generally they're Fortran compilers for mainframes etc.
I'm guessing that with the power of desktops these days and the prevalence of C/C++, micro based scientific computing is looking more attractive.
It's quite spooky to see all these high energy research labs (Eek!) in our web logs. I wonder how many of these people understand the precision issues with most vector units out there.
Perhaps the Pentium 4 will become a standard choice for scientific computing because of the double precision cable vector unit. Just as well we put a lot of effort into SSE2.
Mat 'Lurks' Bettinson, Codeplay Ltd.
Vectorc guys, I get the impression you don't know what you are producing :)
[time to complete]
Latest (relased) Borland compiler, full optimizations on: 142
Hand optimized c code (according to athlon optimization guide) with latest (relased)borland compiler, full optimizations on: 100 (ref)
Vectorc 1.1 demo single precision 3dnow!, optimization hints: 46
Vectorc 1.1 demo, double precision x87: 48
Real code snippet from a real application. Do I have to say more? Normally one wouldn't use a borland suite for high performance critical code, but not everyone can afford to have a high-tech scientific compiler.
Gentlemen, you can't fight in here, this is the War Room!
IF Linux ever gains the popularity we all hope it will, it will stop being the land of all open source. Games will not be open sourced and other really good products that people worked hard enough to earn money from won't be either. Realize that. It's already happening. People need to feed their families, so they sell their hard work.
This guy here is selling a product on Win32. I assume they are making money. The question is simply, if we provide Linux users the option to use our software on that platform, will anyone buy it? They are doing a good thing here. They are taking an application that presumably gives the Win32 platform some advantage and offering to make it available to Linux users. I hate to say this, but if MS Office was ported to Linux, not only would it sell, but the platfom would become more popular. Just because you as an individual flee from closed source, does not mean that a company should not market a closed source product to the Linux platform.
But I have a small warning for you: If your compiler really does produce awesomely better code in that particular area, then if that area becomes popular with the GCC crowd, you are going to see alot of work spent on optimization. In other words, your product may be a spur to make GCC competitive with VectorC.
And I would also point out that I think that if VectorC, available for Linux/someArch, does NOT stir the GCC developers to improve, then your specific market doesn't exist on Linux.
Really sucks: Either you eventually face competition with GCC, or your product bombs on Linux.
"Avast! Prepare for the rodgering!" THWACK! "Arrr.. me nards.."
I am developing an extremely authentic and highly innovative gaming technology for the information superhighway called SuperDuperGameAccelerator. It's really cool. It will accelerate games and cook your dinner too. I was wondering if it's worth porting it to Linux. In fact I'm already porting it to Linux, I just wanted to announce it to the world by posting on slashdot. You know, after the .com crash we can't get any more funding, so we could use all the free advertizing we could find. And I figured slashdot editors are too stupid to recognize the thinly disguised commercial...
___
If you think big enough, you'll never have to do it.
I do high-performance computing, and I'd love to be able to try out Vector-C on some of our P-4 and Alpha Linux clusters, if I could. Right now, we use Intel's icc or gcc on x86 and Compaq's ccc or gcc on alpha, respectively. Pretty soon we are going to be looking at Itanium as well. Some of the time we are hand-hacking assembly just like the game programmers are, which is kind of sad; we would rather be compiling C. What Mat Bettinson said is definitely the case: "micro based scientific computing is looking more attractive."
Despite what some ppl here are saying, it's not an issue if it can't compile the kernel, or if it's not 100% gcc compatible, because most of the things the high-performance computing applications I've seen don't need to spend a whole lot of time in the kernel. However, you do have to make it work with both 2.1 and 2.2 glibc (please please please). The hacks we came up with to make icc work on our glibc-2.2 RH7 boxes are ugly and fragile.
Language issues: C++ is almost never a big deal in HPC, but C/FORTRAN support is great. Having at least partial C99 support is best because then you get float *restrict foo, et al. Also, remember that not all HPC codes are fp. Some of us write integer intensive codes and/or memory intensive codes.
It's not an issue if it's not free-beer or free-software, because research grants will probably be happy to pay reasonable amounts for it -- maybe a couple hundred bucks, say -- but you have to remember that Intel is giving icc betas away for basically nothing, so you can't charge too much. This is not a troll, just trying to be realistic here.
Disclaimer: I am not speaking for my employers. I am not a person who gets to decide how grant money is spent (yet). These are just my opinions.
It really depends on how you want to work it. Binary only releases are popular in certain Linux segments that don't want to hasle with compiling. The problem is of course you're only a kernal update away from having a program that doesn't work.
I think there are aspects that of the Linux Universe that would benifit from a well done multimedia compiler. DVD players, or general Home Theater Computer enviroments, Linux Set Top boxes. etc. Games of course would welcome this too.
But the real problem of course is how to you balance making a comcerial distro for Linux that is okay from the Opensource,and GNU communities perspective.
There are a couple ways to go. You can go the QT way, and have a free and comercial version. The hope being that the various projects will use your libs and compiler, and thus it would be popular for comercial aspects that would actually pay you.
Second would be binary only distro. Not as popular mind you because the person who puts together the Binary Distro (Which I assume would be a person who bought the package) may bit off more than they wanted to chew keeping things up to date. In order for this to work in opensource you'd need a lot of compiler directives. The idea being you could compile the project with GCC, it just wouldn't be as efficiant.
Third would be a hybrid method. Some sort pre-compile on the closed stuff, and an included client compiler that would bring everything system dependent together. Perhaps even making the compiler and libs free to distro, but the development enviroment closed.
My person preference would be number one. Anyone know how well QT does?
I'm sure plenty of people will disagree with my conclusions, but I believe them to be sound. If you want to be successful on Linux (or any other free OS), you need to be libre/free.
That means making the source code available under a GPL-like license.
However, you can still protect your market at the same time. Just make it a requirement that anything compiled with the libre version of VectorC, or a derivative, also be licensed under the GPL or any of the similar licenses recognized by the OSI.
Then dual-license the compiler so that anyone who pays for the commercial license is free to do whatever they want with the resultant binaries, no licensing restrictions.
That way you can contribute to the community by providing a libre compiler, benefit from the community who will likely contribute bug-fixes and enhancements, and still make money from the people who would be willing to pay in the first place.
When information is power, privacy is freedom.
Well, it's a bit more complex than that isn't it. Us porting to a platform involves the following kind of issues;
If we're not doing well in the above, then we'd not consider the platform a priority. Unless, of course, someone wants to give us money to do it. That's not out of the question where it's a proprietary platform such as a games console or, to an extent, the Apple Mac.
Apple is a good target on one level because the vector unit in the G4 is the best vector unit in any consumer CPU. There's also games development and, more importantly on that platform, there's a lot of performance-sensitive multimedia application development. Apple like to crow about how good their CPU is, so one would have thought they'd like a tool on their platform to prove it.
Regarding your other suggestions,maybe down the line we'll be looking at more business orientated high performance computing but that's not why Codeplay started up. From experience we have our work cut out talking to the various corporate players coming from an unknown company with 'play' in their name and 'games' on their web site. I think to be successful there we'd have to spin-off some high performance computing company or something. And definately no chins on the web site :)
Linux is a special case. For all other platforms, we take a straight business viewpoint based on the numbers. To be frank, we've got a LOT of work/platforms on our plate as it is without finding new platforms unless they were going to pay.
The Playstation 2 tools will pay the bills. Linux would just be 'cool', so long as it didn't cost us significant money and didn't generate bad PR. That's my view at the minute.
Mat 'Lurks' Bettinson, Codeplay Ltd.
Implementing C++ properly natively is a requirement so that we can attempt to do the sorts of things VectorC does which no other compiler does. Simply put, if we used someone else's translator - we'd have C++ capability but almost certainly would be no faster than other C++ compilers.
I assure you, we are. The work is almost complete, what remains is the significant internal testing and bug fixing which we estimate will require the rest of the year.
We have a fridge full of Redbull, believe me we can do it. :)
Mat 'Lurks' Bettinson, Codeplay Ltd.
There is another compiler worth considering. Metrowerks has an auto vectorizing compiler for the Intel platform, Code Warrior, which both runs under and targets Linux (Intel & PowerPC). Code Warrior also targets Windows, Mac Classic, OS X, GAMECUBE, PS2 and most other game platforms. Benchmarks can be found on the Metrowerks site (http://www.metrowerks.com), but they're a bit old. We'd be happy to provide up-to-date benchmarks for anyone who's interested.
The real strength, however, is CodeWarrior's optimization and code generation.
I'm biased, though, as I'm a compiler engineer (PowerPC) for Metrowerks, so take all of this with a grain of salt.
--Doug
Editor : Adam Barker
Yes, I can't spell so I need one of those editor types.
...when I spoke to your sales rep at GDC and Siggraph, he indicated that C++ wasn't ready, as you said. He also didn't think your symbol information would be compatible with VTune or the other standard Windows profilers and couldn't tell me of any test suites that your compiler had passed, though he named a couple which were "close." I wouldn't use a compiler which doesn't pass basic conformance testing, and I certainly wouldn't take on a new compiler if I don't know that I can profile its output to prove that its working.
When I tried creating some simple code and looking at the disassembly on the sample machine at Siggraph, the compiler choked on some valid code (it seemed confused by the critical 'volatile' keyword), and the assembly generated was extremely naive about cache use and couldn't even hoist redundant operations out of loop operations.
So far as I can see, I'm supposed to dump my compiler for something that lets me use half a dozen instructions I can get with inline assembly or Intel's _free_ compiler, where Windows is concerned.
Shouldn't you finish your tools and make them work on one platform before you go trying to pitch them on others? Have your tools really advanced so far in the last few months that you're ready to split resources?
Yeah, this'll probably require doing much of the porting work before determining whether there's a market for it. It's called R&D, and it does sometimes lead to dead-ends. Deal.
A few months later, TotalView was released for Linux x86. Guess which is one of our biggest sellers?
I went to SC2000 (the supercomputing tradeshow) last year. Almost everybody I talked was running MPI on Linux on PCs. Linux has made huge inroads in the scientific community working on ahem, clusters, of Linux boxes. It's actually quite amazing, consider the Linux market in this segment was close to nil a few years ago. Something about cash-strapped institutions not having to pay high OS licensing fees...
Anyway, it goes to show you that even though there is a free alternative (gdb/ddd) you can sell tools for Linux if you do things the free things can't, or do it better.
I can explanate how to administrate your network. You must configurate and segmentate it, so it can computate.