Slashdot Mirror

"The open-source compiler that started as a GCC fork"? LLVM is not a compiler. It is a code generator, optimizer and virtual machine, usable as a compiler back-end. It later added a gcc-based front end.

Also, Apple is currently driving development of an alternate BSD-licensed front end named clang.

I predict that LLVM and HLVM will gain steam. People are going to realize that this pair of abstractions is cleaner, leaner, and meaner than the current virtual machine + language + API way of doing things characterized by Java and .NET. The fact that a GPU can be used as a processor transparently where appropriate, just the way Apple already has with LLVM, is going to start the rethink that was cut short by Java and .NET's fiascoes of ownership or patents. They'll also start making development in compiled languages easier.

This will be the open source response to the blurring lines between CPU and GPU task-wise, as the vector computing tasks could be done much quicker on the GPU based on the advances of LLVM, and applications will benefit transparently. It will be very cool.

I've seen a number of projects, from VMs to stuff similar to this project, roll their own JIT solutions all the time, causing it to work well only on a limited number of platforms with a limited number of optimizations. Hopefully if they go this route they'll use something like LLVM and not reinvent the wheel yet again.

I wonder how hardcore C developers will react when/if Apple moves away from GCC.

Likely many will dance in the streets. /me points at clang

Nothing really prevents you from using a "recompiler" at runtime with a C based executable. For example LLVM.

Who, in their right mind, would devote thousands of development hours cobbling something together, then cast it into the wind where basement developers use "what they want, and [get] rid of what they don't?"

Hmm, let's see...

• Apple
• Adobe
• Cray

And of course the usual suspects like Sun and IBM.

Free Software can most definitely be an important part of a business strategy. For example, the company I work for uses it to leverage testing resources of the community. We also get bug fixes back from the community. We think it makes a lot of sense for a large community to share core development responsibility, the sort of stuff you find in university textbooks that is not proprietary in any way.

In the future, companies aren't going to make money selling operating systems, word processors or basic compiler implementations. They're going to make money modifying the OS to run well on custom hardware, selling plugins to do fancy document formatting and developing new compiler optimizations that make all of this run well on their proprietary computer system.

LLVM is the one to watch. Currently, llvm-gcc uses GCC's front end but Apple, a major funder of LLVM, have recently released the source to clang, their C compiler. No GCC in sight. http://llvm.org/ http://heisenbug.blogspot.com/2007/07/compiler-with-good-clang.html

This is the compiler project I get excited about. Hopefully its progress will continue rapidly advancing. From what I understand, unlike GCC and PCC, extending llvm is straight forward, much easier to understand, and just code code rather than decades of retrofit as is GCC.

Take that with a grain of salt as this is not a first hand account.

Maybe LLMV (http://llvm.org) will be of interest to OpenBSD folks.

Highly BSD compatible. Advanced.

True-Freedom BSD-Style License (no Communistic-License like GPL).
http://llvm.org/releases/1.3/LICENSE.TXT

While LLVM is currently being developed using GCC it's intended to stand alone at some point.

From the LLVM web site: http://llvm.org./

Low Level Virtual Machine (LLVM) is:

      1.

            A compilation strategy designed to enable effective program optimization across the entire lifetime of a program. LLVM supports effective optimization at compile time, link-time (particularly interprocedural), run-time and offline (i.e., after software is installed), while remaining transparent to developers and maintaining compatibility with existing build scripts.
      2.

            A virtual instruction set - LLVM is a low-level object code representation that uses simple RISC-like instructions, but provides rich, language-independent, type information and dataflow (SSA) information about operands. This combination enables sophisticated transformations on object code, while remaining light-weight enough to be attached to the executable. This combination is key to allowing link-time, run-time, and offline transformations.
      3.

            A compiler infrastructure - LLVM is also a collection of source code that implements the language and compilation strategy. The primary components of the LLVM infrastructure are a GCC-based C & C++ front-end, a link-time optimization framework with a growing set of global and interprocedural analyses and transformations, static back-ends for the X86, X86-64, PowerPC 32/64, ARM, Thumb, IA-64, Alpha and SPARC architectures, a back-end which emits portable C code, and a Just-In-Time compiler for X86, X86-64, PowerPC 32/64 processors.
      4.

            LLVM does not imply things that you would expect from a high-level virtual machine. It does not require garbage collection or run-time code generation (In fact, LLVM makes a great static compiler!). Note that optional LLVM components can be used to build high-level virtual machines and other systems that need these services.

LLVM is a robust system, particularly well suited for developing new mid-level language-independent analyses and optimizations of all sorts, including those that require extensive interprocedural analysis. LLVM is also a great target for front-end development for conventional or research programming languages, including those which require compile-time, link-time, or run-time optimization for effective implementation, proper tail calls or garbage collection. We have an incomplete list of projects which have used LLVM for various purposes, showing that you can get up-and-running quickly with LLVM, giving time to do interesting things, even if you only have a semester in a University course. We also have a list of ideas for projects in LLVM.

LLVM was started by the Lifelong Code Optimization Project, led by Vikram Adve at the University of Illinois, Urbana-Champaign. Since the first public release, LLVM has grown to include contributions from several other people! We welcome external contributions, so please send e-mail to llvmdev@cs.uiuc.edu if you are interested in contributing code to the LLVM infrastructure.

Maybe LLMV (http://llvm.org) will be of interest to OpenBSD folks.

Highly BSD compatible. Advanced.

True-Freedom BSD-Style License (no Communistic-License like GPL).
http://llvm.org/releases/1.3/LICENSE.TXT

While LLVM is currently being developed using GCC it's intended to stand alone at some point.

From the LLVM web site: http://llvm.org./

Low Level Virtual Machine (LLVM) is:

      1.

            A compilation strategy designed to enable effective program optimization across the entire lifetime of a program. LLVM supports effective optimization at compile time, link-time (particularly interprocedural), run-time and offline (i.e., after software is installed), while remaining transparent to developers and maintaining compatibility with existing build scripts.
      2.

            A virtual instruction set - LLVM is a low-level object code representation that uses simple RISC-like instructions, but provides rich, language-independent, type information and dataflow (SSA) information about operands. This combination enables sophisticated transformations on object code, while remaining light-weight enough to be attached to the executable. This combination is key to allowing link-time, run-time, and offline transformations.
      3.

            A compiler infrastructure - LLVM is also a collection of source code that implements the language and compilation strategy. The primary components of the LLVM infrastructure are a GCC-based C & C++ front-end, a link-time optimization framework with a growing set of global and interprocedural analyses and transformations, static back-ends for the X86, X86-64, PowerPC 32/64, ARM, Thumb, IA-64, Alpha and SPARC architectures, a back-end which emits portable C code, and a Just-In-Time compiler for X86, X86-64, PowerPC 32/64 processors.
      4.

            LLVM does not imply things that you would expect from a high-level virtual machine. It does not require garbage collection or run-time code generation (In fact, LLVM makes a great static compiler!). Note that optional LLVM components can be used to build high-level virtual machines and other systems that need these services.

LLVM is a robust system, particularly well suited for developing new mid-level language-independent analyses and optimizations of all sorts, including those that require extensive interprocedural analysis. LLVM is also a great target for front-end development for conventional or research programming languages, including those which require compile-time, link-time, or run-time optimization for effective implementation, proper tail calls or garbage collection. We have an incomplete list of projects which have used LLVM for various purposes, showing that you can get up-and-running quickly with LLVM, giving time to do interesting things, even if you only have a semester in a University course. We also have a list of ideas for projects in LLVM.

LLVM was started by the Lifelong Code Optimization Project, led by Vikram Adve at the University of Illinois, Urbana-Champaign. Since the first public release, LLVM has grown to include contributions from several other people! We welcome external contributions, so please send e-mail to llvmdev@cs.uiuc.edu if you are interested in contributing code to the LLVM infrastructure.

clang is fairly early on, but so is PCC. PCC supports almost no GCC extensions (e.g. inline asm, attributes, etc), doesn't support C99 fully, and has many other problems. The clang parser is basically done for C and clang has support for several other source analysis tools other than "just code generation". See the slides linked of http://clang.llvm.org/ for details. I'd expect clang to be fully ready for C use in the next year.

llvm-gcc is quite mature (it has built huge amounts of code, including apps like Qt and Mozilla), supports C/C++/ObjC and bits of FORTRAN/Ada if that is your thing. Using llvm-gcc you get the advantages of the LLVM optimizer and code generator with the GCC front-end.

-Chris

clang is fairly early on, but so is PCC. PCC supports almost no GCC extensions (e.g. inline asm, attributes, etc), doesn't support C99 fully, and has many other problems. The clang parser is basically done for C and clang has support for several other source analysis tools other than "just code generation". See the slides linked of http://clang.llvm.org/ for details. I'd expect clang to be fully ready for C use in the next year.

llvm-gcc is quite mature (it has built huge amounts of code, including apps like Qt and Mozilla), supports C/C++/ObjC and bits of FORTRAN/Ada if that is your thing. Using llvm-gcc you get the advantages of the LLVM optimizer and code generator with the GCC front-end.

-Chris

pcc will take YEARS to get the functionality and optimizations that gcc has. Even if it compiles slowly and sometimes generates dumb code.

Either way, they'd much, much better off if they imported LLVM and redirected their compiler brain power to clang.

PCC is interesting, but it's based on technology from the 70's, doesn't support a lot of interesting architectures, and has no optimizer to speak of.

If you're interested in advanced compiler technology, check out LLVM, which is an ground up redesign of an optimizer and retargettable code generator. LLVM supports interprocedural cross-file optimizations, can be used for jit compilation (or not, at your choice) and has many other capabilities. The LLVM optimizer/code generator can already beat the performance of GCC compiled code in many cases, sometimes substantially.

For front-ends, LLVM supports two major ones for C family of languages: 1) llvm-gcc, which uses the GCC front-end to compile C/C++/ObjC code. This gives LLVM full compatibility with a broad range of crazy GNU extensions as well as full support for C++ and ObjC. 2) clang, which is a ground-up rewrite of a C/ObjC frontend (C++ will come later) that provides many advantages over GCC, including dramatically faster compilation and better warning/error information.

While LLVM is technologically ahead of both PCC and GCC, the biggest thing it has going is both size of community and the commercial contributors that are sponsoring work on the project.

-Chris

PCC is interesting, but it's based on technology from the 70's, doesn't support a lot of interesting architectures, and has no optimizer to speak of.

If you're interested in advanced compiler technology, check out LLVM, which is an ground up redesign of an optimizer and retargettable code generator. LLVM supports interprocedural cross-file optimizations, can be used for jit compilation (or not, at your choice) and has many other capabilities. The LLVM optimizer/code generator can already beat the performance of GCC compiled code in many cases, sometimes substantially.

For front-ends, LLVM supports two major ones for C family of languages: 1) llvm-gcc, which uses the GCC front-end to compile C/C++/ObjC code. This gives LLVM full compatibility with a broad range of crazy GNU extensions as well as full support for C++ and ObjC. 2) clang, which is a ground-up rewrite of a C/ObjC frontend (C++ will come later) that provides many advantages over GCC, including dramatically faster compilation and better warning/error information.

While LLVM is technologically ahead of both PCC and GCC, the biggest thing it has going is both size of community and the commercial contributors that are sponsoring work on the project.

-Chris

PCC is interesting, but it's based on technology from the 70's, doesn't support a lot of interesting architectures, and has no optimizer to speak of.

If you're interested in advanced compiler technology, check out LLVM, which is an ground up redesign of an optimizer and retargettable code generator. LLVM supports interprocedural cross-file optimizations, can be used for jit compilation (or not, at your choice) and has many other capabilities. The LLVM optimizer/code generator can already beat the performance of GCC compiled code in many cases, sometimes substantially.

For front-ends, LLVM supports two major ones for C family of languages: 1) llvm-gcc, which uses the GCC front-end to compile C/C++/ObjC code. This gives LLVM full compatibility with a broad range of crazy GNU extensions as well as full support for C++ and ObjC. 2) clang, which is a ground-up rewrite of a C/ObjC frontend (C++ will come later) that provides many advantages over GCC, including dramatically faster compilation and better warning/error information.

While LLVM is technologically ahead of both PCC and GCC, the biggest thing it has going is both size of community and the commercial contributors that are sponsoring work on the project.

-Chris

'm not against Pascal as a language, I just question why we need yet another toolchain. Why not write a good pascal frontend for GCC and then take advantage of all the platforms they support + optimizers, etc. The optimizers in GCC are hardly language specific, most of them work on very abstract machine representations after the language has been compiled to an independent machine type.

The main reason is that GCC is written in C, while Free Pascal is written in Pascal (it's self-compiling). All FPC developers are much more proficient and productive in developing in Pascal than developing in C (although most of them do know C as well). FPC also has a reasonably easy to understand OOP design for its code generator and parse tree and is still small enough so one single person can have a pretty good overview of the entire thing. The same cannot be said of GCC.

As far as generic optimizers etc. are concerned, I've been thinking lately of adding a code generator backend which spits out LLVM code.

Suppose I'm a developer in a company. We work with, say, crypto software, on a variety of embedded platforms [not all of which run Linux]. Please tell me why I should consider Pascal.

Or, suppose I've been dropped into a company with millions of dollars of existing software toolkits [all written in C]. Please tell me why we should port it all to Pascal?

Pascal is cool and all, but until I can be reasonably assured to find it everywhere, in a project [like GCC] that I can be assured will be here in 5 years, I can't really promote it in any of my projects.

GCC has been around for 2 decades or so. It has scores of contributors from all walks of life and industry. Can you say the same for Free Pascal?

unless they intend on taking over their own fork of GCC (a monumental task which would substantially harm their ability to support BSD itself)

A fork of GCC would still be licensed under GPLv2. Don't you think FreeBSD should use a compiler with a BSD-style license? And maybe one whose architecture is a bit more modern than gcc?

Luckily, the back-end already exists, and the front-end is already under active development.

You might like clang from the LLVM project.

You might like clang from the LLVM project.

This is occurring

I'm going to take that out. My point here is that the fork has occurred at the GPLv2 point. If the GPLv3 is considered too onerous, it is entirely possible that the GPLv3 code could never be merged.

I'm also not going to argue that the compiler front-end isn't complex and difficult to replace -- of course it is. I don't personally think there's much value in a re-implementation. That said, I do believe that LLVM provides a potential migration path away from the GPL, if corporations with sufficient resources deemed such a migration to be worthwhile.

This is occurring

I'm going to take that out. My point here is that the fork has occurred at the GPLv2 point. If the GPLv3 is considered too onerous, it is entirely possible that the GPLv3 code could never be merged.

I'm also not going to argue that the compiler front-end isn't complex and difficult to replace -- of course it is. I don't personally think there's much value in a re-implementation. That said, I do believe that LLVM provides a potential migration path away from the GPL, if corporations with sufficient resources deemed such a migration to be worthwhile.

This is occurring

This is occurring

Yeah, I always find this debate pointless on its face. BSD uses huge amounts of GPL-licensed software, so there's no substantial difference. In fact, BSD will be using GPLv3-licensed software, unless they intend on taking over their own fork of GCC (a monumental task which would substantially harm their ability to support BSD itself).

This is occurring -- see LLVM and LLVM-GCC. Several corporations are contributing to LLVM -- including Apple

The open source iPhone development tools currently use LLVM with the GCC front-end. In this case, the gcc driver is used to interface with LLVM, and output LLVM byte-code. LLVM handles the assembly/linking of this byte code as a native executable. The GCC driver simply provides a fully GCC-compatible front-end -- it can (and has been) forked from GPLv2 licensed gcc, and in theory, could be maintained in perpetuity as a fork -- or potentially replaced outright.

This is occurring -- see LLVM and LLVM-GCC. Several corporations are contributing to LLVM -- including Apple

The open source iPhone development tools currently use LLVM with the GCC front-end. In this case, the gcc driver is used to interface with LLVM, and output LLVM byte-code. LLVM handles the assembly/linking of this byte code as a native executable. The GCC driver simply provides a fully GCC-compatible front-end -- it can (and has been) forked from GPLv2 licensed gcc, and in theory, could be maintained in perpetuity as a fork -- or potentially replaced outright.

In asm, you can modify the code on the fly

You can do the same with C and C++.

How about the source code? http://llvm.org/svn/llvm-project/cfe/trunk/

I got it from here (PDF). While they call "obsoleting GCC" a "non-goal", it's almost certainly a goal of other folks that the FSF has targeted, such as TiVo, and therefore I could see people using Apple's work as a starting point.

http://llvm.org/ is one of the better C++ projects I've seen. Quite large, but also clean and tidy.

I have little doubts that Apple will try to use/make another compiler as soon as they can so they can avoid having to share their changes.

I have little doubts that Apple will try to use/make another compiler as soon as they can so they can avoid having to share their changes.

I have little doubts that Apple will try to use/make another compiler as soon as they can so they can avoid having to share their changes.

They have hired Chris Lattner, one of the lead developers of LLVM, which is an optimizing compiler. The bulk of LLVM is released under a BSD-like license, but for now it uses a front end borrowed from GCC.

But Apple is busy fixing that, too.

Nice try but no cigar. If you read the LLVM license it compiles against the GNU libc. Admittedly it is LGPL but it is still GNU code.

Apple had originally been working on merging GCC and LLVM. I'm not at the WWDC this week, but from what I've heard they have decided to do away with the GCC bit and write their own front-end called "clang". Slides are available in the "Using LLVM" section of this presentation. They also further discuss their OpenGL shading language front-end for LLVM.

No, but running full-screen streaming video (even at 480x320) with all that dynamic eye candy is a problem, and a hard one. It's the kind of problem that requires dedicated processors (the device has at least three) and serious optimization (they've been contributing to the LLVM project).

I'm sure porting Darwin to ARM was the easiest part of the whole process. The question is - how much did they strip out? How does that affect the SDKs? Just what level of expertise is necessary to write software for the device? Will they have every built-in application promised at Steve's demo?

I'm guessing that is what is going on. They need people who know the internals to write the applications, because it is so complex and the execution environment so fragile it is a necessity.

I just started looking at LLVM, maybe it is good for what you want.

http://llvm.org/

Stallman also now has a monopoly lock on a crucial piece of operating system infrastructure; the C compiler. There is no other genuinely viable FOSS C compiler in existence, other than GCC.

I snidely mentioned it elsewhere, but it deserves repeating and a link: LLVM is shaping up to be an awesome compiler, with the benefit of being a relatively new and therefore pretty clean and modern codebase.

To compile C and C++, it currently does make use of gcc as a front end, which is the simplest end of any compiler (certainly C at any rate -- C++ is a good deal trickier). I'm not saying it'd be trivial to replace the gcc part -- and LLVM doesn't really have any interest in doing so right now -- but it's certainly closer being possible without writing a new compiler from scratch.

I'm the same way, not a fan at all, but, I do wonder at what point, what would prevent MS from basically thumbing their nose at EU, and saying fine, we'll just withdraw all new products from you market...and if things got worse, just plain stop supporting the products currently out there in EU.

The EU created Airbus just to compete with Boeing. They could do something similar just to compete with Microsoft. And with Linux, BSD, etc. out there, this might be not such a huge investmend. Preinstalled on computers sold in the EU, a standarized Linux distribution with a binary driver interface added (e.g. using LLVM) could have a good stand against MS. Of course, the open source spirit will be lost, but it could be a huge commercial success.

To make a long story short: MS doesn't want to battle the EU. MS would not only loose a big market, but worse a real competitor might be created, subsided by huge amounts of EU money.

As I understand it, as a novice, the only way to communincate or syncronize data is via copies of data passed via something analogous to a socket. A Socket is a serial interface. If you think about this for a moment, you realize this could be thought of as one byte of shared memory. Thus a copy operation is in effect the iteration of this one byte over the data to share. At any one moment you can only syncronize that one byte.

But this suggests it's own solution. Why not share pages of memory in parallel between processes. This is short of full access to all of the state of another process. But it would allow locking and syncronization processes on entire system states and the rapid passing of data without copies.

There is another solution: All data shares the same address space, so it can be accessed by simple pointers; no copying is necessary. To ensure the integrity of the system, all code in the kernel space must be proofed for correctness. To decrease the cost of such proofs, automatic proofers are used. Such systems are already in use: the proofs are done using a type checker on a type system. The type system ensures, that the data is only modified on a certain way, e.g. using certain primitives. When the code is loaded, a type checker verifies, that the code is properly typed, i.e. follows the rules of the type system. Using an appropriate type system, it can be ensured, that correct locking is performed, that onlycertain modules can access specific data, or that each access is monitored by some security system.

Using a type system to ensure the integrity of the data is used in most modern programming languages. E.g. many applets can share the same address space in the same Java VM; but they cannot tinker with each others data -- the type system prevents this. Type checkers in an operating system are already in use. Examples for this are Kernel Mode Linux and Java Operating System.

To use a type system in the kernel context, it is not necessary to use a full blown virtual machine, which interpretes the code and provides garbage collection. But the assembler code must contain type information. This is ensured by modern assembler varaints, e.g. Typed Assemlber Language or the Low Level Virtual Machine.

Using this approach, you can have a better compromise between speed and isolation. Shared memory looks nice on the first glance, but gives an component full access on the shared data. Shared memory does not solve synchronization. Using a proper type system, synchronisation becomes trivial. Just think of the synchronized keyword or the atomic datatypes in Java.

That's where a project like LLVM comes in. Platform-neutral binaries via LLVM bytecode, and full processor-specific link-time native compilation+optimization when a binary is installed. Alternatively, you can JIT the bytecode at runtime. Developers just distribute LLVM bytecode binaries, and the installers/users do the rest. I think the LLVM approach is the future.

LLVM would be one. See http://llvm.org/

But, LLVM uses the GCC front ends, and the GCC folks are actually discussing the integration of LLVM into GCC itself.

That is an imaginary problem, since standard hardware can support runtime safety, garbage collection, and object-oriented dispatch efficiently.

It can, if we impose a type system on top of it. See LLVM for example. You might be interested in LLVA in fact. ("LLVA: A Low-level Virtual Instruction Set Architecture")

Decomposition is good; using the MMU to decompose a large software system (kernel or otherwise) into modules is often bad.

This is pure speculation. You have no real evidence to support this conjecture.

"My idea" is a system very much like the Linux or BSD kernel but written in a compiled language with garbage collection, runtime safety, well-defined primitives for accessing memory and hardware, and a simple object system. [...] The main benefits would be easier testing, easier extensibility, fewer security problems, and better support for dynamically loadable modules.

I'm still not clear whether you actually have processes, or they're simulated on the kernel VM, whether you use the MMU at all, or rely solely on the type-safe IL and code-generation (ie. the CPU is always in kernel-mode and the entire system is essentially running as one big VM).

Also, I've pointed out a serious problem in this approach: DoS attacks. Perhaps this will be addressed when/if you can answer the above questions, but how do you account for resources and charge them to appropriate entities? Without proper accountability, you open up the system to DoS. The failure boundary for a JVM is the JVM itself. If a malicious object started accumulating memory without freeing it, the JVM would fail. What is the failure boundary in your system design?

Let's say you find a way to handle the above memory accounting issue, what if the malicious object now accumulated garbage and freed it immediately, causing a great deal of work for the garbage collector? How is the CPU time tracked and charged to the object in question? Is each object individually schedulable? Does that now make them active objects, aka Actors? They get their own thread and communicate via messages? That fundamentally changes the JVM computational model. You may need a process model of some sort after all.

Notice, the above accounting issues are all existing problems with just about every popular kernel in existence, particularly UNIX clones. EROS/CapROS has solved all of the above issues and Coyotos will inherit those benefits. I'm less familiar with L4, but I believe it too has solved the all of the issues; L4 is currently simply weak the security department compared to EROS/CapROS.

LLVM is an aggressive compiler that is able to do many cool things. Best yet, it has a demo page here: http://llvm.org/demo, where you can try two different things and see how they compile.

One of the nice things about this is that the code is printed in a simple abstract assembly language that is easy to read and understand.

The compiler itself is very cool too btw, check it out. :)

-Chris

it's happened already - some of the main C++ benchmark programs run faster under LLVM than with GCC. Competition is a very good thing. Let's see if GCC's SSA will turn the tide.