Slashdot Mirror
Android ICS Will Require 16GB RAM To Compile
ozmanjusri writes "New smartphones may be lightweight, compact objects, but their OSs are anything but. Ice Cream Sandwich will need workstations with no less than 16 GB RAM to build the source code, twice the amount Gingerbread needed. It will take 5 hours to compile on a dual quad-core 2+GHz workstation, and need 80GB disk space for all AOSP configs. Android developers are also being warned to be cautious of undocumented APIs: 'In almost every case, there's only one reason for leaving APIs undocumented: We're not sure that what we have now is the best solution, and we think we might have to improve it, and we're not prepared to make those commitments to testing and preservation. We're not claiming that they're "Private" or "Secret" — How could they be, when anyone in the world can discover them? We're also not claiming they're forbidden: If you use them, your code will compile and probably run.'"
Nobody will ever need more than 16GB...
If you want news from today, you have to come back tomorrow.
Next Android will need 32GB.... Then 64GB... Soon, Skynet level of resources....
16GB recommended, or else you'll be waiting quite awhile. But, you could build the thing in less RAM than that, right?
This is of course if you don't edit out that -j64 in Google's main Makefile.
A pile of ram, or a pile of time as it thrashes your hard disk? Ram is cheaper than time.
This is a guess as to the reason.
One of the better ways to optimize C++ code for building with GCC is to put all of the source code into one big code file. Or you can build it as a few independent modules, but the code is still quite large. Then you build it with the O3 flags. In GCC, the amount of RAM and CPU used in an O3 compile goes up by quite a lot as the code size in a single module increases. I am not sure what the exact equation is but I think it's an exponential function.
This would easily explain the RAM and CPU usage.
I mean, you can spend $100 and buy a 16-inch horse cock dildo, but that doesn't mean you can shove the whole thing up your ass.
You beat me to it. It's not like 16GB is really asking that much for a dev workstation. I've got 24GB in mine currently.
Sucks if your motherboard doesn't support 16GB RAM or if you have a laptop.
While Android will remain open-source, eventually it will require so much RAM/processing power/etc. to compile that only Google will have the computational resources available to compile it.
Clever!
Quick question for those with giant codebases such as this. How the heck do you test, and debug the software with those kind of lag times? Do you split everything up into smaller pieces or something? If so, then surely there are cases where you need to test something that requires EVERYTHING to be compiled. I can imagine such shot in the dark scenarios to be the stuff of pure nightmares.
Why OpalCalc is the best Windows calc
Ice Cream Sandwich will need workstations with no less than 16 GB RAM to build the source code, twice the amount Gingerbread needed.
2 * 2 = 16?
Gingerbread can be compiled with 2GB.
16GB RAM recommended, more preferred, anything less will measurably benefit from using an SSD.
Emphasis mine. Still pretty beast, though.
I mean, you can spend $100 and buy a 16-inch horse cock dildo
I'll leave that to you. Interesting that you knew the price off the top of your head, though.
Unless the build system is screwed up, recompiling after a change should be relatively fast. Usually source code is stored as lots of smaller files, and each file is compiled separately to produce a separate object file (e.g., .o). Then next time a rebuild is requested, the system should notice what changed, and only rebuild the needed parts. Some parts take the same time each time (e.g., a final link), but it shouldn't take anywhere near the same amount of time.
There are lots of build tools, including make, cmake, and so on. If you use the venerable "make" tool, you might want to read Miller's "Recursive Make Considered Harmful":
http://aegis.sourceforge.net/auug97.pdf
Cue the lovers and haters of "make", here :-).
- David A. Wheeler (see my Secure Programming HOWTO)
From TFA:
5+ hours of CPU time for a single build, 25+ minutes of wall time, as measured on a workstation (dual-E5620 i.e. 2x quad-core 2.4GHz HT, with 24GB of RAM, no SSD).
and a computer that can take 16 gigs of RAM.
Indeed though support for 16GB is nothing unusual these days, if your desktop was made in the last couple of years and wasn't bottom of the barrel it will probablly support 16GB.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
TFA says:
25 minutes of wall time is nothing for a first build. After that updates from changes in the source code will be trivial.
25 minutes to build a complete linux distro is fantastic.
TFA: "5+ hours of CPU time for a single build, 25+ minutes of wall time, as measured on a workstation (dual-E5620 i.e. 2x quad-core 2.4GHz HT, with 24GB of RAM, no SSD)."
/. Summary: "It will take 5 hours to compile on a dual quad-core 2+GHz workstation"
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
In game programming, Incredibuild is a common tool for that. You run it on everyone's machine and it integrates with Visual Studio. Lets you reduce build time a ton since you have a lot of resources to use. Also tends to scale nicely as the larger the project, the more people working on it and thus the more computers available and so on. You can, of course, have dedicated servers just for compiling but many places don't bother, just having it use idle time from office systems as it is amazing how much that can add up to. Particularly since it is very rare for all devs to compile something at once.
Bingo, dev workstations are not 199$ Acer's bought at K-mart
While it is a lot of RAM compared to what many system have, it really isn't a big deal these days. 4GB DDR3 sticks are $25 or less each, and that is for high quality RAM. Regular, consumer grade, LGA1155 boards support 4 of them. So for $100 you can have 16GB on a normal desktop system. My home system I type this on has 16GB for that reason. It was so cheap I decided "Why not?"
They actually can support more, with 8GB chips you can have 32GB on a standard desktop, but those are still expensive.
The enthusiast X79 LGA2011 boards coming out will have 8 sockets and thus handle 64GB. Of course beyond that there's workstation which cost a lot more, but not as much as you might first think.
At any rate, 16GB is now a "regular desktop" amount of RAM. Standard boards the likes of which you get in cheap ($1000 or less) towers support that much, and it only costs $100 to get. It is quite a realistic thing to require, for something high end.
I was looking for something else to do on my old 16-cpu Itanium cluster with 64gbs of shared ram. I think I just found it...
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
16 GB is far more than any desktop user should need, and most laptops simply cannot hold that much, so it's creating a sharp demarcation between user and developer. This is bad. You want your advance users to naturally transition into becoming developers, and making your codebase inaccessible for them prevents that.
IOW, most people have suggestions for improvement for any tool they use. Ideally, it would be trivial for someone to download the source, modify it, recompile, test, and submit improvements. People start with simple things (e.g. misspelled words) and move to more advanced tasks as they gain familiarity. By requiring several hundred dollars of hardware and massive time investments, you are ensuring that users never become developers, just needy consumers whining about feature requests.
Unless the build system is screwed up, recompiling after a change should be relatively fast. Usually source code is stored as lots of smaller files [...] Then next time a rebuild is requested, the system should notice what changed, and only rebuild the needed parts.
I feel your pain brother.
...does a phone really require an OS of that complexity? Don't get me wrong, I have a current generation Android smartphone I bought 2-3 months ago, 4G enabled, it even has an HDMI out, and I completely comprehend that a modern smartphone is essentially a fully fledged computer.
That being said, it's still a phone. And in fact, it's horrible at it. To redial the last number I have to press 3 buttons (1 physical, 2 virtual) and suffer through 3s+ of erratic lag or more. On my 4 year old boring but functional Blackberry, it took under 1s and all I had to do is press the same button twice. Hanging up, redialing, going back to the home screen is slow as molasses. Yes, I can browse the web, access my Google Docs, open PDFs, read books, play chess, watch Youtube HD, etc. etc.
The smart- part of the phone is great and a step forward, however the -phone part of the phone is actually a big step back in my opinion.
What is it actually doing that needs 16GB of RAM to compile?
Speak for yourself. I pay less and take more. Practice makes perfect.
....can't you believe that they're deliberately undisclosed because they don't want to support them in any fashion, as they stated?
The compiler has no visibility into whether the memory space it is executing in is actually mapped to physical ram.
If the compiler doesn't know its own resident size, then how does top know the compiler's resident size, and how does ps aux know the compiler's resident size? I imagine that if a program detects that it's being swapped out, it might be able to adjust its CPU/memory tradeoffs at runtime.
Human nature -- If you document it, people will expect it to be stable (no matter what you may say to the contrary). Undocumented API's have a built-in "we told you so" flavour to them.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
Android has always been RAM-intensive, and it makes sense since you have no choice but to build an entire OS at once
Can't you do the initial build overnight, come back the next day, do more development, and recompile only the parts that changed?
I know nothing about this sort of programming/compiling. :(
I wonder how long a full compile of that takes...
Unless the entire program is in 1 gigantic 8 billion billion billion line file why would it need that many resources or even be able to use 16 GB of RAM?
Assuming it is like a normal program would it not just be a large collection of relatively small files that are compiled one after the other (theoretically number of CPUs + 1 threads running at a time with that many files being compiled concurrently being the optimal solution)?
And I just do not see how you could ever use up 16 GB at any one time.
Troll is not a replacement for I disagree.
This article would be shocking, but considering that 16 GB of memory -- especially the dual-channel DDR3 used for the i5 and consumer i7's -- is so cheap, less than $100, this article doesn't have any shock value. It's just informative. It's letting us know the 'recommended' memory and giving more nerds an excuse to add more RAM. That is the NERDS that don't already have 24 gigs for their virtual machines. :P
16 GB is far more than any desktop user should need, and most laptops simply cannot hold that much, so it's creating a sharp demarcation between user and developer.
I have 8GB in my development system at work. With two copies of Eclipse sucking up a gigabyte each, if I try to compile my C++ software without shutting down the old version, I go over 8GB and start to swap.
16GB would definitely be beneficial; I put 16GB of 1.6GHz DDR3 in my home server when I built it earlier this year and it cost under $150. ECC RAM for the work system would cost more, but would probably pay for itself in a few weeks.
Why would ECC pay for itself in a few weeks?
maybe he meant the productivity increase due to the time saved from the ram upgrade, even through it was the more expensive ecc type ram.
I got a rather workable gaming/development laptop (HP) from an office supply store just 3 years ago. It supports a max of 8gigs of ram. Granted, it was the cheapest laptop with a nvidia 9600GTM with a gig of dedicated ram(really, a better mobile graphics chip than most of the 100 and 200 series), which is to say not the cheapest thing on the shelf.
True, I didn't buy it for Android development, and making the trade off for a few extra minutes of wall time versus a whole new computer is worth it in my short run. But I wouldn't presume that it takes a decade old computer to not support more than 8 gigs of ram. Plenty of cheap/lazy motherboard suppliers out there.
You're completely wrong of course.. Rtfa and be a class act and admit your mistake. Otherwise you're just a common troll.
Note that this is the entire system, so building Ice Cream Sandwich from source is just like rebuilding an entire Linux distribution from scratch from the ground up, from the kernel down to the user tools and the windowing system, etc. If you've ever used Gentoo, this should sound familiar. I wonder if some of the tools that Gentoo users are familiar with to help speed along compilation, such as distcc, could help with this.
Qu'on me donne six lignes écrites de la main du plus honnête homme, j'y trouverai de quoi le faire pendre.
8 gigs is a 2 slot board you can pick those up for less than 40 bucks in desktop form if you dont mind a older CPU and yea a decade old, 16 gig requires a bit more
maybe next years 39.99 boards but its totally not needed at this point
Is this a case of preprocessing gone wrong? Sometimes preprocessing can be a monster because it blows up each .c file into a monster file due to (almost) every .h file being included, which lead to long compile times. This is especially the case when you have large numbers of .c files. Preprocessing was invented as a hack in the time that memory too small for the whole source to be compiled in one time. But when applied to large systems, it causes the compile time to increase, because every piece of code in a .h files is compiled over and over again. I think that it is one of those hacks (short-cuts) that has gone wrong, because it is due to legacy very difficult to get rid of. The D language would be a good alternative, but translating all C code to equivalent D code is impossible. And I guess that in some case, such as operating systems deployed at multiple platforms, you do need conditional compiling, and I am not sure if that can be implemented with D.
16 GB of RAM is an awful lot of it, by today's standards.
The D programming language compiles much faster than C++. It was designed to be easier to lex & parse
So how much of the compile time for C/C++ code is spent processing the characters in the source code and how much is spent processing the intermediate representation and turning it into machine code? If the answer is "most of the time is spent doing the latter", then "designed to be easier to lex and parse" doesn't help much.
(And how much of Android is C/C++ and how much of it is in their Java dialect? How much of that time is spent translating C/C++ to machine code and how much of it is spent translating Java to Dalvik bytecode?)
No one will have the hardware to compile the source code.
Do no evil?
As far as I'm concerned, demanding that all free software be simple enough that anybody could buy a machine capable of compiling it in a reasonable time isn't exactly evil, but it's massively assholic. "Free" doesn't mean "simple enough that a cheap machine can compile it in a reasonable amount of time".
And, in any case, I wouldn't go so far as to say "no one will have the hardware to compile the source code".
I am aware of GCC eating gigabytes when doing some kind of optimisations; this already used to happen some time ago, so I'm not surprised that running 8 copies of GCC in parallel might make some pages hit the swap on a non-ninja workstation.
*If* this is the problem, then I don't see it as a immediate menace to the openness of Android. I remember when compiling the Linux kernel took hours on my Pentium 2. Compiling GCC or Qt still takes half a day on my Pentium IV. I don't expect anything different from a user-oriented machine.
Ideally, it would be trivial for someone to download the source, modify it, recompile, test, and submit improvements.
I'm not holding my breath waiting for that, any more than I'm holding my breath waiting for everybody who has pain in a joint to open themselves up and replace the joint, or waiting for everybody who finds {their car, their bicycle, their local public transportation system, ...} to open up their box of tools and fix it. No, I don't think most people will want to be developers, and I don't think most people would be able to be developers. I don't think that because I want to be a Highly Paid Member Of A Priesthood, I think it because I haven't seen anything to indicate that most people will want to be software developers - they have better things to do with their lives.
People start with simple things (e.g. misspelled words) and move to more advanced tasks as they gain familiarity.
If they gain familiarity.
By requiring several hundred dollars of hardware
How many of the people with the time, energy, and ability to become developers won't be able to afford the hardware at the prices many people here have cited?
and massive time investments
Making changes to the core of Android (or iOS or Mac OS X or Windows or the Linux core or...) is eventually going to require a significant time investment to understand the code in question. If you're making relatively minor tweaks a single application, you may not need a machine so Studly(TM) as to be able to compile the entirety of Android in a reasonable time.
you are ensuring that users never become developers, just needy consumers whining about feature requests.
It might be more like "you are ensuring that users never become needy developers whining that their poorly-conceived patch hasn't been accepted, just needy consumers whining about feature requests". :-) I might well consider a needy consumer whining about a feature request less annoying than a needy developer who cooked up some half-assed "solution" to a problem, causing more problems to the system as a whole than it fixes, bitching because their patch hasn't been accepted or was sent back with a large number of issues.
As far as I'm concerned, an ideal system is largely written at a high level that makes it hard to screw up, with that high level implemented at a lower level where angels fear to tread. This doesn't introduce a barrier to entry, it just moves the high barrier a little bit closer to the core, and perhaps makes it higher.
maybe he meant the productivity increase due to the time saved from the ram upgrade, even through it was the more expensive ecc type ram.
Yes. Just having to shut down the software to recompile it means I can't do any testing while I'm waiting for the compile to finish; that's not a huge amount of time each day but it all adds up in wasted developer time.
They have the same memory controller as the desktop units. Now if your notebook only has 2 slots, as is common, then 8GB is all you can get cheaply since 8GB chips are high priced and you'd need 2 of them for 16GB.
Yeah laptops are a pain if you want/need lots of ram. Some can handle 2x8GB but 8GB laptop modules are bloody expensive.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
But my point is it really could be. The main reason it isn't is lack of need. 4GB is fine for most things, particularly since many things are still 32-bit. However the fact remains that you can now stick 16GB on a standard desktop board, and do so cheaply. That is what makes it a "mainstream desktop" amount of RAM. You don't have to buy a special system with high end parts to be able to get it, nor do you have to spend much on the RAM itself. Anyone who has a modern (meaning Sandy Bridge) chip can have 16GB easily if they wish.
Compare this to 32GB or 64GB. 32GB IS attainable on standard hardware, but is exceedingly expensive. It would be about $800-1000. That is for $150 boards with $200 processors. As such, out of range in any reasonable sense. It is a very high end thing to have.
64GB is even more high end. At this point, no desktop component works (until the SB-E series comes out). You have to buy workstation/server class boards and Xeon processors. Right there you are talking a ton more money. You also have to use registered memory, which costs more. This is very much a high end only situation.
That is why I maintain it isn't that high end anymore. When you can knock it on to a regular board, the kind of thing you get in cheap Dell or Lenovo systems, and when the cost is minimal, just two games, it really isn't high end. It is perfectly attainable for regular users, if there's a need.
Incorrect, they all support 16GB
THe next gen chipset supports 64GB
Intel's most advanced and expensive laptop processors, on sale currently and in the foreseeable near future, only support 8 GB of RAM.
Some are listed as supporting 16GB or 32GB. for example http://ark.intel.com/products/50067, http://ark.intel.com/products/52224
But yeah in general laptop users get shafted on ram because they generally only get two ram slots and even when their platform supports 8GB modules they are frightfully expensive but then IMO if you are trying to do heavy computation on laptops you are probably doing it wrong.
16 GB of RAM is an awful lot of it, by today's standards.
Currently 16GB is the max you can put in a mainstream (but not bottom of the barrel) desktop with reasonablly priced modules and afaict it's been that way for a couple of years now. Desktops based on the high end LGA1366 socket support 24GB of reasonably priced modules.
Once you go byeond 24GB you currently have to either buy frightfully expensive 8GB unregistered modules or buy a server platform that supports registered modules and/or more ram slots.
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
I won't argue what he will or won't be if he doesn't kowtow to you, but your post is far more trollish.
If people are interested, I'll sell 'em for 16 gigabytes worth of ram.
Please login to access my lawn
Sorry, but C/C++ isn't designed for "project files" or the kinds of build systems that MS has been trying to shoehorn those languages into. That is a problem with C/C++ (and a real and serious one at that). But you can't fix it in the compiler or IDE without breaking the language. The next C++ language standard will hopefully address it.
Microsoft... proudly popularizing dumb ideas since 1980.
So.... I can develop for Android ICS. Now! Just to learn Java! *coughs*
As a owner of one of those CPU, I'm glad for that but I hope there isn't some other 8 GB limitation lurking either in the chipset or in the BIOS preventing me to upgrade to 16 GB in the future.
If you built the kernel, all libraries, X, gnome, and all default applications for gnome in parallel you could see a comparison. So you should. That is what they are doing. They are not compiling the OS. They are compiling the entire AOSP.
Having to work for a living is the root of all evil.
Do it too fast and you'll get an ice cream headache.
Uh, C and C++ are per-file languages. That said, GCC (and clang) do support precompiled headers, which seem to be what you are rambling about. If you have a project header, you can pre-parse it and automatically add it to the start of every source file pretty trivially.
I am TheRaven on Soylent News
I guess it depends what you're doing too. Compared to all the time you're working on code, design, architecture, documentation, meetings and using cached compiled code, how often do you really make a full recompile and how long do you wait? Are you instead thinking ahead or are those minutes the usual slack people have to get a cup of coffee or something? I mean if you are working for a software giant on a huge project that takes forever to compile, I doubt you'd be stuck with 4GB of RAM. You just have to have a reasonable business case that getting a fast machine would save you X minutes a day.
Live today, because you never know what tomorrow brings
yea if you want the overhead of compression / decompression on top of virtual memory.
At least in the Mac OS 7 days, replacing some hard disk accesses with LZ compression/decompression operations was a net win. This is why Connectix introduced RAM Doubler, the first compressed virtual memory manager for a home computer. How has this tradeoff changed meaningfully since then?
if you were IN CHARGE what would you want? your devs sissy slapping cause everything is in (compressed) swap or 100 bucks in ram?
The motherboard of the PC I'm typing this on doesn't have slots for 100 bucks in RAM. So I'd need a new motherboard, a new CPU, and possibly a new copy of some operating system whose license is tied to the motherboard. This might run a bit more than $100. Or by "IN CHARGE" did you mean "money is no object"?
Sounds like your employers are behind the curve. We've been standardising on 4gb of RAM for our end user desktops to run Outlook, Word and Excel. Since 2007.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
my i7-2720 supports 16gb. and it is about to be superseded by ivy bridge...
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Apple only "support" 8gb because they only support apple memory upgrades. And they choose not to sell 8gb DIMMs. Hence, it is unsupported by apple. The machine will take 16gb just fine, and kits are readily available from OWC.
I run: Windows, OS X, Linux, FreeBSD. Just because you have a hammer, doesn't mean everything is a nail.
Can I see the math? Even rounding up to 30 minutes wall time on 8 CPUs would only get you a maximum of 4 hours CPU time. Unless I'm missing something?
Sounds like your IT department is staffed with people that know nothing about IT. IF you have to travel with your dev boxes, then a Lunchbox PC would be a perfect choice for a very high end setup that is easily portable.
Do not look at laser with remaining good eye.
I have 24 gig in my Old G5 quad core Apple desktop and I used every drop of it. I also know a lot of other people that use 16 GB or more ram in their desktops daily.
I am guess you dont know that they edit TV shows and movies as well as render CG on computers. All on desktops. AVID on it's own will suck up every drop of ram you hand it.
Do not look at laser with remaining good eye.
If one builds a highly memory-constrained machine on purpose
Not all machines in use were built this year. An existing PC motherboard designed to take 32-bit CPUs can't be upgraded to 4 GB, let alone 16 GB.
(only installing 1GB RAM for a Android build machine, for example)
Then I guess that kills the project to make Android self-hosting ;-)
Does this mean they'll be releasing the source code?
A Core 2 Duo with 4 GB RAM, but you'll be waiting forever.
Just like with games, there's minimum specs that technically will work, just nobody's going to be happy with it. This is recommended specs, what the average developer could be happy with.
Same with my ASUS laptop, but it will take 8GB DIMM as well.
Noting the comment that anything less than 16GB would benefit from an SSD, I suspect the issue is that there are many small files and so the seek latency of spinning disks are what causes the pain.
It should work just fine, might take twice as long though. (Which is still not bad, our whole system takes many hours to build because we're basically building up a whole distro for multiple architectures and machine types.)
I build Gingerbread for single device (CM7.1 to be specific) pretty often and it takes 20+ minutes to build.I use quad core Phenom 955, 8GB RAM and SSD.
Somehow I doubt ICS will need considerably more time to build.
I know people are building Android in reasonable time on lesser spec machines.
This is probably a spec of shared build server they use at Google.
So basically they need a BEAST