Slashdot Mirror
Android ICS Will Require 16GB RAM To Compile
ozmanjusri writes "New smartphones may be lightweight, compact objects, but their OSs are anything but. Ice Cream Sandwich will need workstations with no less than 16 GB RAM to build the source code, twice the amount Gingerbread needed. It will take 5 hours to compile on a dual quad-core 2+GHz workstation, and need 80GB disk space for all AOSP configs. Android developers are also being warned to be cautious of undocumented APIs: 'In almost every case, there's only one reason for leaving APIs undocumented: We're not sure that what we have now is the best solution, and we think we might have to improve it, and we're not prepared to make those commitments to testing and preservation. We're not claiming that they're "Private" or "Secret" — How could they be, when anyone in the world can discover them? We're also not claiming they're forbidden: If you use them, your code will compile and probably run.'"
Nobody will ever need more than 16GB...
If you want news from today, you have to come back tomorrow.
Next Android will need 32GB.... Then 64GB... Soon, Skynet level of resources....
16GB recommended, or else you'll be waiting quite awhile. But, you could build the thing in less RAM than that, right?
This is of course if you don't edit out that -j64 in Google's main Makefile.
A pile of ram, or a pile of time as it thrashes your hard disk? Ram is cheaper than time.
Seriously, you're complaining that you need $100 of RAM?
This is a guess as to the reason.
One of the better ways to optimize C++ code for building with GCC is to put all of the source code into one big code file. Or you can build it as a few independent modules, but the code is still quite large. Then you build it with the O3 flags. In GCC, the amount of RAM and CPU used in an O3 compile goes up by quite a lot as the code size in a single module increases. I am not sure what the exact equation is but I think it's an exponential function.
This would easily explain the RAM and CPU usage.
http://xkcd.com/303/
tabletroms
what are troms? and what do they have to do with tables?
While Android will remain open-source, eventually it will require so much RAM/processing power/etc. to compile that only Google will have the computational resources available to compile it.
Clever!
Quick question for those with giant codebases such as this. How the heck do you test, and debug the software with those kind of lag times? Do you split everything up into smaller pieces or something? If so, then surely there are cases where you need to test something that requires EVERYTHING to be compiled. I can imagine such shot in the dark scenarios to be the stuff of pure nightmares.
Why OpalCalc is the best Windows calc
Ice Cream Sandwich will need workstations with no less than 16 GB RAM to build the source code, twice the amount Gingerbread needed.
2 * 2 = 16?
Gingerbread can be compiled with 2GB.
16GB RAM recommended, more preferred, anything less will measurably benefit from using an SSD.
Emphasis mine. Still pretty beast, though.
Unless the build system is screwed up, recompiling after a change should be relatively fast. Usually source code is stored as lots of smaller files, and each file is compiled separately to produce a separate object file (e.g., .o). Then next time a rebuild is requested, the system should notice what changed, and only rebuild the needed parts. Some parts take the same time each time (e.g., a final link), but it shouldn't take anywhere near the same amount of time.
There are lots of build tools, including make, cmake, and so on. If you use the venerable "make" tool, you might want to read Miller's "Recursive Make Considered Harmful":
http://aegis.sourceforge.net/auug97.pdf
Cue the lovers and haters of "make", here :-).
- David A. Wheeler (see my Secure Programming HOWTO)
From TFA:
5+ hours of CPU time for a single build, 25+ minutes of wall time, as measured on a workstation (dual-E5620 i.e. 2x quad-core 2.4GHz HT, with 24GB of RAM, no SSD).
TFA says:
25 minutes of wall time is nothing for a first build. After that updates from changes in the source code will be trivial.
25 minutes to build a complete linux distro is fantastic.
TFA: "5+ hours of CPU time for a single build, 25+ minutes of wall time, as measured on a workstation (dual-E5620 i.e. 2x quad-core 2.4GHz HT, with 24GB of RAM, no SSD)."
/. Summary: "It will take 5 hours to compile on a dual quad-core 2+GHz workstation"
note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
In game programming, Incredibuild is a common tool for that. You run it on everyone's machine and it integrates with Visual Studio. Lets you reduce build time a ton since you have a lot of resources to use. Also tends to scale nicely as the larger the project, the more people working on it and thus the more computers available and so on. You can, of course, have dedicated servers just for compiling but many places don't bother, just having it use idle time from office systems as it is amazing how much that can add up to. Particularly since it is very rare for all devs to compile something at once.
While it is a lot of RAM compared to what many system have, it really isn't a big deal these days. 4GB DDR3 sticks are $25 or less each, and that is for high quality RAM. Regular, consumer grade, LGA1155 boards support 4 of them. So for $100 you can have 16GB on a normal desktop system. My home system I type this on has 16GB for that reason. It was so cheap I decided "Why not?"
They actually can support more, with 8GB chips you can have 32GB on a standard desktop, but those are still expensive.
The enthusiast X79 LGA2011 boards coming out will have 8 sockets and thus handle 64GB. Of course beyond that there's workstation which cost a lot more, but not as much as you might first think.
At any rate, 16GB is now a "regular desktop" amount of RAM. Standard boards the likes of which you get in cheap ($1000 or less) towers support that much, and it only costs $100 to get. It is quite a realistic thing to require, for something high end.
I built Gingerbread on a 4-year old mac with 4gb. It took a while, but build times are being reported to take ~25 mins.
I was looking for something else to do on my old 16-cpu Itanium cluster with 64gbs of shared ram. I think I just found it...
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-source-code.html
Google does iterative builds, meaning small changes take minimal amount of time to compile. If you have to build from scratch though, it can still be a big headache...
Unless the build system is screwed up, recompiling after a change should be relatively fast. Usually source code is stored as lots of smaller files [...] Then next time a rebuild is requested, the system should notice what changed, and only rebuild the needed parts.
I feel your pain brother.
...does a phone really require an OS of that complexity? Don't get me wrong, I have a current generation Android smartphone I bought 2-3 months ago, 4G enabled, it even has an HDMI out, and I completely comprehend that a modern smartphone is essentially a fully fledged computer.
That being said, it's still a phone. And in fact, it's horrible at it. To redial the last number I have to press 3 buttons (1 physical, 2 virtual) and suffer through 3s+ of erratic lag or more. On my 4 year old boring but functional Blackberry, it took under 1s and all I had to do is press the same button twice. Hanging up, redialing, going back to the home screen is slow as molasses. Yes, I can browse the web, access my Google Docs, open PDFs, read books, play chess, watch Youtube HD, etc. etc.
The smart- part of the phone is great and a step forward, however the -phone part of the phone is actually a big step back in my opinion.
What is it actually doing that needs 16GB of RAM to compile?
Oh hay java has undocumented APIs: everything in sun.*. glibc has undocumented functions too.
Just maybe android also has APIs that are implementation details too.
But no, it must be a conspiracy, or they're just a bunch of mouthbreathing derps. Everyone on slashdot is always a superior genus of being who could do better if they lowered their mighty selves to bother.
Ice Cream Sandwich will need workstations with no less than 16 GB RAM to build the source code, twice the amount Gingerbread needed.
I've been wondering, for quite a while, exactly what patents Microsoft seemed to believe Android infringes upon - but those memory requirements are definitely Microsoft-esque!
And the Redmond folks have lots of prior art too...
#DeleteChrome
....can't you believe that they're deliberately undisclosed because they don't want to support them in any fashion, as they stated?
The compiler has no visibility into whether the memory space it is executing in is actually mapped to physical ram.
If the compiler doesn't know its own resident size, then how does top know the compiler's resident size, and how does ps aux know the compiler's resident size? I imagine that if a program detects that it's being swapped out, it might be able to adjust its CPU/memory tradeoffs at runtime.
Human nature -- If you document it, people will expect it to be stable (no matter what you may say to the contrary). Undocumented API's have a built-in "we told you so" flavour to them.
Sometimes boldness is in fashion. Sometimes only the brave will be bold.
Android has always been RAM-intensive, and it makes sense since you have no choice but to build an entire OS at once
Can't you do the initial build overnight, come back the next day, do more development, and recompile only the parts that changed?
I know nothing about this sort of programming/compiling. :(
I wonder how long a full compile of that takes...
Unless the entire program is in 1 gigantic 8 billion billion billion line file why would it need that many resources or even be able to use 16 GB of RAM?
Assuming it is like a normal program would it not just be a large collection of relatively small files that are compiled one after the other (theoretically number of CPUs + 1 threads running at a time with that many files being compiled concurrently being the optimal solution)?
And I just do not see how you could ever use up 16 GB at any one time.
Troll is not a replacement for I disagree.
The D programming language compiles much faster than C++. It was designed to be easier to lex & parse. Maybe despite all the cool new features of the language (nice threading model, super-duper generic programming, optional garbage collection), something mundane as compilation speed will be the "killer feature" that gets people to migrate to D?
This article would be shocking, but considering that 16 GB of memory -- especially the dual-channel DDR3 used for the i5 and consumer i7's -- is so cheap, less than $100, this article doesn't have any shock value. It's just informative. It's letting us know the 'recommended' memory and giving more nerds an excuse to add more RAM. That is the NERDS that don't already have 24 gigs for their virtual machines. :P
Do they use precompiled headers? I've seen many open source projects neglecting or doing it poorly. It can make wonders.
You're completely wrong of course.. Rtfa and be a class act and admit your mistake. Otherwise you're just a common troll.
No one will have the hardware to compile the source code.
Do no evil?
Note that this is the entire system, so building Ice Cream Sandwich from source is just like rebuilding an entire Linux distribution from scratch from the ground up, from the kernel down to the user tools and the windowing system, etc. If you've ever used Gentoo, this should sound familiar. I wonder if some of the tools that Gentoo users are familiar with to help speed along compilation, such as distcc, could help with this.
Qu'on me donne six lignes écrites de la main du plus honnête homme, j'y trouverai de quoi le faire pendre.
Is this a case of preprocessing gone wrong? Sometimes preprocessing can be a monster because it blows up each .c file into a monster file due to (almost) every .h file being included, which lead to long compile times. This is especially the case when you have large numbers of .c files. Preprocessing was invented as a hack in the time that memory too small for the whole source to be compiled in one time. But when applied to large systems, it causes the compile time to increase, because every piece of code in a .h files is compiled over and over again. I think that it is one of those hacks (short-cuts) that has gone wrong, because it is due to legacy very difficult to get rid of. The D language would be a good alternative, but translating all C code to equivalent D code is impossible. And I guess that in some case, such as operating systems deployed at multiple platforms, you do need conditional compiling, and I am not sure if that can be implemented with D.
That's complete bullshit. The numbers cited in the artical are from a file in the prebuilt directory describing who long it takes Google to build only the toolchain (gcc and stuff) .
16G just to *compile*?
I'm gonna slap the next java dude who even thinks of looking crosswise at a C++ programmer.
I am aware of GCC eating gigabytes when doing some kind of optimisations; this already used to happen some time ago, so I'm not surprised that running 8 copies of GCC in parallel might make some pages hit the swap on a non-ninja workstation.
*If* this is the problem, then I don't see it as a immediate menace to the openness of Android. I remember when compiling the Linux kernel took hours on my Pentium 2. Compiling GCC or Qt still takes half a day on my Pentium IV. I don't expect anything different from a user-oriented machine.
They have the same memory controller as the desktop units. Now if your notebook only has 2 slots, as is common, then 8GB is all you can get cheaply since 8GB chips are high priced and you'd need 2 of them for 16GB.
Have they always been this bad and I've just never realised? The ammount of plainly wrong summaries on here is far too high these days. Please editors, do something about this.
I can envision the future. Developing for the free and open Android platform will only be possible on Google's developer server farms, after you have pledged your firstborn, registered all your personal data up to and including your great grandmother's maiden name, and ceded all rights to every byte of your sources to the company that does no evil. But it will be totally free...
and they compile the source in ram.
Seriously, if caches worked that great, gcc would precache all needed includes and .cpps in ram, then compile, with the harddisk not even raising a watt.
gcc is dumb, its a per file compiler, not a heres my project tree, go figure it out dumbass.
relying on scripts such as make to farm out multiple gcc execs is so 1970s.
Come on GCC, process real project files for once.
It seriously needs rethink of architecture.
Cya in 2020 when they have done it, 5 years after MS in 2015.
Liberty freedom are no1, not dicks in suits.
But my point is it really could be. The main reason it isn't is lack of need. 4GB is fine for most things, particularly since many things are still 32-bit. However the fact remains that you can now stick 16GB on a standard desktop board, and do so cheaply. That is what makes it a "mainstream desktop" amount of RAM. You don't have to buy a special system with high end parts to be able to get it, nor do you have to spend much on the RAM itself. Anyone who has a modern (meaning Sandy Bridge) chip can have 16GB easily if they wish.
Compare this to 32GB or 64GB. 32GB IS attainable on standard hardware, but is exceedingly expensive. It would be about $800-1000. That is for $150 boards with $200 processors. As such, out of range in any reasonable sense. It is a very high end thing to have.
64GB is even more high end. At this point, no desktop component works (until the SB-E series comes out). You have to buy workstation/server class boards and Xeon processors. Right there you are talking a ton more money. You also have to use registered memory, which costs more. This is very much a high end only situation.
That is why I maintain it isn't that high end anymore. When you can knock it on to a regular board, the kind of thing you get in cheap Dell or Lenovo systems, and when the cost is minimal, just two games, it really isn't high end. It is perfectly attainable for regular users, if there's a need.
How much memory do I need to recompile iOS?
I won't argue what he will or won't be if he doesn't kowtow to you, but your post is far more trollish.
So.... I can develop for Android ICS. Now! Just to learn Java! *coughs*
If you built the kernel, all libraries, X, gnome, and all default applications for gnome in parallel you could see a comparison. So you should. That is what they are doing. They are not compiling the OS. They are compiling the entire AOSP.
Having to work for a living is the root of all evil.
Do it too fast and you'll get an ice cream headache.
yea if you want the overhead of compression / decompression on top of virtual memory.
At least in the Mac OS 7 days, replacing some hard disk accesses with LZ compression/decompression operations was a net win. This is why Connectix introduced RAM Doubler, the first compressed virtual memory manager for a home computer. How has this tradeoff changed meaningfully since then?
if you were IN CHARGE what would you want? your devs sissy slapping cause everything is in (compressed) swap or 100 bucks in ram?
The motherboard of the PC I'm typing this on doesn't have slots for 100 bucks in RAM. So I'd need a new motherboard, a new CPU, and possibly a new copy of some operating system whose license is tied to the motherboard. This might run a bit more than $100. Or by "IN CHARGE" did you mean "money is no object"?
Can I see the math? Even rounding up to 30 minutes wall time on 8 CPUs would only get you a maximum of 4 hours CPU time. Unless I'm missing something?
If one builds a highly memory-constrained machine on purpose
Not all machines in use were built this year. An existing PC motherboard designed to take 32-bit CPUs can't be upgraded to 4 GB, let alone 16 GB.
(only installing 1GB RAM for a Android build machine, for example)
Then I guess that kills the project to make Android self-hosting ;-)
'In almost every case, there's only one reason for leaving APIs undocumented: We're not sure that what we have now is the best solution, and we think we might have to improve it, and we're not prepared to make those commitments to testing and preservation. We're not claiming that they're "Private" or "Secret" — How could they be, when anyone in the world can discover them? We're also not claiming they're forbidden: If you use them, your code will compile and probably run.'"
... and clearly tell people DO NOT USE them if they wont take technical measures to prevent it. They are inviting trouble with this attitude.
Does this mean they'll be releasing the source code?
A Core 2 Duo with 4 GB RAM, but you'll be waiting forever.
Just like with games, there's minimum specs that technically will work, just nobody's going to be happy with it. This is recommended specs, what the average developer could be happy with.
Noting the comment that anything less than 16GB would benefit from an SSD, I suspect the issue is that there are many small files and so the seek latency of spinning disks are what causes the pain.
It should work just fine, might take twice as long though. (Which is still not bad, our whole system takes many hours to build because we're basically building up a whole distro for multiple architectures and machine types.)
I build Gingerbread for single device (CM7.1 to be specific) pretty often and it takes 20+ minutes to build.I use quad core Phenom 955, 8GB RAM and SSD.
Somehow I doubt ICS will need considerably more time to build.
I know people are building Android in reasonable time on lesser spec machines.
This is probably a spec of shared build server they use at Google.
Having just knocked together a "cheap" machine with 16GB of RAM, I can see i hardly ever use any of it for applications as such, but my machine sure uses every byte of it (nearly) as disk cache. By Cheap, I mean basic graphics, mid range CPU, reasonable HDD size and speed (no high end stuff - just a nicely balanced dev/vm box)
I'm guessing that the "16GB RAM recommended, more preferred, anything less will measurably benefit from using an SSD" line is saying "if you can't cache it all, you're gonna need an SSD". As it churns through the build, a heap of (read only) files will be accessed over and over, and if you can keep that in RAM, things go massively faster.
Yes, I believe it's time Android had an official package management architecture + repository of some sort.
Open Source code for big projects is big and messy - no surprises there.
It's how it gets on to agile end-user devices, like smart phones that is interesting - and open source solutions have NO answer to that: most of smartphone manufacturers are busy at researching, developing and deploying fast and efficient run-time engines, often proprietary (no surprise there either - R&D costs real money) - open source stuff coming down the pipe (whether delivered by web pages or on device) just aren't efficient delivery mechanisms. Smartphones for example get their agility through highly efficient, binary and proprietary code running on limited hardware.
If people think that running such bloated code on the PC is the vision of tomorrow - dream on
So basically they need a BEAST