Slashdot Mirror
Preload Drastically Boosts Linux Performance
Nemilar writes "Preload is a Linux daemon that stores commonly-used libraries and binaries in memory to speed up access times, similar to the Windows Vista SuperFetch function. This article examines Preload and gives some insight into how much performance is gained for its total resource cost, and discusses basic installation and configuration to get you started."
This is exactly why live CDs like Damn Small Linux (and Knoppix, if you have a ton of ram) run so fast if you load the CD image to ram. Ram is fast!
Shiny. Let's be bad guys.
I read a guide on the Gentoo forums a while ago about copying different directories into ram to "preload" them.
http://forums.gentoo.org/viewtopic-t-296892.html
I never actually tried it, but I might now that I have 4gb ram! A daemon to help automate this process would be welcome, though.
I find it odd that the article "highly recommends" (RTFA before replying) installing this on a desktop Linux machine, but Vista's implementation is seen as "RAM hogging" and considered "bloat." I'm curious what sort of logical argument underlies this, as the "goodness" of preloading seems to change based upon which operating system it is implemented in. It is *almost* as if there is no logical basis, but, surely that can't be the case with the erudite, level-headed Slashdot crowd, right?
I don't know what rock you were under, but preload has been available for a while:
preload 0.2 release: 2005-09-01
And it was there before as it was packaged in Gentoo (back when it was still popular) and Suse 9.3
Custom electronics and digital signage for your business: www.evcircuits.com
This seems an odd article given preload made it into distros as a standard component for Fedora Core 1 (RHEL3). Its been around since the late 2.4 kernel series was still mainstream. What was the significance of the article? It didn't even update the numbers to a modern hardware config.
Just let it go. This pissing match over innovation serves no useful purpose.
it doesn't make GNU/Linux as *fast* Microsoft Vista.
Is this functionality available in Apple's OS X?
I never had any luck with preload the times I tried it (a year or two ago?). Nowadays I use alltray for preloading often used apps that are a bit chunky such as Firefox or Openoffice. Openoffice also has a built in preload feature...but you can use alltray anyway for the same effect.
Once you start despising the jerks, you become one.
You mean like losing the data after a few hours of no power?
now we need to go OSS in diesel cars
I had no idea, I had never even heard of it :/ Does it come by default with any distros?
innovation = first time to do something from your point of view
invention = first time to do something ever
Note how MS is always careful to point out they innovate.
*flush*
Tsunami -- You can't bring a good wave down!
The submitter is the author of the blog, and is merely paraphrasing the whitepaper written by the author of the software -- and that is two years old. Nothing new or interesting here, just someone trying to draw eyeballs to his blog.
It's actually a little different than the preload that's been in Gentoo for years. The core functionality is of course the same, but now a daemon runs that caches libraries and updates the linkage periodically. So, it can possibly give much more performance, since everything is always up-to-date. It will be standard in Hardy Heron when it comes out.
In Linux, most things never reach the x.0 stage, no matter how mature they are.
The idea of using a ramdisk (and then storing commonly used binaries, including loadable libraries) was around back in the 1980's on Sun Solaris,
/tmp..
it's an old, but good idea.
The first thing that went in though was usually
Bavarian Purity Law of Rice Krispie Squares: Rice Krispies, Marshmallows, Butter, Vanilla.
SuperFetch was one of the first things that I had to disable in Vista. I had downloaded a linux distro (a large .iso file) using Firefox, and for the next two weeks, everytime I rebooted my computer I would have to listen to my hard drive chug away for the next 10 minutes while it loaded the file into memory. (The new resource monitor in Visa is nice -- that is what helped me track down the problem).
My computer is MUCH faster now that SuperFetch is disabled. Like night and day.
Vista users respond positively toward the speed boost everytime we "Reload" their Vista. The downtime and data lost as a result of "Reload" might irritate some disgruntled users, but most of them enjoy the free break at the expense of the company.
Nothing in those Linus thingy could beat that user satisfaction. I might be bias though.
The Amiga one of the same era was very good. You had a recoverable RAM disk, which functioned the same as a standard RAM disk, but would maintain its contents on restart. That meant reboots were lightning quick, and any data you stored in the RRAM disk was still there.
Shame we haven't got back to that level of functionality.
"I've got more toys than Teruhisa Kitahara."
invention = first time to do something ever And what do you call the first time it works properly and reliably?
Only if you run out of cold spray.
I'm envisioning a sensible sort of preload program in gentoo right now:
... *snip* ...
*Preloading commonly used data, libraries, and binaries...
gcc OK
make OK
libc-dev OK
emerge OK
kernel-src OK
*Preload done, 3827K of USE Flags, 2TB of source code, and one compiler, and firefox to surf forums.gentoo.org for better use flags while you compile loaded into main memory
Photos.
In Linux, most things never reach the x.0 stage, no matter how mature they are.
This reminds me of a geek girlfriend I had... she told me she was 29.9.9.12.1 years old. But when I met her in real life, i was suprised she had a daughter... 17.1.25 years old!
When information is power, privacy is freedom.
I have a pretty good amount of memory on my current machine - 2Gb - and I mostly just never close any applications, especially with the big ones like Gimp just reusing the already open instance when you open a new file. I suspect that preload would not actually be all that useful for me in practice; I'm still goign to enable it to see if I'm wrong, though.
Trust the Computer. The Computer is your friend.
I'll say somethings not right there, I loaded Puppy on a 1 Gig USB drive, and tried it on my 7 year old homebuilt, and it was way slower than even XP.
Jerky mouse, freezing up, crashing, it brought back the days of Win98SE.
I thought it was because the USB load was "experimental".
I'll try DSL, or some other lighter weight distro and see what happens.
If it don't GO... chrome it. ~ Frank Banks
You mean Linux adapted something from Windows instead of the other way around? What's next, a sane proactor i/o api?
Not really. Caching policies like this have been around for longer than Windows has even existed. Most of the things that Linux "adopts" from Windows or Macintosh originally came from UNIX or mainframes. Even in 2008, there is hardly an original idea in any of those operating systems. And preload itself is, of course, older than Vista.
You can be mad at Vista for a number of reasons, but SuperFetch is not one of them - I have noticed a decent speed improvement because of it, and look forward to having something similar in Linux.
It's not clear to me why this should be a separate user process; what it's doing is simple enough that whatever is doing can be done directly by the kernel. In fact, I wouldn't be surprised if you could get the same speedups by simply tuning a couple of kernel parameters.
Does anyone knows the difference between the two projects? Does preload have a better algorithm for selecting the files to read? Does it also use this special syscall?
Except when you were coding something that ran away in memory, corrupted the RAD (the name of the recoverable RAM disk), destroying everything you hadn't saved for an hour or so. :)
(The Amiga didn't have an MMU originally, and even when they got it, the OS didn't support memory protection due to the shared message passing)
c++;
And then you go and run Gnome on it...
Deleted
Linux do cache all requests to the disk so I really don't see why you would need to use toram to get more speed. The only thing I can see is the ability to use the DVD drive. Sure seek times during boot are going to be eliminated, but you can do a precache stage in the boot instead.
But I always have wondered does linux cache the compressed blocks, or the uncompressed?
As far as I can tell, Kubuntu 7.10 has readahead installed by default. I can't find much information on whether preload is to replace that, or if they work together or what.
Has anyone got any insight into what's better, or if they will peacefully work together? I'd prefer faster app times over faster boot times as I hardly ever reboot.
"If you make people think they're thinking, they'll love you; But if you really make them think, they'll hate you." - DM
tried it out on my little eeepc and it definitely made a difference, on average its sped up all loading times by about 30 percent. This is especially good because i upgraded to a 2gig stick of ram but most programs hardly need that much ram and on average im left with about 1.2 gigs just sitting there doing nothing, now the ram is more productive and the loading time is noticably faster eg. firefox on a cold start without preload took 10 seconds to load before, now on a cold start it loads in 6 :). Also since the cpu is relatively slow it means fetching data and the overhead of moving it around it cut down alot. I'd love to shake the creators hand for this plucky little piece of software :) thanks!
I thought read ahead caching in the hard disk would have accomplished 90% of what this does.
I guess being smarter is more beneficial than I had thought
-- Programming with boost is like building a house with lego. It's a cool but I wouldn't want to live in it
Hours? Where'd you get that figure?
Last I heard it was a big deal when someone published that RAM keeps its data for a few minutes of no power..
Truth arises more readily from error than from confusion. -Francis Bacon
I don't know what Vista you're using, but my Vista is noticably slower than XP for everything I do (on the same computer I had XP on before) even after turning off all fancy graphics and many unused services and background apps. Maybe SuperFetch is helping and it would be even slower without it, I can't really say, all I know as a user is that the system is much slower, and it's beefy hardware I'm talking about too.
Anyway, I would rather OS developers focus on making the system faster, optimising existing code etc., than introducing bloat, and then adding layers upon layer of additional bloat on top of that to meagerly try mitigate the effects of the original bloat.
It would seem to me things like SuperFetch/preload would have an interesting "side effect" in terms of how people evaluate new software. For example, if I use Word every day, and then install something like OpenOffice and try compare the two, Word will in effect gain an "unfair advantage" - I'm going to perceive OpenOffice as being much slower than Word. (OK, that's a bad example, since OpenOffice really is pretty slow to load, but this would distort the situation to be even worse.) Same story for Internet Explorer for example (most of which loads with Windows, so it'll always be 'super-prefetched') and, say, FireFox or Opera. Most people have little patience, so they'll try the new software for a day or two, get annoyed because it takes so long to load, and go back to whatever they were using.
Regardless, preloading always has a resource cost, and that cost must at some point be carried by other applications. It's a case of trying to make the system 'faster most of the time but slower some of the time'.
What I think you'll find with systems like this is that some application developers will realise that as such a system inherently does not "fairly" allocate resources, some will start trying to "game the system" - e.g. for no other reason than this their application will install resident (e.g. via a tray icon or even some kind of service), thereby ensuring the application is "used" "every day" and prioritised.
I also suspect MS give their own apps priority in SuperFetch, but can't prove that (that's the nature of proprietary software). Hard to say though, since Word and Excel 2007 often do take ages to load on Vista, so maybe not.
Wake me up when it reaches 1.0, it's just a pre-beta release for now.
After 3 days without programming, life becomes meaningless
- The Tao of Programming
I call that version 1.0, and preload isn't there yet.
After 3 days without programming, life becomes meaningless
- The Tao of Programming
Well then you'd certainly have lost the data after a couple of hours, wouldn't you?
SIGSEGV caught, terminating
wait... not that kind of sig.
Wanna give me a quick howto on doing this on the EEEPC? I'd love to do this with mine .. any particular gotchas?
; -- the corruption of government starts with its secrets. a truly free people keep no secrets. --
Looks like I hit a nerve! Overrated and offtopic do not mean disagree!
"Offtopic -- A comment which has nothing to do with the story it's linked to (song lyrics, obscene ascii art, comments about another topic entirely) is Offtopic."
"Overrated -- Sometimes you'll run into a comment which for whatever reason has been moderated out of proportion -- this probably means several moderators saw it at nearly the same time, thought it was Funny, Insightful etc, and their scores added together exaggerate its relative merit. (A knock-knock joke at +5, Funny) Such a comment is Overrated. It's not knocking the original poster to say so, but it's probably better to spend your mod points on comments which are deserving of being moderated up."
Does any of that look familiar?
And anyway, I believe prebinding has been deprecated for Leopard as it didn't speed things up much.
You could do it in the kernel, but you shouldn't. The kernel keeps track of files using inodes and device numbers, not paths, which may be volatile between reboots (udev+kernel can dynamically assign device numbers to kernel devices, filesystems are identified and mounted by scanning for UUIDs or labels in superblocks, etc). A tracking daemon can monitor system calls and keep a small database with logical paths, access patterns, and so forth; the user-space view of activity tracks intent better so the statistics can be more meaningful.
Moreover the act of caching the file is easily accomplished by a low-priority user-space task which speculatively reads the files which may be referenced soon. In this fashion the kernel memory manager does not need to be changed in any way; we are not creating a new kernel memory pool which would need new logic under memory pressure. In the case that RAM is suddenly needed for storing application pages or an unexpected demand-loaded program binary or library, we can flush these buffered files just like any other cached file; it's not treated any differently. It's just a daemon touching files (with the hope of a benefit in startup times of applications that require them).
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
In a few years we'll all be using solid state drives, anyhow. With 10,000x fetch rate, who'll know the difference? At that speed it makes me wonder when ram will be phased out completely. What's that, you say? Fewer parts, lower overhead, less environmental impact, and smaller form factor all for the cost of a few ms access time I'll barely notice anyways? Sounds like a good deal to me.
"Sounds like 7 years ago you had no idea how to build a decent computer..."
Sounds like you're quite an arrogant asshole...
If it don't GO... chrome it. ~ Frank Banks
Note the 379M number. That is the amount of data read from disk and kept in ram. When an application needs to malloc and no completely free memory, yes it will free up those pages (it ideally picks the cache least likely to be needed again). But absolutely, disk contents are kept in disk cache, but only after load. And no, memory leaks aren't hopelessly pervasive.
What preload does normally happens implicitly during boot. It's hard to demonstrate on init scripts effectively, but log into gnome right after boot, and the disk will thrash like crazy. Log out, kill every last process of that user, log in again. It will be quite dramatic. preload aims for that subsequent experience without the pain of the first.
So what preload brings is simple, and all that has to happen is simple, know which files are relevant to typical usage ahead of time, and be aggressive about 'cat file >
Linux implicitly aids this, but the user interface side still subjectively 'feels' bogged down because it won't proactively load things it doesn't know you'll need, despite the ability to derive this historical data in user space. If preload takes idle time (let's say, for example, while services with arbitrary sleeps and while waiting for username and password) and proactively gets cache populated, it is more IO work in the aggregate (disk will be hit up for things that will never be needed), but it will feel smoother out of the gate.
XML is like violence. If it doesn't solve the problem, use more.
was her daughter hot?
Now that you mention it, she asked me to fork, but her mother was monitoring all processes - I was rebooted, her daughter had her privileges revoked, and ~/ was made readonly.
As fun as it is to poke fun at MS for updates incurring reboots, most modern distros end up issuing a kernel update every couple of weeks, so linux is not immune. Also, I have some misbehaving drivers on my laptop that currently preclude suspending, and thus shut down when I have to travel with it, so it's often the case my linux laptop incurs resets that flush cache. Finally, the simple reality is if you want to build a platform for the masses, you have to deal with some users that even with everything lined up to avoid cache-dropping reboots, will want to shut-down or reboot because they feel like it (or know suspend on their particular hardware still draws a few watts, and want to save that). OO.o whether you like it or not is a significant piece of that equation as it has the largest mindshare among the free suites, afaict. As much as I find it critical to use pipes in the shell to make use of various programs, it's not something I relish trying to explain to a complete novice.
Meanwhile, if you close the app every time without rebooting, you won't be so bad off in the case that preload aims to help. Firefox starting once induces it's disk needs to be cached, starting it again should have a high disk cache hit rate. Leaving the app open just to avoid an eventual startup penalty can be a waste of memory and work against performance if you starve memory available for disk caching.
XML is like violence. If it doesn't solve the problem, use more.
Cowardly troll boy said: "Sounds like those things aren't mutually exclusive."
Thanks for sharing that it runs in your family, but we all can surmise that from your callow comments.
If it don't GO... chrome it. ~ Frank Banks
You could bugfix the two-year-old software, do a release on SourceForge, announce on Freshmeat, and THEN post on Slashdot. I bet that would not only boost readership, but it would be a readership that appreciated your efforts. All it would take is a relatively minor bugfix to be a real release. Run splint over it, or put it through dmalloc, fix compiler errors for gcc 4.2, or a dozen other things. A few minutes work, perhaps.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
AmigaOS booted as quickly as it did from floppy disk because the core elements of the kernel were in ROM, along with some key services. Mind you, Amiga learned a few tricks from UNIX.
no gotchas if you are using ubuntu or eeexubuntu, just follow the instructions on the preload homepage.
Even if very strictly only talking about reboots that go to firmware, depending on your system and what kernel you are changing from/to, the new kernel may do something odd like hang or some hardware/driver interaction might get confused by the situation (if the driver was unable to unload, and so the case of two driver loads coming without a driver unload or intermediate firmware initiation confuses some set of devices gravely, a few instances to the point of panic). As far as I can tell, most common cases kexec works well, but it isn't universally safe even if every driver is unloaded and precautions are taken before the change.
/boot on some trivially accessed media (i.e. a USB key) and / could be on any number of inconvenient devices for the firmware to get at. I just think the frustration system firmware is blown way out of proportion, and instead of insisting of doing away with it, demand features that facilitate things like fast boot and exotic boot loader configurations. A BIOS that would skip straight to a USB port (making all the relevant assumptions along the way) and go would go a long way paired with a USB flash storage device.
But I wouldn't dictate avoiding firmware as being *that* significant, *particularly* on desktop platforms with relatively fast BIOS. From a user convenience factor, kexec still means the entire process space/memory/everything goes away. You start init over again with a new kernel and initrd, just like you would from grub. It's not exactly a consolation that a kernel update will not get you into firmware if it still resets everything your doing like a full system reboot would. I can see with some card firmwares being annoyed with startup time, but it's generally not that bad. If really disconcerting, a number of cards/BIOSes let you disable the firmware for the card and the linux driver will still work, you just need
XML is like violence. If it doesn't solve the problem, use more.