The New Linux Speed Trick

Brainsur quotes a story saying " Linux kernel 2.6 introduces improved IO scheduling that can increase speed -- "sometimes by 1,000 percent or more, [more] often by 2x" -- for standard desktop workloads, and by as much as 15 percent on many database workloads, according to Andrew Morton of Open Source Development Labs. This increased speed is accomplished by minimizing the disk head movement during concurrent reads. "

Comments

I've noticed it...

[Anonymous Coward]

I'm having trouble getting ACPI working in my laptop in the 2.6 kernel (it's a bad implementation on the part of my laptop). The 2.4 series used to work (sometimes) so I installed Mandrake's 2.4 kernel and 2.6 kernels on my laptop. Using 2.4.x again was like switching to a horse and buggy from a sport-cars; KDE was that much faster with the 2.6.x kernel running the show.

Re:I've noticed it...

[Anonymous Coward]

Well, upgrading will not automatically give you a faster computer, not for all hardware. On my old Compaq Armada laptop (450 mhz, 128 mb), Mandrake 9.2 using the 2.4 kernel and KDE 3.1 was slow but usable... same with Win2000.

I upgraded to Mandrake 10.0 Community Edition (you know, post-beta but not officially released...). It has kernel 2.6 and KDE 3.2. Looks nice, but now it is slow enough to be unusable. Mozilla takes 20 seconds to start, HD swapping furiously.

I believe I read that the new kernel does take up some more memory, perhaps my laptop has reached its limit?

I am currently learning to compile the kernel, so I'm going to try to do a low-fat version with all unneccesary file systems, drivers et al removed. I should probably use just about anything than Mozilla and KDE also. Firefox plus IceWM perhaps.

Re:I've noticed it...

[atomic-penguin]

Be careful, Mandrake doesn't like this. You will break RPM dependencies.

Not trying to troll here. I maintain a computer lab, and after customizing a kernel, I actually broke things that were not even needed for that system. I turned off something and ended up without network support. I turned off the 77 Mb initrd, not needed on a system with 128 Mb of system memory and broke it. I fixed it, eventually...

I haven't touched the Mandrake kernel since, it was just too much hassle.

Re:I've noticed it...

[gazbo]

Well hold on a minute: this is talking about the speed increases from a particular subsystem - the IO scheduler. Upgrading from 2.4 to 2.6 changes a hell of a lot more than just that, so your speedup could be from any number of things.

From the sound of it you're talking about perceived speed for a desktop user, as opposed to measured server throughput. If this is the case, I imagine the biggest speed increase comes from the fact that (I believe) 2.6 offers far lower latency in the kernel, allowing it to be preempted in more cases. This will make a desktop environment far more responsive.

Re:I've noticed it...

[arunarunarun]

You can fix it. There's a kernel patch that allows you to use a fixed ACPI DSDT rather than the original one, and there are fixed versions of the DSDT available, put up by people who've fixed it. You can even do it yourself using Intel ACPI tools.

I did this on a Compaq Presario 2100 laptop. Lookup The ACPI4Linux project.

Re:I've noticed it...

[atomic-penguin]

Yes, for some reason it was default in Mandrake 9.1, I don't know why. Mandrake is not my top choice for any practical use. I personally use Slack on my home machine and production servers. But, I was disgusted by some of the defaults in the Mandrake kernel.

Most of you are missing the point here...

I did not have trouble installing a kernel on Mandrake, I had a problem "Customizing" it. Note Parent's post


I am currently learning to compile the kernel, so I'm going to try to do a low-fat version with all unneccesary file systems, drivers et al removed...

I was just warning the parent that strange things may happen when tinkering with these NON-Vanilla kernels

Cache?

[Anonymous Coward]

Whatever happened to cache. If you can anticipate the head movement surely you have already read the data before and it should be in the cache????

Re:Cache?

[4of12]

IIRC, the OS is supposed to do some caching (in the old days a sync command helped flush buffers onto disk before shutdown), but it's not an explicit kind of think like this persistent memory-mounted filesystem, which I've always thought was interesting.

If you ever think about how inefficient it would be for the system to go read /bin/ls every time you typed the ls command you could see where caching is a damn good idea.

Doing read-ahead, write-behind and maintaining coherency isn't easy, from what little I understand.

Re:Cache?

[Erik Hensema]

Sure, and both Linux 2.4 and 2.6 do caching and read-ahead (reading more data than requested, hoping that the application will request the data in the future).

The I/O scheduler however lies beneath the cache layer. When it's decided that data must be read from or written to disk, the request is placed in a queue. The scheduler may reorder the queue in order to minimize head movements.

Also, 2.6 has the anticipatory I/O scheduler: after a read, the scheduler simply pauses for a (very) short period. This is done in the assumption that the application will request more data from the same general area on the disk. Even when other requests are in the I/O queue, requests to the area where the disk's heads are hovering will get priority.

While this increases latency (the time it takes for a request to be processed) a bit, throughput (the amount of data transfered in a time period) will also increase.

It did take a fair amount of experimenting and tuning in order to make the I/O scheduler work as well as it does now. However there still may be some corner cases where the new scheduler is much slower than the old.

Re:Cache?

[p3d0]

Well that's the dumbest question I have read in quite a while, and I'm willing to burn my karma to say so. No wonder you posted as AC.

Obviously there is caching and prefetching and all that good stuff. Give the kernel developers some credit. This is about optimizing what the OS does when all those things fail and you really do need to go to the disk.

Re:Cache?

[mp3phish]

Not really...

You can know approximately where on the physical disk the data is being stored by its address. If you know enough about the geometry of the disk, you can schedule what they call an "elevador sort" which means they queue up reads, sort them by address, and then read them all at once, in order they are on the disk.

Say you queue'd up 5 reads. All randomly placed on the disk. If you sort them from the center of the disk to the edge of the disk then they can be read much faster and with less movement of the head.

I don't know if this is the type of sorting they are doing, but it has been done in the past by scsi controllers, and this might just be moving this logic into the kernel.

Re:Cache?

[drinkypoo]

In the old days, a sync command was pretty much the only thing responsible for flushing buffers to disk. Here's an amusing anecdote for you: The proper way to do a Xenix shutdown used to be sync and then haltsys. However, if you typed fairly fast, or if there were a lot of buffers to flush, the halt would complete before the sync, so you typed:

sync
sync
haltsys

So that while you were typing sync the second time, the system was still writing them out. This is of course necessary only because sync returns before the sync operation completes.

SCSI

[Zo0ok]

Dont SCSI drives do this themselves?

Re:SCSI

[NfoCipher]

SCSI is still so expensive and an aging technology. The idea is to get more out of the cheap IDE drives.

Re:SCSI

[B1ackDragon]

Since it mentioned that the OS is keeping "per-process statistics on whether there will be another dependent read 'soon'", I really doubt the drive controller would even be able to do that, much less want to.

Re:SCSI

[pararox]

As a college student, I feel proud to say I've access to a quad-Xeon SCSI machine; this bad thing truly burns.

I run WebGUI on this machine, which recieves some 3 and a quarter million hits per month. Nothing to raise the eye brows at; but check it: on this machine the average uptime value is some 0.80. My personal (p3) machine, running a BBS, mail, bittorent, and web service maintains a constant 1.3+.

I've guaged the importance of SCSI drives in the equation via a (sadly) messy, but soon to be SourceForged Perl program. The result, confirming that which I've heard repeatedly, is that SCSI drives truly make the difference.

Re:SCSI

[DuSTman31]

Yeah, I think so. IIRC it's called tagged command queueing - the drive can have multiple requests pending and instead of doing them first come first served, they're fulfilled in order of estimated latency to that point.

I believe Western Digital's recent Raptor IDE drives have the same feature.

The benefit of this seems contingent upon having multiple requests pending, which AFAIK is hard on linux as there's no non-blocking file IO. To me, this reads like a workaround for that.

Re:SCSI

[KagatoLNX]

ATA is basically the SCSI protocol (the good part) over IDE. There's a reason why some SATA drives appear as SCSI adapters under Linux.

Expensive, yes. Aging, no. Ten years ago people said SCSI was the future. Now everyone runs it, they just don't know it.

IDE in its original form has never been able to keep up with a 10k RPM (or higher) disk.

I think what the parent post is alluding to is Tagged Queuing. Tagged Queueing allows you to group blocks together and tell the drive to write them in some priority. That sort of thing is used to guarantee journaling and such. Interestingly, the lack of this mechanism is why many IDE drives torch journalled fs's when they lose power during a write--they do buffering but without some sort of priority. You can imagine I was pretty torqued the first time I had to fsck an ext3 (or rebuild-tree on reiserfs) after a power failure.

The reason that the kernel helps even with the above technology is that the drive queue is easily filled. Even when you have a multimegabyte drive cache and a fast drive, large amounts of data spread over the disk can take a while to write out.

This scheduler is able to take into account Linux's entire internal disk cache (sometimes gigs of data in RAM) and schedule that before it hits the drives.

Re:SCSI

[awx]

I'm sure you mean load value, not uptime value. An uptime of 0.8 days isn't really that impressive...

Re:SCSI

[Mike McTernan]

which AFAIK is hard on linux as there's no non-blocking file IO.

Surely if you have 10 threads of execution, or 10 processes for that matter, all waiting to read and write to the disk it doesn't matter if the IO is non-blocking or not; the kernel or HDD controller should still be able to optimise the order in which IO requests are served to minimise the seeking required.

I'm just suprised that this technology wasn't already in use... I can remember reading about the elevator algorithm as an example of a disk scheduler at university some time ago. Seems obvious, although perhaps difficult to implement well though - good job to the Linux folk!

Re:SCSI

[jcupitt65]

Yes, Linux has always had (variations) on the elevator algorithmn.

This is different: the scheduler isn't trying to minimise head movement for a list of pending read requests (which is what elevator does), it's gathering statistics about the IO behaviour of each process and trying to guess in advance without being asked what each process will ask for next.

If it guesses right, the data will already be in cache when the process does a read() and the request will succeed instantly.

Re:SCSI

[fizze]

Tagged Command Queuing is supported since the early days of SCSI. modern drives support up to 255 entries. So, from my understanding, the drive optimizes it for low seek time and high throughput, so this new kernel feature of 2.6 might either double the effort on SCSI drives for the same output, or, what I presume, these two features might even conflict. sure, TCQ can be disabled, but why implement in software what already exists (and works) in hardware ?

Re:SCSI

[Anonymous Coward]

An uptime of 0.8 days isn't really that impressive...

It's obviously been a long time since you used Windows.

Re:SCSI

[roystgnr]

The benefit of this seems contingent upon having multiple requests pending, which AFAIK is hard on linux as there's no non-blocking file IO.

What do you call it when you pass O_NONBLOCK to the read function, then? If you're thinking of asynchronous I/O, Linux has that too, although I don't know if it's as portable & standardized.

Re:SCSI

[Rich0]

The interesting thing is that modern elevators do this as well - at least the fancier ones do.

If you have a 30 story building, you can either put in dumb elevators which fill 3/4ths of the building to meet demand, or you can put in a much smaller number of elevators using techniques like express elevators and using software which keeps track of usage patterns and puts elevators on floors before somebody even hits the button...

Re:SCSI

[Manfre]

Your post seems more like a plug for your site than anything else.

Re:SCSI

[Rich0]

Because not everybody has a SCSI drive?

I don't get the impression that these features actually conflict though.

What does TCQ require? Simply the ability to know what requests are outstanding and where the drive head is. The kernel has to keep track of the former anyway, and keeping track of the latter isn't that difficult if you know what the last request was.

A cheap IDE drive usually doesn't support TCQ. So which makes more sense - increasing drive expense by putting smarter electronics on the drive, or using that super-smart CPU which is currently just sitting aroudn waiting for the disk anyway?

Sure, on a high-end server it makes sense to invest in the fancy hardware. However, adding a SCSI controller and fancy drives to a PC can add $100-200 to the cost of a system. For a $600 system that is a huge increase. (And $600 can get you a lot these days - I spent that to obtain an Athlon 64 system - borrowing only a flopppy drive, CD-ROM, and video card from a more ancient computer (gee, all three of those probably cost $50 total (I don't need 3D)).

Re:SCSI

[shic]

A question... I've asked this before without an appropriate answer.

If I'm writing a user-land program which memory maps a large file, modifies it in memory - then uses msync() to write to disk - what can be safely assumed?

• Can I assume that the thread (or process?) calling msync() will block until the data is successfully written to stable storage?
  
• Can I assume that in a sequence of calls msync()_1 .. msync()_n that the writing of the data associated with msync_i implies that for all j<i that msync()_j has successfully completed?
  
• Do I need to consider the specific storage hardware before I can draw any conclusions?
  
• Can I influence the order in which pending writes are flushed from user space?
  

Re:SCSI

[Qzukk]

looks like as long as you use MS_SYNC as a flag, on a file on local hardware, you can trust that the data is at the harddrive, if not on it (thanks to the drive cache). Not sure what happens if you try this on a network file system, whether it forces the hosting computer to flush to disk, or if it only forces the local computer to flush to the host.

As for order of pending writes, I don't think you get to have a say on any particular writes, but you can sync after writing to commit everything so far.

See also man 2 sync.

Re:SCSI

[Gr8Apes]

Expensive? No. Not really. Unless you want 75GB or larger low profile drives. I bought 5 new HH 36GB 7200 rpm drives for $100. Match that with a Mylex ExtremeRaid 1164P 3 channel controller I got for $50 (yep, egay on that one;), and that makes a really nice fast RAID5 ~140GB array. And it's expandable. (just think, (5) 8 drive arrays striped ~1.2TB for $550 with virtually no CPU overhead!)

The downside? Noise and heat. 5 scsi drives generate a lot more heat and noise than one drive. The data transfer and latency can't be beat for running a DB or doing movie editing. The maximum throughput from the controller's 3 channels is only 240MB/s, but there's also the 64MB cache on the controller.

Last time I checked, IDE (ATA, SATA, whatever) couldn't even come close to this. Granted, I have not checked out the newer SATA raid controllers, but they cost far more than an equivalent SCSI3 setup, so why bother at this time (if cost is an issue, this is not always the case).

Re:SCSI

[jesup]

ATA is definitely not SCSI-over-IDE.
ATAPI is SCSI-over-IDE however.

I wrote the IDE/ATA drivers for the Amiga. The Amiga SCSI drivers accepted "SCSIDirect" commands from applications. Internally, all IO commands were converted to SCSIDirect commands for execution. To implement ATA, I added a SCSIDirect->ATA translator (which wasn't that hard - about 3 weeks from start to working, booting system - and I implemented just about all SCSI commands even semi-reasonable (all of CCS I think, plus quite a bit).

Doing it this way made implementing support for ATAPI CDROMs (something I did as a contract after Commodore folded) Very Easy. :-)

Re:SCSI

[silas_moeckel]

I beleive you can trust that it's on disk localy unless your running a hardware raid controller with battery backed up ram cache (nearly as good as disk). There are some broken raid controlelrs out there that will do that without the battery backup. This has allways been an issue with SMTP servers as by RFC they have to flush to disk before they respond with message received. MS Exchange was broken in this regaurd last I checked.

Re:SCSI

[Martin Maciaszek]

So which drives implement Tagged Queueing correctly? Is there some list with drives that I should avoid?

Re:SCSI

[Afrosheen]

And the last time I checked, those 7200 rpm drives have little advantage over ATA or SATA drives of the same speed rating, unless they have large caches or something.

There's no reason to go scsi in an array like this unless you're using 10k rpm drives or higher.

Re:SCSI

[Afrosheen]

Personally I prefer elevators that return to the ground floor after a preset amount of time.

Nothing is more frustrating than waiting for an elevator that's up on a high floor to come down, only to find that nobody is on it. It's one of those 'wtf' moments.

Re:SCSI

[Fulcrum of Evil]

Not sure what happens if you try this on a network file system, whether it forces the hosting computer to flush to disk, or if it only forces the local computer to flush to the host.

Depends on the server - you can request it, but the server isn't obligated to comply.

Re:SCSI

[TheLink]

Actually 200GB 7200 rpm ATA drives could be faster than 36GB 7200 SCSI drives IF you are willing to sacrifice 164GB of capacity.

Using just 36GB out of 200GB will make your seek times a bit lower. If you put the 36GB partition in the outer and faster region of the disk sequential throughput would be significantly higher.

But, hey just get WD raptors if they're within your budget (and 10K-15Krpm SCSI drives aren't).

Re:SCSI

[Chester K]

which AFAIK is hard on linux as there's no non-blocking file IO.

There certainly is a fully-featured asynchronous IO API in Linux 2.6.

Re:SCSI

[TheLink]

If you're talking about load of 0.8, AFAIK load is roughly the average number of processes running or ready to run over a certain period.

So comparing machine performance by using load values e.g. 0.8 or 1.3 can often be meaningless. Unless you know they are running the exact same loads and the faster machine just finishes stuff faster and thus there are fewer concurrent processes and thus a lower load.

That said burning is a good term for quad-Xeons especially the top end ones...

Re:SCSI

[TheLink]

Thing is, whilst the CPU is waiting for disk, it could be waiting on you instead and drawing your UI and other stuff.

But if too much of the CPU is tied up pretending to be a drive controller, you may find it annoying enough to buy a smarter drive controller.

Some people get dual CPUs - that way one CPU does the UI and the other becomes the drive controller ;).

Why SMP stuff seems more responsive is because schedulers aren't perfect AND throwing two cpus on one task doesn't always make things faster so you might as well use the other CPU to do other things, whereas if you only have one CPU, throwing 100% at a task does make it faster than if you just threw 50% at a task.

So, if you only have one CPU, sometimes it's hard to tell whether to let it spend more time figuring out disk stuff, or to let it do other things. Whereas most drive controllers ain't gonna be drawing your UI :).

Re:SCSI

[jesup]

Tagged Queuing in SCSI is a good thing.

Trying to do all the reordering in the OS (as suggested in several posts here) seems like a good idea, but ignores some issues:

1. Disks aren't a uniform array of blocks, and even if you have disk geometry it's almost certainly at least simplified, and probably a total lie. (you can query a SCSI drive for the next "slowdown", but what that means is ill-defined, and not that useful anyways.)
2. Because of (1), you don't know when blocks are on the same track or not.
3. it may be more efficient head-movement-wise not to switch directions.
4. When reordering requests, head position matters, but so does rotational angle, and the OS scheduler has no idea about that.
5. Hardware raid. Now you really don't know what the geometry is or what the IO scheduling parameters are. Not to mention letting the raid controller make use of multiple drives efficiently for small requests.
6. This PDF shows that tagged queuing in SCSI drives is a win even with host IO ordering, when the IO stream is random or fairly real-world (lots of independant streams of reads and writes). There's minimal or no win for sequential IO.

Tagged queuing in SATA/etc drives is a step in the right direction, though last I checked it wasn't equal to TQ in SCSI. Native Command Queuing in SATA will probably give similar performance to TQ in SCSI.

This PDF on Native Command Queuing is even more interesting.

Re:SCSI

[the_skywise]

I worked for a phone company in a large downtown office building that employed thousands. The building needed a new elevator system so, to save money, they had an experimental elevator system installed.

Instead of pusing an up or down button and waiting for an elevator, you had a number pad with an LCD display. You punched in the number of the floor you wanted to go to and the LCD would display the letter of the elevator you were assigned to. There were no floor controls inside the elevator. The system would cache people onto floors, maximizing elevator efficiency, right?

Well, in theory... but not in practice.

The building wasn't populated evenly on all the floors. Of a 20 story building, 4 floors were in heavy use (customer service departments) 2 were used only for storage, 2 were for the switching lines and the rest of the floors had typical light office usage.

So in rush hour in the morning the system has a notoriously bad habit of maxing out one elevator at a time for the 4 customer service floors before allocating the next. Even MORE ironic was that the program would get into a loop of cycling the same 2 elevators to serve the same heavy use floors. (One would be going up as the other would be just finishing its cycle and the program wouldn't allocate a third elevator until the one coming down was filled.)

So you'd walk into the building at rush hour in the morning (when everybody should be wanting to just go UP) and there's a bank of 10 elevators, 5 to a wall and there'd be about 40 people crowded in front of 2 elevator doors for about 5 minutes waiting to board.

But it gets WORSE! Alot of people would see friends they knew who had already "punched in" and would just get in line with them without punching in themselves and screw up the whole count.

Then you had the "smart" people who would "punch in" a floor many times to get the system to allocate a third and NOT cache them in with the other loads.

At times it was alot of fun just playing with the elevator system than actually doing work.
(one more interesting factoid... someone writing the program must've taken the above into account because there were times when I would work late and the coworkers and I would leave at 10pm from the 16th floor. We'd both punch in within seconds of each other and get TWO elevators.)

1,000 percent?

[lonegd]

That seems rather high. Either something was broken/badly coded or someone's been adding a couple of zero's ;)

Linux Devices has an article on the 2.6 network features here http://linuxdevices.com/articles/AT7885999771.html

Re:1,000 percent?

[aastanna]

Seems OK to me, that's a 10x improvement, and that was the theoretical high end example. Since they said it would commonly increase speed by 2x, and 15% for databases, it seems right in line.

I suppose that since database data is generally grouped together and read in a big chunk there's less room for improvment.

Re:1,000 percent?

[gowen]

My guess is that it's a fairly specific, non-standard load that will garner a 1000x gain

My guess is that you haven't spotted that 1,000% is not 1,000x. A 10-fold increase isn't completely implausible for a workload whose read pattern matches the assumptions built into the anticipatory scheduler.

Re:1,000 percent?

[tonywestonuk]

Isn't 1000%, 11x?
15% = 1.15x
100% = 2x
200% = 3x
300% = 4x ..
900% = 10x
1000% = 11x

a % = (a+100)/100 x

Re:1,000 percent?

[maxwell demon]

1,000 percent is 10x, but 1,000% improvement, being improvement by 10x, is 11x as good.

Just as 50% is half, but 50% improvement is three halves as good.

Re:1,000 percent?

[warrax_666]

Well... a 1000% = 10 (by the definition of %), but a 1000% increase is equivalent to multiplying by 11. (As you correctly stated).

Re:1,000 percent?

[rikkus-x]

But it's speed reduction, so...

100% = 1/2 the time.
200% = 1/2 of 1/2 the time, which is 1/4 the time.
300% = 1/8 the time.
1000% = 1/1024 the time.

Which is a 1023/1024 improvement, or only 0.999x, so disk access is in fact slightly slower!

Yes, I'm really bad at maths.

Rik

Re:1,000 percent?

[TheToon]

Though as databases (at least DB2) use this technique already, maybe databases will not see any improvements at all. Prefetching, sequential I/O detection, elevator seeks, aynch and parallel I/O... already in there, esp. on raw devices. On files you still have fragmentation issues. Maybe it will help there.

Re:1,000 percent?

[diegocgteleline.es]

1000x is not a inveted number, just run "find /" while you're running "cp /dev/zero /tmp" (all on the same disk). Measure how many time takes in 2.4 and how many time it takes in 2.6 with anticipatory scheduler. Nobody has mentioned, BTW, the CFQ scheduler. CFQ is the coolest thing you'll find for desktops. AIUI, you've a list of queues for each process, and you just round robin on them. This provides *extreme* fairness for desktops. In fact, Jens Axboe used that to provide "io priorities". That means you can give "ionice -20" to a process and that process will have more disk bandwith than other processes. Or, in other words, you can run your updatedb cron jobs with a lower io priority. So you won't notice it while you're using openoffice/mozilla because the updatedb cron job is getting less io priority... This makes linux a bit more "realtime OS", since we'll be able to say "I want that apache has been guaranteed 80% of the disk bandwith". And that will be true _always_, no matter if your users are doing stupid things....

Re:1,000 percent?

[jargoone]

Yes, I'm really bad at maths.

And apparently Englishes, too.

Cool

[JaxWeb]

It seems there are two IO modes you can choose from, at boot time.

"The anticipatory scheduling is so named because it anticipates processes doing several dependent reads. In theory, this should minimize the disk head movement. Without anticipation, the heads may have to seek back and forth under several loads, and there is a small delay before the head returns for a seek to see if the process requests another read. "

"The deadline scheduler has two additional scheduling queues that were not available to the 2.4 IO scheduler. The two new queues are a FIFO read queue and a FIFO write queue. This new multi-queue method allows for greater interactivity by giving the read requests a better deadline than write requests, thus ensuring that applications rarely will be delayed by read requests."

Nice, but this is making things more complex. I admit I'll just keep all kernel settings at wherever Mandrake sets them as. Will other people play about and specialise their system for the task that it does?

Re:Cool

[Alrescha]

"and there is a small delay before the head returns for a seek to see if the process requests another read."

It's early, but did read/write heads suddenly develop intelligence while I was napping?

A.

Re:Cool

[CyberGarp]

It may be more complex, but it works damn well. Squeezing an extra 30-40% out of my laptop doing disk i/o has been really worth it. Especially since I've been editing and processing 600M files on it lately.

Re:Cool

[MrIrwin]

"I admit I'll just keep all kernel settings at wherever Mandrake sets them as. Will other people play about and specialise their system for the task that it does?"

Perhaps that is why the default setting is the one indicated for desktop users.

And yes, if I were using a Linux box for specific server tasks then I would tweak the settings to get a bit more performance out of it.

Re:Cool

[PyromanFO]

This troll comes up in any thread that has anything to do with Linux at all. Who the hell said anything about asking people to choose? This is for developers and hackers to mess with. The distro you're using will choose for you, just like Microsoft chooses what Windows drivers you have loaded by default. Does every person who runs a Dell Windows machine have to decide what version of the driver to use? No Dell installs it for them. However power users can install newer/beta drivers if they want. Same thing here, power users can enable this if they want. If not you'll never have to know about it or touch it.

Sorry for biting on the troll but I felt like explaining it.

Re:Cool

[Jeff DeMaagd]

It sounds a lot like software version of the tagged command queueing that SCSI and high-end ATA drives have. I think having it in the OS would sort of defeat the drive's feature but the OS has more memory and horsepower available to it to reduce average access time.

I think this would work to minimize the impact of a slow access drive in a heavily multitasking system too.

Re:Cool

[minus9]

And do you honestly think it's going to ask them?

Re:Cool

[dave420]

At the moment linux distros ask you everything from how you take your coffee to which side you dress. It was more of an illustration than an example. Nice of you to take it seriously, though ;) If Linux is going to dominate the computer world, it has to appeal to as many people as possible. The linux community can't bellyache about MS's dominance if this is how the community spends its time. Don't try to run before you can walk. Super-fast IO doesn't matter if the software isn't on anyone's machines. First, get people using it then improve it as they demand.

Re:Cool

[dave420]

I love slashdot. I'm talking about the story, and it's modded as offtopic. I think the words "to my bias" should be added to the mod options :-P

"offtopic to my bias"
"troll to my bias"

etc ;)

as the only way you get modded accurately is if you're in the same camp as the moderator. I'm clearly not.

Re:Cool

[Des Herriott]

You are a troll. Either that, or just plain ignorant.

"if energy is wasted on projects like this" ? Optimisation of the low-level IO scheduler is most certainly not a waste of energy. As the parent-poster said, users don't need to fiddle with this at all, but power-users and sysadmins can tune it to get the most out of the machines if they so choose. This is much more relevant to the enterprise market than it is to home users with a Walmart PC.

So what if "Joe Public" doesn't want this feature? If there are people who do want it (and this isn't some of kind of eye-candy add-on), then it's damn well worth implementing.

You seem to be under the impression that someone working on projects not important to you is somehow wasting their time. Would you like the smart people who are doing this good work to drop everything they're doing and design pretty user interfaces for Joe Public instead? Maybe you should tell them that directly, although I suspect you'll be told exactly where to go.

Re:Cool

[dave420]

I'm saying is there seems to be 2 camps in the linux world - those hell-bent on bringing it to the end user, and those hell-bent on improving it for techies. Most people out there aren't techies, so where does that leave linux? A highly-technical, computer-literate-user only operating system.

It's fine - you can say/think what you want about me. Just watch as linux goes precisely nowhere on the desktop because of things like this.

What's useful to techies isn't necessarily useful to the home user. Until linux is useful to the home user, bill gates gets the last laugh. You can't seriously expect most of the windows users out there to format c: and get linux in its current state??

Re:Cool

[julesh]

there seems to be 2 camps in the linux world - those hell-bent on bringing it to the end user, and those hell-bent on improving it for techies

Yes there are. And this is a good thing, for two reasons:

1. because the 1st of those groups are techies, and if Linux isn't good for techies, they won't be able to do what they need to do for the users.

2. because Linux's predominant use at this time is in server environments. Most servers are run by techies.

What's useful to techies isn't necessarily useful to the home user

That's true. But neither is it a problem for the home user. There is plenty of space for both extremes here, as long as they learn to work with each other.

Also note: Windows has many capabilities that are only useful for techies. You just probably don't know about them. Have a look at all the interesting things you can do if you mess around with the registry, particularly in HKLM\Control\CurrentControlSet\Services\*\Paramete rs

Re:Cool

[dave420]

trust me - I know how to get windows to do things :)

I know what you're saying, but if linux continues being techie-oriented, it will never get onto the home desktop. Windows isn't techie-oriented, but it can be used by techies. Linux, on the other hand, is techie-oriented and can only be used by the few lucky non-techies who has a linux install that doesn't need fixing.

I'm just saying that people are bitching that linux isn't on every desktop, and the answer is right in front of their face. And no, it's not microsoft. Or sco. Or IBM. Or sun. Or any other scapegoat-of-the-month. :)

Re:Cool

[Des Herriott]

Your basic problem is that you somehow believe that these two camps are somehow mutually antagonistic, and effort put into improving the inner workings of the kernel somehow detracts from the "end-user experience". This simply isn't the case.

In general, kernel hackers don't write pretty GUI's or design highly usable interfaces, and HCI experts don't optimise low-level scheduling algorithms. These are orthogonal parts of Linux, and your belief that improving the kernel's low-level efficiency somehow makes the end-user experience worse is frankly ridiculous.

Of course maybe you're just a very good troll and you're successfully wasting mine and others' time :-)

Re:Cool

[Fyndo]

Err, actually, though, a huge focus of this work has been to make the computer more responsive to interactive tasks, making it seem faster and smoother, without making the OS suck in a server environment. It's been done precisely in order to make linux on the desktop work better, the desktop distros can set the thing to be smooth and responsive to desktop users, the server users can make it churn out SQL queries faster, and everyone wins, what's the problem?

Re:Cool

[chez69]

I don't want to start a holy war here but how is it with 12 meg files?

Yay

[Anonymous Coward]

I remember when this feature was added to NT Service Pack 4. Performance on the enterprise database server I was managing increased something like 45%.

It's nice to see smart features like this added to Linux.

Re:Yay

[Wiz]

Errrr, this is nothing to do with the filesystems involved though! It'll work just as well on resierfs as ext2!

File optimisation as you describe it is completely DIFFERENT!!

Re:Yay

[dustmite]

I assume you mean Win98 ... what has Win98 got to do with the price of eggs? NTSP4 has nothing to do with '98, which came out in August 1998. '98 had notably slower I/O performance than NT/2k.

Re:Yay

[dustmite]

Ah, sorry, didn't see the modded down parent. Still, Windows 98 came out in August '98.

Why not combine those two methods?

[maxwell demon]

Is there any reason why the prediction code (anticipatory scheduler) and the extra queues (deadline scheduler) couldn't be combined in a single scheduler to give us the best of both worlds?

Re:Why not combine those two methods?

[mirko]

what would you have expected the kernel 2.8 to bring you ?
</joke>

Basically, I think this is like the windows system settings : you either privilegiate front end services (GUI) or back end services (apache, etc) but you cannot do both because some would be optimized for reactivity, the others to handle the workload... like a ferrari and a truck... this doesn't work nor excel in the same way.

Re:Why not combine those two methods?

[Anonymous Coward]

I believe that the anticipatory sched uses the model of the deadline sched. See "Linux Kernel Development" by Robert Love.

Re:Why not combine those two methods?

[Anonymous Coward]

like a ferrari and a truck... this doesn't work nor excel in the same way
Does Not?

Re:Why not combine those two methods?

[sylvester]

What sort of George 'verbal abortion' Bushism is " privilegiate "?
You (presumeably) wanted just 'privilege' as a transitive verb:

you either privilege front end services (GUI) or back end services (apache, etc)

Given the name "Mirko", I would imagine that the grandparent was Finnish. Finnish is not an indo-european language, and has very very different suffixing rules from English. They commonly then derive interesting suffix-forms of words.

Nice of you to point out the mistake like an ass, though. (Yes, just like I'm doing.)

-Rob, a Canadian in Finland

Re:Why not combine those two methods?

[chefren]

You can form really nice, short and clear words like: epajarjestelmallistyttamattomyydellaansakohan. Iam not going to even try to disassemle and translate that :).

Re:It kindof Early to be Slashdoted

[stecoop]

Alright Its back... Whew its Yahoo page - that would be bad if Slashdotters could freeze that page.

Amiga Disks

[tonywestonuk]

When I had an Amiga (aroung '91ish), even though It was fully multitasking, I learnt to never open any app while another was loading. If you did, you could hear the disk head moving back and forward between two sectors on disk every half second or so, slowing both app launches to a crawl. Waiting until one loaded, and launching the second was many times faster.

I've always wondered why there wasn't something in the OS to force this behaviour, Ie, making sure that App 2 access to the disk is queued until app 1 has finished. Isn't this one of the reasons Windows takes ages to boot? (many processes all competing for the one disk resource?).

Re:Amiga Disks

[jtwJGuevara]

Isn't this one of the reasons Windows takes ages to boot? (many processes all competing for the one disk resource?).

Which version of Windows are you referring to? While risking to sound like a fan boy here, I must say that the OS load times for XP are quite fast compared to previous versions and to most vanilla linux distributions I've tried in the past (Mandrake 9.x, Redhat8/9). Whether or not this is in relation to resolving two processes arguing over access to read from the disk, I don't know. Does anyone have any more information on this?

Re:Amiga Disks

[MrFreshly]

I think windows loads slow because of all the damn spyware and un-necessary default drivers and services and IE preloading...etc...

IMHO Default windows config is kinda like a Redhat with everything and then some.

Start in safe mode and watch all the crap that tries to load - a ton of it is not needed.

If you tighten your install by removing a lot of the extra services, spyware, and a few performance tweaks - you'll see a major speed increase over all.

I use XP, but I don't like it much anymore...It's stable, but I'm really looking forward to a better desktop for Fedora and the new Core 2 release.

Re:Amiga Disks

[dave420]

Windows only takes ages to boot if something's wrong. My machines (p3 1ghz, p4 2.4ghz) all boot up in under a minute... I'd try taking a look in your error log :)

Re:Amiga Disks

[marvin2k]

I've always wondered why there wasn't something in the OS to force this behaviour, Ie, making sure that App 2 access to the disk is queued until app 1 has finished.

There are two big problems with the OS making such a blunt decision:
1. The starvation problem: What if app 1 never finishes it's access? Then app 2 would never run and "starve" in the run queue.
2. The fairness problem: Both apps have been run in parallel so why should app 1 get all the IO resources? If they run with the same priority then they should also get the same fair share of the IO resources.

Schedulers *do* make fairness decisions but only on the lower levels as on the application level the behaviour is simply to diverse and it's hard to find out what could be considered fair in any given situation.

Re:Amiga Disks

[jarran]

Because it's a lot more complicated that you suggest. What happens if A gets in first, but is doing an extremely long a disk-bound task? B will never get chance to access the disk. It could even be that B would stop after a very short amount of disk access, in which case it will have to wait until A is done, even though interleaving the reads would have been the "right thing to do".

Being multi-user complicates things even further. Sure, you are a single user on a desktop machine, and you double click on two programs in rapid succession, queuing them for loading one after the other may be the right thing to do. But what if those programs are actually being loaded by two different users? Can we completely lock out one user just because they started loading their program slightly later? Again, what if user A runs emacs, and a fraction of a second later, user B runs ls? Under your system, B effectively has to wait as long as it would take to load emacs, plus as long as would take to load ls?

You can't even realistically seperate the queues by user. In many situations, a single unix user may be running on behalf on many physical users (AKA human beings ;) ), e.g. in the case of any kind of server.

I'm not saying that any of these problems are intractable (Linux is now doing a pretty fine job), just that they aren't as even remotely as trivial as queuing loads one after another.

Oh BTW, thanks for bringing back happy Amiga memories. Them were the days! :-)

Re:Amiga Disks

[Tony Hoyle]

The OS load times for XP are fast at the moment... Wait until SP2 arrives and you'll be back to the good old days.

Current boot -> useable on my SP2 box is 2-3 minutes. It was 1 minute before SP2... I think it's doing some kind of remote update or something as it accesses microsoft.com during this time (when my internet was down it was 7-8 minutes to boot as the DNS had to timeout).

Re:Amiga Disks

[tonywestonuk]

From there webpage Seagate barracuda drives have 100Mbyte/sec max, and >58 MByte/sec avg transfer speed. Emacs is 4.2Mb (Excluding required librarys). I make this approx .1 Second to load emacs, as long as the load has exclusive access to the drive. 1/10 of a second is not long to wait as far as the user is concerned.

Re:Amiga Disks

[SpaceLifeForm]

The GPP was referring to an Amiga, not Windows. Entirely separate beasts.

The problem with seeking excessively applies to *any* computer with movable heads on the hard drives.

Re:Amiga Disks

[TheNetAvenger]

The OS load times for XP are fast at the moment... Wait until SP2 arrives and you'll be back to the good old days.

Current boot -> useable on my SP2 box is 2-3 minutes. It was 1 minute before SP2...

And this is not reflective of all SP2 test systems. In our labs, this is unique to a certain set of chipset confirgurations. Also please note the word 'beta' on the SP2.

90% of the systems in our lab boot faster with SP2 than SP1 or RTM versions of WindowsXP.

So you may be ready for the good old days, but everyone else won't have to be.

If you are seeing this type of delay, then bug it, if you are not participating in the beta, then tell someone that is, so they can bug your system configuration.

I am currently typing this on a 1.7ghz Laptop, and my FULL boot time is 17secs. Standby is 1sec, and Hibernate is 3secs from power off.

(And this is in contrast faster than any of our high end test systems in the lab can acheive with any *nix distribution booting to the XWindows GUI desktop.)

The previous poster saying that Windows BOOT time is slow has not apparently used Windows in a long time, or is just talking crap.

Re:Amiga Disks

[shyster]

I've always wondered why there wasn't something in the OS to force this behaviour, Ie, making sure that App 2 access to the disk is queued until app 1 has finished. Isn't this one of the reasons Windows takes ages to boot? (many processes all competing for the one disk resource?).

AFAIK, the reason Windows used to take ages to boot was that drivers and services were started sequentially and no optimaztion was ever done for the boot process. Windows XP, OTOH, had a goal of less than 30 seconds for a cold boot. In order to achieve this, new BIOS specs were implemented as well as optimization of the boot process. The main things done to speed up the boot process were doing driver and service initialization and disk I/O in parallel, and prefetching. MS claims a 4-5x increase in speed using a chunked read of all boot files, but others disagree and think that prefetching accounts for most of the increase.

With a new PC and a fresh install of XP, it's very possible to get to the desktop in less than 30 seconds. Even with my aging PIII-500MHz laptop (without the BIOS optimizations called for by MS) and with additional startup software, my PC is usable in less than a minute. To be honest, it's the one reason I switched to XP from 2000.

Re:Amiga Disks

[Ouroboro]

I've always wondered why there wasn't something in the OS to force this behaviour, Ie, making sure that App 2 access to the disk is queued until app 1 has finished. Isn't this one of the reasons Windows takes ages to boot? (many processes all competing for the one disk resource?).

You run into a problem where you don't know when app 1 has finished loading in order to start loading app 2. Why, because a loading app looks no different than a running application. You could possibly get around this by having the app set some flag that says "Hey I'm starting up, give me a few extra bashes at the disk before making me yield." The problem with that solution, is what happens if that application never releases the flag. You've got an application that has nearly exclusive access to the disk, and may not be interested in giving it up.

Re:Amiga Disks

[schroedlzone]

Win XP is strange, everyday I leave work and leave mozilla, outlook and netbeans or something open. When I return the next morning and move the mouse and try to use something Windows chugs away at a crawl for two or three minutes before I can get any response...
I've got a p4 and plenty of memory for these apps. Irritating as all get out.

Re:Amiga Disks

[LWATCDR]

Actually even early versions of Novell used a system called elevator seeking. The system worked like an elevator the head moved in one direction of util it hit the lowest track/sector in the que then it changed direction. Not nearly a slick as the new system but a big improvment over a first come first serve system. Now that we have so much more memory and cpu power Linux and other OS can use more complex systems

Re:Amiga Disks

[ckaminski]

Yup. When something is inactive for any amount of time, it pages it to disk. The only cure is to remove your page file. Worse than that, however, is when windows, when doing file copy operations, decides to commit 450MB of your 512MB's of your system memory to the filesystem cache, even when you have the machine configured to give applications priority. Mm... nothing like page-thrashing your machine to death.

Re:Amiga Disks

[greygent]

Windows doesn't take ages to boot, troll. Not since 95/98 several years ago. Windows XP probably boots as fast or faster than most Linux distributions. It boots quicker in fact, because it loads stuff in parallel, not one after the other.

Re:Amiga Disks

[noselasd]

Hmm, sounds exactly like the early 2.4 kernels.
Did MS borrow some MM code here perhaps ? ;)

Re:Amiga Disks

[bestguruever]

When I'm repeatedly banging out ls commands to monitor how far along my home built export process is, 1/10 of a second certainly is to long to wait. Don't you people understand how important it is for us application admins to have detailed timely statistics?

Re:Amiga Disks

[dustmite]

My experience has been that a clean install of Windows (XP or 2K) loads pretty quickly, but once you install a few basic apps (even just a few basics like e.g. Office and VS .NET) boot-up time slows down a lot (startup time both before and after logon, i.e. total time until computer becomes useful).

Re:Amiga Disks

[colinleroy]

You've got an application that has nearly exclusive access to the disk, and may not be interested in giving it up.

MacOS before OS X worked a bit like that (not for the disk, but for the CPU). It was called "cooperative multitasking" and the applications' developers had to put calls to yield() (approximate, I don't remember) here and there in their code. That's the reason why holding the mouse button down on a menu stopped the rest of the machine (Clock stopped blinking, for example).

Re:Amiga Disks

[harryk]

I like your comments here. Something that I've been experimenting with is setting up small ~/bin directories within the user directories that store some of the basic commands. The real thought though is setting up ~/bin as a symbolic link to a RAM drive. This way specific programs could have near instant access to the binary part of any application.

Granted this still causes a bottle neck if the client needs to run something from the local disk, but I'm trying to reduce latency. I'm not 100% sure how to test performance with this, but I was thinking that running time /bin/app and time /mnt/ramdrive/bin/app might give me a good idea, but some of that still relies on local disk access.

Anyway, just thought I'd comment on something I'm trying.

Re:Amiga Disks

[jarran]

Again, it really isn't that simple. If you know the average throughput of the drive in your machine, calculate how long emacs should take to load. Now, load emacs. Did it take as long as you expected?

The drive in my machine supposedly does more than 10MB/s average, and yet emacs takes much much longer than half a second to load.

If you believe you will actually in real life achieve the performance the manufacturer claims the drive is capable of, you are, to put it politely, slightly naive.

Re:Amiga Disks

[jarran]

I seem to remember that some versions of Unix (including maybe Linux??) have a "sticky" bit which can be set on executables, which causes them to stay in memory after they have been used once. Although, I've never used it, so I could be remembering totally wrong!

I think that "time /mnt/ramdrive/bin/app" shouldn't access local disk in some shells (eg bash), as "time" is built into the shell, rather than an external command. Of course, "app" might access the disk. I'd create some big programs that do nothing. Although, I suspect that you'd have to be clever to avoid stuff like demand-loaded executables defeating your benchmarking. Maybe a program which sums a large amount of static data?

Always hard to find benchmarks that relate well to real life.

Re:Amiga Disks

[Alan]

They had this feature in windows 3.x, 95 and DOS, it was called "lack of multitasking", or sometimes "cooperative multitasking",and ensured that you could only run one application at a time, ensureing the highest possible performance from each individual application.

Re:Amiga Disks

[budgenator]

damn spyware, that's the biggest problem, the spyware starts and tries to phone hone before the networking is even initalized; a microsoft techie pointed me to spybot s&d, to clean the crap out and get the boot time back to something reasonable. I suppose that I might learn enough about windows to do it manualy but why bother? I don't use windows except on rare occasions.

Re:Amiga Disks

[ChrisMaple]

No, you're flaming.

Re:Amiga Disks

[4ntifa]

Weird. In my memory, every single Windows OS up until XP booted quite slowly.

Re:Amiga Disks

[jesup]

Back in the day (Amiga 3000 introduction circa 1990) we could boot an Amiga off a 40MB disk from power-on (including loading a replacement for the ROM image) to fully up (including mounting NFS mounts) in ~15 seconds; 7 seconds from a warm boot. Effectively we chunked the load of much of the OS (including GFX, Workbench (desktop), etc) as part of the softload of the ROM image into RAM. On top of that, on warm boot we checksummed the ROM image in RAM (as well as various major OS library structures), and if they were good we didn't reload them. (Remember, the Amiga ran without memory protection.) Some of the boot script that followed would cause overlap (anything tha spawned servers/processes), other parts were sequential but generally were very fast and often cached (like "ASSIGN XYZ: dh0:xyz" commands and the like).

At the time, Win3.1 took Quite A While to boot. Getting past the BIOS alone usually took longer than the Amiga took to finish booting.

Re:Amiga Disks

[Scooby Snacks]

That used to be true, but it isn't anymore. chmod(1) has this to say:

The `sticky bit' is not described by POSIX. The name derives from the original meaning: keep program text on swap device. These days, when set for a directory, it means that only the owner of the file and the owner of that directory may remove the file from that directory. (This is commonly used on directories like /tmp that have general write permission.)

Re:Amiga Disks

[Scooby Snacks]

Same with versions of Windows prior to 95 (and in 95 through Me, it's true of 16-bit apps).

Re:Amiga Disks

[BigDuke]

Which one of your Open Source buddies have you been smoking the hooka with???

Have you run out of tin foil or something?

Re:Amiga Disks

[drinkypoo]

My current theory (I am more than willing to have it shot down) is that programs are allocating a shitload of memory and a bunch of it gets swapped out if you don't have a lot of ram. My Windows XP boot times were cut in half when I went from 512MB to 1GB of memory.

I've found the opposite

[redcliffe]

I've actually found that on my machine, a pretty much standard desktop, response is a lot slower on 2.6.5 than 2.4.22. Not sure if I got something set wrong in the compile, but moving the mouse and stuff like that seems a lot jerkier under load. I use a USB mouse and keyboard, so maybe that's part of it. Anyone else seen similiar?

Re:I've found the opposite

[Nikademus]

This is due to the fact you are probably still using /dev/psaux instead of /dev/input/mice.

Re:I've found the opposite

[Outsider_99]

I found that my mouse was more responsive. That was the only performance difference betwen 2.4 and 2.6 for me

Re:I've found the opposite

[StormyMonday]

Yup. I've seen the same thing with 2.6.3. When there's a CPU intensive process running, the mouse becomes very jerky. It doesn't seem to make a difference whether it's on the mouse port or USB.

I've also seen a problem where the Web browser (either Mozilla or Firefox) pegs the CPU. Bad Javascript in a Webpage soemwhere? Never saw this on my old 2.4.23 kernel.

Re:I've found the opposite

[DuSTman31]

Yes. I found that. Then I realised I had the hard disk in PIO only mode. A recompile with DMA support and it's smooth as silk.

Re:I've found the opposite

[bflong]

Make sure that you set X's "nice" value to 0. Some distros set it to something like -10 so that X is not disturbed by other procs. Under 2.4, this was a good thing. However, under 2.6, with it's superior scheduler, the kernel will keep interrupting X and you will see lagging performance. Google for it to get a better explanation.

Re:I've found the opposite

[IdleTime]

Well, my experience is that 2.6 is much more snappy than 2.4 and I have seen no mouse lagging that you are talking about and I do run an Oracle database on my machine too.
When that is said, I also have to mention that I use a different scheduler than what is mentioned here, I use CFQ (Completly Fair Queueing) found in the Andrew Mortons mm-patches and started with the following argument to the kernel:
elevator=cfq

Not sure where to get those patches, but under Gentoo you can get it with "emerge mm-sources"

For the average desktop users using either KDE or Gnome or similar, I have found that CFQ gives a more snappy desktop than AS or deadline.

Re:I've found the opposite

[alexborges]

Check out what happened to your graphic drivers. It might be that your particular graphic drivers are not well portedd yet.

Re:I've found the opposite

[ksuMacGyver]

I had the same problem until I noticed that my chipset drivers were not being compilied into the kernel! (make oldconfig from 2.4-->2.6 is no good as a lot of the options have had name changes). Now it is so much faster than 2.4, I don't think I could go back. :)

Re:I've found the opposite

[Starborn]

No, he meant -10 - IIRC the problem is with the new scheduler in 2.6, increasing process priority has a tendency to cause 'hiccups' in process execution (deliberate) in an attempt to stop a few high prio/high cpu usage processes from killing interactivity, although throughput is mostly unaffected.

Of course this is from memory of someone explaining it to me real quick, so i'm probably wrong - but he did mean -10.

Re:I've found the opposite

[klevin]

I've found a similar problem on one of my machines. All kernels after 2.4.19 have incredibly choppy audio. But only on one machine (an AMD Athlon system with a LSI 1010 based scsi controller, 3com 905C ethernet and a Turtle Beach Santa Cruz sound card).

None of my other systems have this problem.

I'm an end user

[Anonymous Coward]

Can someone please explain how this is done in laymans terms.
This would be grate for my laptop, as the harddisk slows down the entire system. It takes like 30sec to load up mozilla. My laptop seems to spend half the time reading stuff from the harddrive. It takes 5sec to start xterm!

Re:I'm an end user

[eggstasy]

Are you sure its the disk? What are the specs on your lappy? You may be running low on memory, so the system will use a part of your disk as virtual memory. This results in bits of data being swapped from disk to memory and back, which of course will increase disk access and slow everything to a crawl.
If you have an old computer, and are unwilling to upgrade its hardware, try "downgrading" your software to older versions.
I can browse the web fairly decently on my 10yr old 486, but it's running windows 3.11 with opera 3.62 :)

Re:I'm an end user

[troon]

Sounds to me like you have insufficient RAM. 2.5" disks are, in general, slower, and if your system is being forced to page a load of stuff out to swap in order to fit the new processes in, it will slow down.

Re:I'm an end user

[Bronster]

Are you sure its the disk? What are the specs on your lappy?

I don't know about the parent, but my laptop is a P4 2.66 Ghz with 512Mb memory and a 60 Gb hard disk (Sony Vaio PCG-FR825P). I'm seeing massive slowdowns whenever the disk needs to be accessed, and I suspect (this is a new machine replacing my older Toshiba piece of shit) that the hard disk being only 4200rpm is my main problem.

I've plugged in an external 7200rpm USB2 disk and compared a complex database operation both on the internal and external disks - it was much faster with the /var/lib/postgres/data mounted on the external disk. Hardly a reliable benchmark (and I can't show the transactions - they're part of a piece of software I'm writing for work), but enough to make me wish I had a faster internal disk!

I'm tempted by the downgrade (some things feel slower than my P2/266 Acer 510T from back in 2000), but then I wouldn't have the DB engine for the work I'm doing.

Re:I'm an end user

[claar]

If your processor is at least 400Mhz or so, I'd echo the other replies to your comment about making sure you're not out of RAM; but also, remember that most laptops come with a 4200 RPM Hard drive. If you can afford it, I've found it's a *huge* performance improvement to upgrade to a 5400 or even a 7200 RPM drive. Be careful to purchase a hard drive that's compatible with your laptop, though.

Stolen from SCO

[Anonymous Coward]

Obviously, this was stolen from SCO. This was based on their UNIX software and was available in the baseline from 10 years ago. It only shows that Linux, once again, is not an innovator, but just copies code from SCO to achive its scalability.

Re:Stolen from SCO

[bfg9000]

[Associated Press] Darl McBride (news, quote) and Steven P. Jobs (news, quote) today announced the blessed merging of their respective companies into the one true innovator: Scapple (symbol:SCPL). Scapple is now the biggest Unix vendor, the inventor of Unix, the newest Unix, the best looking Unix, the most closed Unix, the laggiest Unix, the hippest Unix, the most artsy-fartsy Unix, and the most litigious Unix.

Look forward to more great Innovations(TM) from Scapple in the near future, coming to a courtroom near you! And don't forget to check out Scapple's iSueMusicStore for more great Innovations(TM)!!!

But how?

[builderbob_nz]

is accomplished by minimizing the disk head movement

I was always under the impression that modern hard drive designs hide the physical disk bits and pieces from the PC. So how can software predict where the heads are?

Re:But how?

[Anonymous Coward]

Clusters close together are going to be close together on the disk surface. They're not actually talking about controlling the head movement directly, but minimising head movement by realising how a hard disk works in relation to sector accesses .

Re:But how?

[pseudorandom]

The absolute translation of logical block to head position is unknown to e.g. Linux. While it is possible to reverse engineer the physical disk layout by looking at timings, for general purpose computing this is going way too far. I think the upcoming ATA-7 hard disk standard has some more options to get information about the layout of the disk, but I'm not sure of that.

Anyway, simple sorting on LBA address will typically reduce head seeks to a large extent, resulting in most of the potential benefit. It is important however to make sure that multiple requests are available to the driver to sort.

Not too unreasonable.

[mfh]

1000% written as a decimal factor is 10.00, or a 10-fold improvement. When dealing with latency times measured in milliseconds, that's not too out of the ordinary. I'm no expert, but look at this situation: (someone correct me if I'm wrong)

Say, if a block is read on one end of the platter, then 10 subsequent reads are read in close proximity at the other end, followed by an 11th read at the beginning again, a predictive seeker could re-prioritize the 11th seek to be right after the first. That would cut down on quite a bit of head movement, as well as improve the seek time for that 11th block without negatively affecting the others too much.

Re:Not too unreasonable.

[prescot6]

Your example would definitely result in a performance gain, but only by about 10% (since only 1 of the 10 remaining reads would benefit).

But I suppose that rearranging your example a little could change that. So, let's say the requests are scattered all across the platter. So the head would be zig-zagging accross from one request to the next. If those requests were then re-orderized then all of them would result in a significant performance boost.

I'm no expert on this by any means, though :)

Re:Linux Speed (Or Lack Thereof)

[Anonymous Coward]

Try going outside. Find out about these things called "women".

Or switch to using BSD. Then you get computers and women.

Re:Linux Speed (Or Lack Thereof)

[Eastree]

>Try going outside. Find out about these things called "women".

And this would help my computer how?

Disk Transfer QoS

[johnhennessy]

I think Solaris 10 (or maybe a later version, I can't remember) is suppose to support a concept of Quality of Service applied to disk accesses.

Is anyone in the Linux world considering this ?

This is probably more applicable to the enterprise market, but surely any scheme of informing the scheduler about the expected disk transfer characteristics has to improve performance.

On the other hand, it might be just Sun trying to re-invent uses of buzz words to sell their products.

Re:Disk Transfer QoS

[Xouba]

Two words: IRIX, XFS.

IRIX had some sort of "quality of service applied to disk accesses", as you wrote, thanks to XFS. The filesystem allows defining zones that have a "minimal throughput" configured. I can't say more about it because I know only by referrals of another people O:-)

XFS is available for Linux since 2.6.0 and 2.4.24, IIRC, and I think this feature is also available in the latest kernels. Though it's still experimental, IIRC.

Re:Disk Transfer QoS

[diegocgteleline.es]

Yes, It's done. Ssearch for "CFQ scheduler". It's in the -mm tree. You've "io priorities" so you can tell apache "you've been guaranteed 80% of the disk bandwith". Or run updatedb cron jobs at a lower io priority so they don't interfere with mozilla/openoffice... Irix has it already for years. Linux has it now. MS is planning this for Longhorn...(it makes the OS a bit more "realtime" so you won't have video pauses because your videos have a highter priority...)

Re:Disk Transfer QoS

[illumin8]

I think Solaris 10 (or maybe a later version, I can't remember) is suppose to support a concept of Quality of Service applied to disk accesses.

This is being added as a necessary feature in order to support hardened "containers" or whatever they will be called when Solaris 10 is released.

Basically, if you've got several virtual images of Unix running on a single box, you need to be able to throttle not only disk usage, but CPU, memory, and other resources, especially if you have any mission critical virtual servers running on the single physical server.

The power of being able to create hundreds of virtual servers on a single hardware platform is useless without the ability to guarantee some type of QoS to each individual image.

Re:Disk Transfer QoS

[diegocgteleline.es]

Yes - it's called CFQ. You can say "this process will have 35% of the disk bandwith guaranteed". It's ion the -mm tree.

Re:Disk Transfer QoS

[drinkypoo]

XFS' Guaranteed-rate I/O feature is indeed an experimental option in the latest kernels.

Re:Anti-MS Patent

[Anonymous Coward]

I believe this feature is in NTFS.

Re:Linux Speed (Or Lack Thereof)

[Anonymous Coward]

*ugly* women? no thanks :)

Benchmark

[zz99]

Here's an older benchmark made by Andrew Morton showing the anticipatory scheduler vs the previous one.

The benchmark was made before 2.6.0, but I still think it shows the big difference from the 2.4 IO scheduler.

Quote:
Executive summary: the anticipatory scheduler is wiping the others off the map, and 2.4 is a disaster.

Re:Anti-MS Patent

[Tribbin]

Stealing what? The algorithms?

The end-(Windows)-user benefits from it.

That's the price of freedom.

And any additions MS makes to the code must be made public.

So then everybody benefits.

Retro is still cool ?

[Anonymous Coward]

It's great watching the "modern" computer industry discover all the toys and optimisations that where essential engineering for the systems I used to use in the '70s & '80s.

All the wonderful stuff like disk seek optimisation, interleaved memory (Even MMU came to the moden computer about 15 years after everyone else had it) were technologies that made systems stand out from each other.

Because of the speed of things these days, lots of that tech has been largely ignored, until now when we're starting to hit hard performance barriers again. Now we have to invent the technology og the '70s all over again. It's nice to see all this stuff comming back though.

Oh, come on...

[mdb31]

...to achieve the O(1) timing, quite a leap forward that we had not even thought of!

The NT scheduler has been O(1) like, eh, forever.

Our kernel produces far superior performance due to providing hooks for the COM layer

Yeah, whatever. There is no COM anywhere near the NT kernel, and the latest and greatest from Microsoft, the .NET framework, isn't even based on COM anymore

Nice troll...

Re:Oh, come on...

[Anonymous Coward]

Linux's 2.6 constant time scheduler is better in the sense that it also creates new processes in constant time, unlike NT. NT's internals hasn't really changed dramatically in a long time, they're basically missing out on a lot of things that happened in the last 4 years.

This article is just another example of such a case. The anticipatory scheduler algorithm was not published until 2001. With Linux you get these sorts of benefits integrated in reasonble time, with Microsoft you have to wait between W2K and Longhorn to get any of the new goodies.

Re:Oh, come on...

[Anonymous Coward]

For very large values of 1....

Re:Oh, come on...

[spectecjr]

This article is just another example of such a case. The anticipatory scheduler algorithm was not published until 2001. With Linux you get these sorts of benefits integrated in reasonble time, with Microsoft you have to wait between W2K and Longhorn to get any of the new goodies.


You seem to have missed the point.

The "Goodies" as you put them, were already in place in Windows NT 4.0. Possibly earlier.

Re:Oh, come on...

[spectecjr]

The NT scheduler most certainly is NOT O(1). But the NT kernel was written using Pascal calling conv. b/c the earlier kernels were written in Pascal. And most of Windows (but not the kernel) is still COM including IE, Outlook, Word, etc. Why do MS people insist that Windows includes technology it clearly doesn't. Just try to start a couple of apps at once and watch your screen freeze for over 500 ms and then tell me the scheduler is O(1). Do you even know what O(1) means?

You obviously don't.

Being disk IO bound does not mean that thread scheduling is not O(1).

The thread scheduler is indeed O(1).

Your example is one of being IO bound.

I've run tests to prove this. And before anyone links to the IBM benchmarks, they don't test thread scheduling efficiency - they test how quickly a thread can be woken up and made active from a signalled kernel object.

what's old is new again

[hb253]

Didn't Netware do this, oh, 15 years ago? I think it's called elevator seeking. What's old is new again.

Re:what's old is new again

[Yokaze]

Elevator seeking is looking at the current request queue and bundle requests which are close together to minimise head movement. This is indeed old. IRC, Linux had it since 2.2 something.

The anticipatory scheduler tries to anticipate future requests (who would have guessed that?), and is relatively new

Re:what's old is new again

[Anonymous Coward]

It isn't just elevator seeking. This logic can actually pause a new request for a little but if it thinks that another process is about ready to do a read in the general area where the head is. The elevator seek simply sorts all the requests into the order. This mehotd tries to anticipate a request before it is issued. Kind of like holding the door open in the elevator for you instead of you missing it and having to catch the next one.

Re:what's old is new again

[joker784]

Elevators that look into the future - straight out of Hichhiker's Guide to The Galaxy :-)

OS 101

[dioscaido]

Unless I'm mistaken, these disk read algorithms have been around for the past 20 years. I seem to remember studying them during my intro to O.S. class years ago.

Re:OS 101

[juergen]

In response to all this "old stuff" postings, yes, ladder algos have been well known for a long time. The difference is, usually they were used to schedule multiple pending I/O requests. The new linux schedulers (especially the anticipatory one) make guesses about future requests in addition when deciding where to seek to next. Tis is also a major differene to SCSI tagged queueing.

-Jurgen

fragmented information

[millahtime]

"database data is generally grouped together and read in a big chunk"

It sounds like what your saying is that non database data on a disk is fragmented and that is why the head has to move all over the place.

CFQ

[kigrwik]

The cfq scheduler in the -mm (Andrew Morton) trees gives very good results in a desktop use.

With anticipatory or deadline, I'm experiencing awful skips with artsd under KDE 3.2 every time there is a heavy disk access, but it's [almost] completely gone with cfq.

To use it, compile a -mm kernel and add the 'elevator=cfq' to the kernel boot parameters through Lilo or Grub.

See this lwn article for more info.

Re:CFQ

[rzei]

Great, I thought I was the only one with bad artsd performance.. I haven't yet given -mm tree a shot but I guess I should.

However mplayer performs nicely under whatever load, reading from wherever. I blame artsd's performance on it's horrible mp3 plugin which doesn't seem to buffer reads at all, nor does it "silence" errors in mp3 files, when compared to mplayer's every supported mp3 lib for example.

BTW I haven't investigated this a bit, artsd just has been skipping for forever, and I that's just what I've conclueded in time..

Re:CFQ

[lscotte]

Agreed... I've been using the cfq scheduler for a while (few months) and it really seems to perform nicely both on desktops and servers.

I'm surprised the person who posted this article didn't do some research first and .... Oh wait, never mind, this is /. :-)

Re:CFQ

[Ice_Balrog]

Try going into the KDE Control Center -> Sound and Multimedia -> Sound System, and unchecking the box that says "Run with the highest possible priority (realtime priority)". I've heard that the Anticipatory Scheduler can sometimes cause problems (skipping and such) with applications that have a low nice level.

Re: CFQ

[Omniscient Ferret]

It's cool to see different schedulers coexist; it looks like freeblock scheduling could supplement or improve anticipatory scheduling, for example.

Re:Anti-MS Patent

[mydigitalself]

no: stealing the concept (probably by analysing the code) and writing it themselves.

so if MS make any improvements in their own implementation of the concept, then the code would not be made public and MS benefits and not everyone else.

to elaborate (and in some ways i believe this is what SCO are arguing), lets say i see an open source application that does something neat. it probably won't be patented because the author expects someone to contribute any modifications back. but lets so i don't because i'm a greedy commercial corporate and so i effectively copy the IDEAS behind the application. my code may look quite similar to theirs, but i certaintly have not infringed on the GPL (or have I - i'm no lawyer!).

so if this neat application had an "open source patent" in that anyone infringing on the patent would not be liable for millions, but rather they would be liable and forced to open up the source code of their particular implementation.

Real benefits...

[greppling]

...for the typical desktop workload would come from a better cooperation between applications, glibc, and the kernel.

Let me start by claiming that optimizing desktop performanceis all about optimizing I/O patterns (contrary to what all Gentoo users think :P). My KDE startup is about three times as fast when I everything is in the disk cache, so it is clear where the bottleneck. (Just try logging in to KDE after boot, then log out and log in again.) A concentrated effort of

• passing on the right hints from KDE via glibc to the kernel (e.g. an madvise() call when loading executables giving the hint that probably most part of the file will be needed later on),
• trying some anticipatory reading of config files/libraries etc. from startkde where it is known that they will be needed, and that they are hopefully laying contigiously on the disk,
• optimizing disk layout for the common access patterns

would IMHO make a far bigger difference for the desktop experience than optimizing compiler flags by using gentoo or using a preemptible kernel.

There has been a lot of discussion about this on the kde-optimize list (with Andrew Morton participating), so maybe we can hope that KDE 3.3 will offer some improvements.

As an aside, yes, we all hate the windows registry, but I think we should admit that for boot time optimization it is the right thing to do (having everything in one file that is layed out in one contigious block on the disk.)

Re:Real benefits...

[Lord of Ironhand]

I see another benefit: this should also put less strain on the hard disk itself, thereby possibly letting it last longer.

My two latest HDD features occured during periods of intensive random seeking so I'm quite willing to believe this might have a significant effect on HDD life. Even if it's not enough statistical evidence, common sense also tells me things wear faster when they're doing heavy work...

But it would still be an interesting experiment: operate a number of identical servers and let them do something that generates a lot of small random reads (such as a simulating a heavy duty Usenet server). Run half of them on 2.4 and half on 2.6, and see if drives fail significantly quicker in the 2.4 systems. Anyone got some spare HDD's left? ;-)

Re:Real benefits...

[_LORAX_]

That's why I compile my gentoo with -Os

I realize that the overall size of the binaries and the improvement in cache hits, less IO, and better prelinking will improve the system a whole lot more than shaving %5 off the cpu time of an application.

Re:Real benefits...

[Anonymous Coward]

but I think we should admit that for boot time optimization it is the right thing to do (having everything in one file that is layed out in one contigious block on the disk.)

This is oh-so-wrong in so many ways. Let me name just a couple of them:
1. Because damned near everything causes changes in the registry, the registry is NEVER layed out in one contigious block on the disk. Instead, the constant editing forces it to be scattered across the entire disk.
2. I can make a better argument that it actually slows the boot process because, in order to get the parameters associated with just booting, the system must read thousands upon thousands of other registry entries that have nothing to do with the boot process! I would much prefer a smaller file that only has the parameters need for booting and NOTHING ELSE!

Coupled with the inherent risk of "placing all your eggs in one basket", the registry most decidedly is NOT the right thing to do!

Re:Real benefits...

[realnowhereman]

Depends -- the trade you are making is start up time, for performance once running. It's not unlike the trade between bandwidth and latency. A truck full of DVD's will beat anything in bandwidth terms; but it's no use to you.

Re:Real benefits...

[bobbozzo]

Smaller binaries also use less RAM/VM, often resulting in better system performance, which is one reason why MS optimizes all their stuff for size.

You're misunderstanding something...

[warrax_666]

AFAIK the "anticipation" bit is not so much about predicting head movement, but is more about reducing head movement. Reads
cause processes to block while waiting for the data (and can thus stall processes for long amounts of time if not scheduled appropriately), whereas writes are typically fire-and-forget. This last bit means that you can usually just queue them up, return control to the user program, and perform the actual write at some more convenient time, i.e. later. Since reads (by the same process) are usually also heavily interdependent, it is also a win to schedule them early from that POV.

That's my understanding of it.

Re:You're misunderstanding something...

[Mr. Hankey]

It certainly doesn't, and app developers have the option to use the raw device if they're really that worried about it. Oracle e.g. can do this. Still, the IO subsystem has improved significantly in the 2.6 kernels. Worst case blocking from large chunks of IO have been reduced. Instrumentation on system IO is also improved, you can finally tell that a Linux system is IO bound and not just at a load average of 1.2 for no reason. (About time.)

FWIW, SCSI drives and controllers have implemented command queuing and reordering for ages. SATA will also implement it, though not quite to the degree that SCSI does. If your transactional systems run faster with SCSI drives than they do with similar parallel IDE drives, there's a good chance that command queuing and reordering has something to do with the improved performance. An OS' kernel implementation can add some extra optimizations that the device might not see, since it has a bigger picture of what's going to the disk.

You might even want to give the 2.6 kernel a look if you're doing large transactional operations. Despite your assertion that Linux is somehow dead, development is moving along quite happily. ;-)

Re:You're misunderstanding something...

[Lost Race]

Writes aren't really "queued", in the sense of a simple list of writes waiting to be serviced. Rather the application write causes block buffers to be allocated and marked dirty. Periodically dirty buffers are written to the media. Application write latency occurs when there is no free memory available and dirty buffers must be written (or memory pages written to swap) before a new buffer can be allocated. Thus the application must wait for a physical media write to complete before its write can be serviced.

I have found that in some circumstances flushing buffers more frequently (by tuning kernel vm parameters) can completely avoid this write latency and speed up write-heavy applications significantly.

Speed-ups

[jd]

I've often wondered what would happen if such I/O speedups were put into hardware. There's plenty of RAM on modern controllers, but caching adjacent tracks is not as efficient as caching distant tracks, so as to minimise the need for moving the read-heads long distances.

Alternatively, have multiple read-heads on a single arm. 3 would be a good number. The idea here would be that you could pre-seek either side of the disk, before finishing a read by the currently-active arm.

Re:Speed-ups

[AlecC]

Effectively Scsi does I/O speedups. Firmware, not hardware, but so is everything. And the speedups by giving Scsi a lot to do and letting it do it in its preferred order can be significant. But Scsi cannot "see" processes - nor file systems. The OS can work out that a process is reading a file and read the next bit of the file - where Scsi would read the next bit of the disk, if it did so at all. The OS can see when you ahve reached EOF, or closed the file, and there is no point prereading.

You don't mean multiple heads on an arm. Multiple heads on an arm would all move together, and you couldn't use two at the same time - the feedback servo which keeps it on track can only respond to one track. What you mean, I think, is two groups of arms (all the arms move together). Manufacturers have looked at that but decided against it.

The arms and associated actuators are some of the most expensive parts of the drive. If you are going to double this cost, why not throw in a few more platters andd an enclosure and have twice the capacity - and twice the throughput.

Putting two actuators in the drive increases power consumption a lot, and heat as well. Both are real problems for current drives. And a "specialist" drive doesn't have the economies of scale, and could cost more than twice as much as two simple drives - which, together, have the same number of heads and twice the capacity.

The real killer is turbulence. If you have two arms on the same surface, each is flying in the wake of the other. And, unlike its own wak, the other alters dynamically, so that seeking arm 1 can perturb arm 2.

Google has it right - lots of dumb hardware, lots of clever software. What we need id filesystems whos allocation patterns are "Raid aware". Particularly Raid 0, I can see file system allocation patterns which could (in conjunction with the otimisations mentioned here) greatly improve performance.

Re:Speed-ups

[maxwell demon]

You don't mean multiple heads on an arm.

Are you sure? Of course only one of them would be in use at any time, but if there are, say, three heads on the arm, the average reposition time should be 1/3 of the original (because every head covers 1/3 of the disk).

Of course this assumes head switching time is shorter than the head moving 1/3 of the disk.

Re:Speed-ups

[Ranten_N_Raven]

Multiple heads per arm? Been done!

I have what I tell visitors is "My Whatzit." Then, I aske them: Whatzit?

It's the head unit from an old Apollo system that head-crashed around 1988. It has two heads per arm, so the head seek movement only had to travel half the distance (And this thing is MASSIVE--it weighs as much as a modern desktop-sized drive), so that reduction in movement is even more significant.

By the way: Computer Science types figure it out MUCH more often than do those with a EE. Go figure!

Re:Speed-ups

[AlecC]

Rotational latency is now longer than average seek. With escalator seeking (as mentioned throughout this thread), most seeks are short anyway. Some large disks used to have multiple heads (I used some on 14 in Fujitsu monsters) but the cost benefits are the wrong way round on current disks.

Head switching time is quite long - about a 1/10 of a worst case seek on one drive I have seen - but not long enough to come into this equation.

I think I've heard of this

[TheVidiot]

Doesn't this involve a green marker, and tracing along the edge of the hard drive? Faster and less distortion?

Preemptive and Defragged?

[stecoop]

I always heard the Linux wasn't preemptive and that is why embedded developers shy away. Is this the first step towards resolving preemptive issues?

Also, It sounds like that if Linux had a defrag utility that the data could store data on the disk the way it would be accessed. If the OS would watch to see how the data is being accessed, it could then re-arrange the data dynamically. Example - you access File A which accesses File B and File C, the OS would recognize this and re-arrange the data in that order A, B, and C during low CPU usage times.

Re:Preemptive and Defragged?

[Anonymous Coward]

Linux has had preemptive patches for ages. As far as I can tell, the work from Timesys Corp is the leading implementation. It uses a variety of patches to make linux fully preemptive and suitable for use as a hard realtime os.

The only reason a disk would need defragged is if the FS sucks so badly that it causes massive fragmentation. Utilizing better storage methods such as certain RAIDs, LVM, and any of the Linux or Unix filesystems drastically reduces any problems one might have.

As an example, I'm using UFS2 (FreeBSD 5.2.1) with soft updates and have been using the same UFS slices for well over a year (not always 5.2.1) constantly (webserver, fileserver, and I do some light compiling) and have 0.2% fragmentation. You're just used to terrible filesystems.

Re:Preemptive and Defragged?

[Doctor Crumb]

Firstly, the 2.6 kernel allows pre-emptive scheduling. Supposedly it was introduced because Linus got tired of his mp3s skipping while he compiled things.

Second, Linux doesn't need a defrag utility. Linux filesystems (Ext2 and Ext3) allocate files properly, using clustering and inodes. The need to defrag comes from the bad design of FAT, which works great on a 8088 processor with tiny files on a 1Meg drive, but is terribly inefficient on anything past a 386.

Of course, there does exist a 'defrag' utility for linux. It just won't gain you much at all.

Re:Preemptive and Defragged?

[Zoolander]

The thing with Windows' startup time is that all services aren't started when you can log in. They are still loading in the background.
That's how it seems to load so fast.

Re:Preemptive and Defragged?

[Zoolander]

The point of the new scheduler(s) is that most access to the disk by a process is sequential (i.e. many blocks at a time), so if another process wants to access some other part of the disk, it most often pays off to let that process wait for a while before serving it, since the original process most likely will want to get more data from your current block.
That way, you don't need to move the head nearly as much as if you responded directly to the other process.
Robert Love has written an excellent article about the new schedulers here: I/O Schedulers

Re:Preemptive and Defragged?

[Brandybuck]

I hate those Windows tricks. People think it's faster because they see the desktop, but it's really still in the middle of loading.

Ten seconds after boot you get the login prompt. But the system is waiting, not doing anything. Not even the network is fully up and running. Then you log in an five seconds later you see the desktop. But you cursor is an hourglass and you can't do anything with it for a while. Then it flickers to an arrow and back to an hourglass. If you try to start an application during this time, odds are it won't get started.

I did a test once on a dual boot W2K and FreeBSD machine. From poweron to the display of Slashdot in the browser on the desktop, W2K/IE took fifteen seconds longer than FreeBSD/KDE/Konqueror.

Re:Our take on it from inside MSFT

[BCoates]

Your comment is meaningless gibberish studded with technobabble.

I believe you, you must really work at Microsoft.

Databases and reliable commits

[hughk]

One of the big things about databases is the reliable commit where all the crud you have done in a transaction either gets committed or backed out. The writes then become kind of important, and you really do want them to complete before continuing with anything else.

This messing with the I/O queue may make things interesting for the journalling process which is kind of vital to integrity. File placement could become even more important for this (and also the placing of journal/log files).

The rest seems to just effectively be a modified elevator (wait a bit before moving).

Re:Databases and reliable commits

[Lozzer]

The change to the scheduler doesn't change the semantics of O_SYNC or fsync. Making sure your data got to disk is no different to what it was before.

Re:Databases and reliable commits

[hughk]

The problem is whether O_SYNC writes can be prioritised over reads. The scheme discussed here is orientated towards read-often/write-sometimes which is normally quite reasonable. However, if you really are waiting on that write, then it needs to be expedited.

Re:Databases and reliable commits

[joib]

PostgreSQL at least uses write ahead logging, and the manual claims that it's a standard technique. See here.

I guess you're correct that the anticipatory scheduler can reduce write performance when you're waiting on fsync(), but the logic seems to be that the improved read performance will more than compensate for this (some other poster said 15 % improvement for "typical" database loads).

Re:Our take on it from inside MSFT

[dave420]

Yeah, and where I work in Microsoft, Bill Gates frequently drops in with the latest copy of Linux User, espousing his love for Tux. Why, only yesterday he said he cries himself to sleep wishing he thought of Linux, instead of Windows.

Zealotry is all fine and dandy, but delusional zealotry just lands people in jail.

You need help, buddy.

Idea...

[cca93014]

Would it not be possible to write a very basic adaptive network that "learns" what the best values for these parameters are for each individual machine, based on a history of its workload?

Re:Idea...

[cca93014]

Doh! Was meant to include this quote in the above post - mods can ignore the above post please...

You can tune your anticipatory scheduler to improve its functionality. There are five basic parameters you can alter to change the way the wait-before-seek times function: read_expire, read_batch_expire, write_expire, write_batch_expire, and antic_expire.

Would it not be possible to write a very basic adaptive network that "learns" what the best values for these parameters are for each individual machine, based on a history of its workload?

Re:Anti-MS Patent

[dave420]

So, in turn, should the Linux community cease developing/including things that are "inspired" by Windows?

Oooh.... evil Microsoft must be thwarted!

[October_30th]

Hold on there. What you're doing there is arguing for software patents - that one should be able to own an idea.

It's kind of sad that the free software advocates sometimes get so carried off by their pathological hatred for Microsoft and corporations that they don't see that they're about to become "the enemy" themselves.

Free is free. If you start to restrict the use and availability of your code by requiring the release of any modifications to the public, it's not free code anymore - no matter what RMS says.

Re:Oooh.... evil Microsoft must be thwarted!

[mydigitalself]

you assume too much. i do not have a pathological hatred for MS, in many of my posts i have argued against exactly such extremism...

what i suggest stems from the fact that there is an obvious imbalance in the battle between open and closed source projects. i'm not suggesting a traditional patent model i'm suggesting a caviat to such a model that could actually encourage the sharing of implementation rather than restricting the intellictual property.

i'm not an uber-low-level-geek, but lets say there was some sort of logical algorithm that is being implemented in open source. anyone at MS (for example!!) could look at that source code, extrapolate the algorithm and re-implement it without giving anything back. what i am suggesting is that one is free to implement the "open-source-patented" concept/algorithm/idea (its all very fuzzy in Patent Land), but any implementations thereof should be subject to a contract that enforces the open publication of the implemented code.

Mixed units

[AlecC]

"sometimes by 1,000 percent or more, [more] often by 2x"

Nice mixed units - why not have 10x/2x or 1000%/200%?

Actually, we have a serious missed opportunity here: where is our folksy comparison unit? We have football pitches for length, African Elephants for weight, Libraries of Congress for volume and/or data. Where is a nice "just folks" ratio? Politicians promises (10x delivery)? Admans truths (100x reality), real virgins (1/1000000 of a porn site)? /. insight (1/1000 of words)?

Re:Mixed units

[julesh]

This unit works nicely enough - it is still a meaningless figure that sounds as though it was pulled out of the air. I mean -- just what does it mean to say that performance impoved by 1000%. I measure performance as the length of time I have to wait for an operation. Improvement means it reduced. So the amount of time is reduced by 1000%? In that case, it would now be negative - the job is done before I ask for it, by a factor of 9 times the length of time would previously have taken?

well, why don't..

[zogger]

... you help out then? If there are more tips n tricks out there, help to implement them if you have the skills and memory.

Re:Anti-MS Patent

[skrowl]

You can't patent an idea, you can only patent an implementation. Surely the Windows implementation would differ enough from the Linux one to escape any kind of patent violation. Even if it did violate a patent, do you think Linux as a whole could sue Microsoft in court and win?

Well done, Nick Piggin and Andrew Morton!

[toby]

Looking forward to using this myself - am building a colocated web server box which will be running 2.6.

[ot]

[mirko]

Thanks but my father is Croatian and my Mom's French :o)
Anyway, you found out that I indeed am not a native English speaker, hence the neologistications.

Re:[ot]

[sylvester]

Meh. It was a shot in the dark. Mirko sounds like a standard finnish name, although apparently it isn't. (I've been pondering writing a "Random Finnish name generator." I think I'll do it closer to when I leave, though. :-)

Kernel comparison on a SMP system

[k-hell]

2CPU.com has a Linux kernel comparison of 2.6.4 and 2.4.25 on a SMP system with interesting results.

Re:Kernel comparison on a SMP system

[Monkey]

Looking at their results, I don't understand why the performance of 2.6.4 is 5% slower than 2.4.25 when serving static web pages and yet it rocks as a file server with 2.6.4 being a whopping 89% faster at serving files than 2.4.25.
Given the steep skew in the results, are we really looking at how well Apache runs on 2.6.4 vs Samba?

Re:When will Linux do direct IO?

[Anonymous Coward]

Direct IO has been available in Linux since 2.4.0 at least.

Schedule for Interactivity

[ehack]

Desktop Linux needs a scheduling policy specific to interactivity. I guess this may happen the day a decent interface gets slapped on the Linux base. Until then, we dance the same dance - every release is faster than the previous one by the benchmarks, and feels more horrid than the previous one.

Surprise, the Mac has the same reactivity problem now thanks to its Unix (Mach) kernel, while the previous Mac OS 9 crashed regularly, couldn't multitask, but has a much snappier user-experience. Apple has been adressing this issue - which they recognize- for 2 years now, and have almost but not quite fixed it with their current Panther release.

It is time we found a way to benchmark a user experience in order to prevent over-optimisation for number-crunching.

Most of my posts get marked down as trolls - think hard: How can you solve a problem if you refuse to admit it exists ?

Re:Schedule for Interactivity

[BlueLightning]

the day a decent interface gets slapped on the Linux base.

I'm sorry but that, my friend, is a troll. Or at the very least flamebait. You do have a point, but you turn your potential audience off by saying things like that.

Desktop interactivity is being worked on, though. A lot of the discussions on scheduling in the 2.6 kernel have mentioned it, and personally I think 2.6 has improved in terms of visible desktop performance in some areas.

Re:Schedule for Interactivity

[ehack]

Exactly how many Joe Users are going to recompile the kernel to improve their scheduling ? This is a typical Linux answer: There is a way to do X, for the technical specialist,so we don't need to change the default behavior of the system.

May I give a counterexample ? Joe, your local garage mechanic goes into Walmart and
buys a Linux box for $300 for his grandma, and tacks on some old monitor he had in his house. When the grandma plays DVDs, the system stutters because Fred Developer in his wisom believes that a Linux box is probably a MySQL web server, and builds in a database scheduler as default. Is Joe supposed to learn to recompile the kernel? Would Joe recognize a kernel if it bit him in the ass ?

Re:Amiga Disks and CD-ROMs

[pmjordan]

Yeah, the same thing happens under Windows if you read from CD-ROM. The whole thing just slows to a crawl if you try to read two files at once. I'd assume it's a hardware problem, (long seek times, large error margins) not necessarily Windows' fault, but I don't use CDs much anymore (hooray for ethernet and huge hard drives) so I don't know.

Of course, this raises the point that aligning the data on a game CD or DVD for a console is a science in itself. PC game development is easy in comparison! (plonk everything on the hard drive)

phil

Someone should point out...

[Mgdm]

...that the Red Hat "kernel development systems engineer"'s name is Stephen Tweedie, not Tweed :)

Re:Anti-MS Patent

[Anonymous Coward]

Haha, this is totally stupid bullshit - since Windows 2000 already has this feature, it is Linux that copied the idea from Windows.

So should Linux kernel benefit from this idea although it's copied from proprietary software, only implementation is different? I say sue their ass!

Seriously, this is so typical of the open sores crowd that it goes on my nerves - all good ideas come from Linux, Windows is "lame" and "bloated", UNIX OS'es are slow, etc.

If you look at Linux, there are few geniuine innovations, it's mostly coding in open source what already exists in closed source (that's why Sun's Joy said he never bothered with Linux 'cause there's nothing new in it).

I second that.

[Qbertino]

A friend of mine who does desktop work on Linux exclusively (lot's of developement) recently compiled himslef a 2.6 kernel and reports a very large, noticable increase in overall speed.
I'm using Debian Woody with a Nvidia Patched 2.4 kernel, so I'm reluctant to go through all the backporting and Nvidia recompiling fuss, but I'll guess I'm gonna do it sooner than I initially thought.

Re:I second that.

[incuso]

New nvidia driver is 2.6.x compliant. I am using it on my PC with 2.6.4.

M.
--
Monete Italiane

Re:I second that.

[Noltar]

As already stated, the Nvidia 5336 driver is 2.6 compliant. You should have no problems grabbing the kernel source and nvidia kernel source (or a binary package) from backports.org if you want to stay with Woody. I just helped a friend go through a similar process (his first time building a kernel) and it wasn't a real issue. From backports.org you need the following packages for your custom kernel: kernel-source-2.6.4 kernel-headers-2.6.4-1 kernel-headers-2.6.4-1- (They are in the kernel-image directory rather than kernel-source, or their own for some reason)kernel-kbuild-2.6-1 kernel-package module-init-tools modutils Then for building your own nvidia driver: from the nvidia-graphics-driver-binary-i386 directory: nvidia-kernel-source-1.0.5336-5 nvidia-glx_1.0.5336-5 That should take care of all of the dependencies for both, I think. Then you just install the packages, build your kernel, build the nvidia module, and install the 2 resulting debs. Of course, you can use prepackaged backports of both if you want, which are also available. Good luck.

Re:When will Linux do direct IO?

[Tony Hoyle]

Umm it does.

If setting an option can save 50% of IO times that says more about Solaris than Linux, IMO.

Re:NOVELL/NETWARE DID THIS IN 1991

[AlecC]

No, this is not the elevator algorithm. This is an anticipatory algorithm that pre-queues reads that it expects the application to do in the future. Linux already has the elevator algorithm - had it before Windows, I believe.

Slackware!

[Allen Zadr]

I was able to get the 2.6.4 kernel running on Slackware in less than 4 hours (most of that, compile and configure time). No broken dependancies at all.

However, I wouldn't even try that on RedHat or Mandrake without having the .config file and a list of distribution specific patches.

This was on a Celeron 1GHz laptop, and honestly, I couldn't tell the difference in speed beyond any custom compile. Custom meaning unnecessary device drivers are removed, and the ones that I need are compiled in (as opposed to remaining modularized).

Re:Slackware!

[jdunn14]

I exclusively use RedHat on my boxes (laptop, desktop, tablet) and I always compile a vanilla kernel from kernel.org. Can't stand to use the redhat one. Here's a tip, try 'make rpm' to build a 2.6.x kernel package that can just replace the redhat one. Only dependency I ever had to worry about was DRI. rpm -Fvh complained about the new package not having DRI so I turned it on in the kernel, ran make rpm, tried the rpm -ivh again, and all was well.

However, I NEVER compile from redhat's kernel-source rpm. That patch pile never has built for me correctly. Most of the time it just doesn't build at all...

Re:Slackware!

[ncc74656]

I was able to get the 2.6.4 kernel running on Slackware in less than 4 hours (most of that, compile and configure time). No broken dependancies at all.

However, I wouldn't even try that on RedHat or Mandrake without having the .config file and a list of distribution specific patches.

I have 2.6.x on a couple of Gentoo boxen; it was a fairly painless upgrade for one, while the other was built with it from the start. You'll want to fetch development-sources instead of vanilla-sources (or whatever, but that's what I normally use).

The only snag I've run across was getting ivtv to control a WinTV PVR 350 on a fresh install with a 2.6.3 kernel. It wouldn't control the tuner properly, so all you'd ever get from it was static. Upgrading to the just-released 2.6.5 fixed that...with a few more tweaks, my MythTV box will be in good shape.

Re:Slackware!

[OpenGLFan]

(anecdotal evidence, sample size of one, etc.)
Slackware was my first and favorite distribution; my home computer still runs slackware.
Having said that, upgrading Mandrake cooker to 2.6 took around an hour, and 15 minutes of that was me not knowing enough about the alsa sound system to run the initial "alsaconfig" or whatever to get sound to work. Apart from sound, it all worked.

Slackware's still my favorite distribution, but Mandrake has their act together.

Re:Slackware!

[Allen Zadr]

Nooooo!

I'm behind the curve. Must download 2.6.5 immediately. Must not fall behind!

IO Accuracy

[fhafner]

so how does this effect IDE drives in terms of IO read/write accuracy? We use SCSI drives for low level mass data processing and mining because what you write to the disk is guaranteed to be what you can read from the disk in the future.

IDE disks don't have the same guarantee. Does the new 2.6 kernel improve this?

I also wonder if this reduces hard drive wear for longer lifetimes....

Re:IO Accuracy

[pe1chl]

No disk ever guarantees that you can read it back in the future, and the interface does not influence that much.

Maybe you are referring to the bus parity on SCSI drives, but UDMA EIDE drives have block CRC.

Both SCSI and IDE drives have had FEC and bad block remapping for ages.

BSOD scheduler has been O(-1) like, eh, forever

[Anonymous Coward]

I guess that makes the BSOD scheduler O(-1) or so - the more you use the computer, the faster that scheduler works.

Re:BSOD scheduler has been O(-1) like, eh, forever

[Sensitive Claude]

I guess that makes the BSOD scheduler O(-1) or so - the more you use the computer, the faster that scheduler works.

Oh sure, you think this is funny now, but if you get too many BSOD you travel back in time, and the next thing you know you are confronted by burley men in armor with sharp pointy swords.

I don't understand why you are also transported to England. That's just how these things tend to work out.

How can I set the boot parameters?

[incuso]

I would like to test it, but I looked in the kernel documentation and I was not able to find out what parameter I have to pass to lilo for selecting the appropriate scheduler.

M.
--
Monete Italiane

Re:How can I set the boot parameters?

[Zoolander]

Use 'elevator=as' (or cfq, or deadline)
The anticipatory scheduler is the default for the vanilla 2.6 kernel.

DID THEY REALLY?

[Jussi K. Kojootti]

Elevator seek (which has been in Linux for a while btw) looks at the current request queue, this is about anticipating future requests.

Re:_New_ Kernel?

[Allen Zadr]

Funny, the 2.6 kernel has been out with this feature for months. It's old news. I've been running 2.6.4 on Slackware 9 for almost a month now. Yes, all the nifty features turned on. No, I really don't see much of a difference (beyond the standard improvements of a kernel compiled without all the crap I don't need).

And if you look above to this post, you can all see a great deal of decent explanations of what 1000% increase actually means (11%).

Not me....

[Allen Zadr]

I haven't noticed the difference myself, but then again - if this is a new trick, and I've been running 2.6.4 for a month and 2.6 was first released several months ago...how is this a "New Linux Speed Trick"?

Yawn

[Pig Hogger]

This increased speed is accomplished by minimizing the disk head movement during concurrent reads.

Nothing new there. Since the dark ages, Novell has been using elevator seeking to precisely achieve this.

Trouble is, it is easy to implement when you have precise control over the disk heads positionnings, which was the case with the "old" ST and ESDI interfaces, where the OS directly specified a head and track/cylinder and sector number.

But nowadays, with EDI and SCSI drives that only have absolute sector numbers and automatic bad sector remapping, it becomes harder to specify directly the precise exact mechanical movement and thus optimize it.

And there is so much optimization a drive firmware can do, because only the OS will truly know for sure what's scheduled next in terms of disk space.

Re:Yawn

[Anonymous Coward]

You are right, there is nothing new there... if they were talking about elevator seeking style movement. This article is about Linux making use of Anticipatory Scheduling which is completely different and quite new.

Re:Anti-MS Patent

[TheNetAvenger]

ok, i know this is evil and all - but lets say MS decide to implement this as a concept (so without "stealing" code)... the linux community will have given them something and received (probably) nothing in return.

Not to burst your bubble, but the NT scheduler already implements predictive disk I/O concepts.

Nice that Linux is finally catching up though...

2.6.x faster in other ways, too

[aussersterne]

Aside from much better I/O performance, 2.6.x also has much better performance on my notebook (IBM T-series ThinkPad).

I don't know if it's due to SpeedStep support being in the kernel or what, but when I was running 2.4.x with the pre-emptible kernel patches, switching from wall power to battery power meant massive slowdowns, as though I had switched from a PIII-1GHz to a 100MHz Pentium classic. Simple commands like "ps" would take seconds to complete and screen redraws were visible. The whole system would feel like sludge. In spite of this fact, battery life was relatively poor. The combined effect (much slowed system, very short battery life) meant that it was difficult to get anything at all done on battery power.

Now with 2.6.x, when I switch to battery power, there is no perceptible slowdown whatsoever when compared to wall power, and battery life is much improved. Downside: suspending 2.6.x kills USB-uhci, so I've had to compile it as a module and hack up my suspend/resume scripts to reload it each time. But for the speed increase, it's well worth the trouble.

Re:2.6.x faster in other ways, too

[barnaby]

The usb issue is fixed in 2.6.5.
Details here

Heh.

[Mr Z]

It is new with respect to 2.4.x. The anticipatory scheduler was introduced 2.5.x-mm and made its way into the kernel by the time 2.6 was released.

Re:Heh.

[Allen Zadr]

Except for RedHat backporting (which may be part of the RedHat ES 2.4 kernel), I can't find any reference to the standard 2.4 set getting this scheduling.

If this exists, could someone kindly point me to it?

--
New account to Karma bonus in 6 days!

Re:Heh.

[Mr Z]

I'm not sure where you'd find it, but you might make some headway searching for "anticipatory scheduler" on kerneltrap.org. This scheduler was discussed multiple times on that site.

--Joe

UF

[soloport]

...but it's not an explicit kind of think...

Dmitri? Is that you?

Standard mistake

[fizbin]

You're making the standard mistake of assuming that the labor pool of "people who work on linux" is of a fixed size, and that man hours are interchangeable.

Linux doesn't work like that. The vast majority of people who work to improve linux aren't doing it because they're getting paid, and instead work on or focus on what interests them. If someone is focusing on feature X, that's not necessarily taking any time or energy away from feature Y - if they weren't doing X, they might very possibly not be contributing to Linux at all.

Seriously, complaints like this remind me of a manager coming in and discovering that some developers were talking about the finer points of thread interactions in a specific application and saying: "Who cares how the threading works? I just want something the customers can use!"

If it makes you feel better, you should learn to simply ignore discussion of technical features that upset you - this discussion does not in fact take away from discussions of user friendliness nor does it imply that the user will be forced to follow this discussion in order to use the outcome. If the user wishes, anyway, to follow this discussion then they might glean something interesting from it, but supplying the users with extra optional information can't be a bad thing, can it?

And as for access time, I have to ask: making the computer as a whole more responsive to my actions won't make me like using it better? Maybe 10% isn't going to make much perceived difference most of the time, but when it means the difference between a stutter-free movie playback and the occasional dropped frame, I'm going to notice.

Re:New Kernel?

[Progman3K]

It is.

It is 100% percent faster a whole 1 time faster.

So that makes whatever it was doing twice as fast now, right?

So it's 200% faster or 2x as fast.

eh, still confused/

NPTL is a key component of the new speed

[Stonent1]

If you compile GLIBC with NPTL support you'll see even more of the new kernel in action. I quote from LinuxJournal.com,

NPTL brings an eight-fold improvement over its predecessor. Tests conducted by its authors have shown that Linux, with this new threading, can start and stop 100,000 threads simultaneously in about two seconds. This task took 15 minutes on the old threading model.

What goes around comes around.

[Muad'Dave]

Shoot, I remember debugging the elevator seek driver for Interdata/Perkin Elmer/Concurrent OS/32 systems in the late 80's. This isn't new technology, not even for Linux. Remember the code that Tivo released back to the community? It was their implementation of an elevator-style seek mechanism for their PVR's.

Re:Dude

[julesh]

Read the post. He wasn't suggesting defragmenting, he was suggesting disk optimisation, which in Windows-land is performed by the defragmenter. The idea is that you put files which are likely to be needed together (e.g. program binary files and shared libraries that they reference) close together on the disk. The OS builds up statistical usage information about whats used and then at a later stage a disk optimiser process rearranges the disk for faster performance.

This has been known to result in slight performance improvements, and versions of it have been around in Windows since win 98, I believe, although Win XP marketing material has claimed it as something entirely new...

How does this affect RAID?

[TDot]

I run the 2.6 kernel, in a RAID1 (a partition mirrored on two drives) configuration. How does this neato scheduling algorithm work with RAID? I suppose with RAID1 it simply makes writing on each individual disk faster, because the disks are treated as individual units, but what about RAID0/4/5 ? Or, does RAID not have any effect, because it's higher level than actual disk-writes? Yeah I suppose that's probably right..

Re:How does this affect RAID?

[Kourino]

Actually, one of the big thing about as is that reads are favored above writes. Streaming writes in the presence of reads are generally somewhat slower (up to 30% slowdown has been noted in some cases), but small reads in the presence of streaming writes can be insanely faster.

Doing some googling, I've read that RAID setups and disks with deep TCQ won't benefit much at all from AS.

Re:How does this affect RAID?

[pe1chl]

With a RAID1 disk implemented in Linux software, there should be an additional level of optimization possible.
Having a RAID1 of 2 disks is like having a dual-processor system: there are 2 disks, with 2 head positions, available to read data in parallel.

With good optimization techniques this should yield double the troughput of a single disk. "hardware raid" controllers seldomly make this optimization.

Of course the disks have to seek to the same position for writes, but for reads they are independent.

How much of that is implemented in 2.6 ?

Re:How does this affect RAID?

[TDot]

Excellent question and great points, how much IS implemented in 2.6? It'd be neat to see if they've taken the RAID to that level. I'm taking a degree in CS at University of Waterloo, we're learning about these things (hard-drive optimizations of these types) currently, it'd be neat to know how Linux works on RAID.

Re:How does this affect RAID?

[pe1chl]

In fact the RAID code seems to be orphaned.
A while ago I noticed that when a single sector read error occurs on one of the mirrored partitions, the kernel takes the entire partition offline immediately. The user then has to do a "raidhotremove" and attempt a "raidhotadd", and the entire partition will have to be synchronized again.

I think it would be more reasonable in this case to read the sector from the other disk, attempt to write it on the disk that returned an error, read it again, and if that succeeds keep the disk online. There could be a counter that limits this procedure to a certain number of occurrences and then still take the partition offline as "unreliable".
As it is now, the soft-failed disk is not kept in sync anymore, and when an error occurs on the other disk the system dies. A recovery attempt would increase the chances of a running system, especially when the disk has automatic bad sector remapping and will write the corrected sector at a different disk location.

I sent this suggestion to a mail address listed in the code and documentation. The reply was that he thought it was a good idea, but had transferred the maintenance to someone else. I sent the message to the new maintainer and heard nothing.

While I probably could write this enhancement myself and submit a patch, I think I have little chance when it does not go through an official kernel maintainer. So I did nothing yet.

Re:Windows boots slowly???

[silicon not in the v]

I've seen people here trash Windows for a lot of reasons, but come on! Every version of Linux I've tried takes many times as long as Windows to boot. Windows comes up much faster than Linux in every case I've seen. I understand why you want so much uptime on Linux because it's such a pain in the arse to wait for it to boot.

Re:Windows boots slowly???

[PitaBred]

Do you ever question what "boot" means"? When a linux system lets you log in, EVERYTHING is already started and running. When Windows shows you the desktop, there is still a ton of stuff getting started in the background (or the foreground even) and it's still unusable. Windows doesn't start any faster, it just shows you pretty pictures sooner.

Nothing new here, move along

[tuxlove]

We implemented something very similar to this years and years ago at a company where I used to work. It sped up certain operations mightily. However, nothing comes for free. We found that it improved throughput at the cost of responsiveness. A great thing if you don't have users waiting for "ls" to finish, sitting at their shell window. A very bad thing if you do. It's the age-old tradeoff of throughput versus responsiveness. Just imagine a slider between the two poles, and set it where you want it. But you can't have both, unless you find some way to remove outright inefficiency, which doesn't seem to be what they've done here.

Re:Nothing new here, move along

[Kourino]

It's funny that you mention that, since one of the tests that Andrew Morton ran to see what AS was improving, and how much, was to time how long it took to get a login on a machine that was doing large streaming writes. Generally, 2.4 sucked a lot, and AS went somewhat faster than 2.5 deadline.

AS favors read throughput over write throughput, which will generally be perceived as an increase in responsiveness ... for some tasks during certain workloads, like this one, of course.

Re:Anti-MS Patent

[PitaBred]

If Linux is catching up, why does Samba file sharing under Linux run faster than under NT?

Elevators (was: Re:SCSI)

[FooAtWFU]

If you have a 30 story building, you can either put in dumb elevators which fill 3/4ths of the building to meet demand, or you can put in a much smaller number of elevators using techniques like express elevators and using software which keeps track of usage patterns and puts elevators on floors before somebody even hits the button...

Who here remembers SimTower? :)

Re:*Real* direct IO, fanboy. That's crap in scsi/s

[colinleroy]

ioperm(2) isn't for disk reads, but rather for accessing memory slots.

And why, oh why, must Anonymous Coward have to act as if it knew about stuff, without even trying to google said stuff?

The Renaissance

[EXTomar]

And we all would have benifited from this if they simply shared in the first place instead of spending 20-30 years "rediscovering" it.

One programmer likened the 70-80s as The Dark Ages. There were cabals and secret voodoo that people sat on and didn't share and you ended up with an ignorant masses that only thought "this is as good as it gets". Hopefully this renaissance sticks because it doesn't matter how good or cool your technology is if you bury it for 20 years without another person knowing.

Not possible!

[shani]

Every person I've ever met that used an Amiga said they were perfect in every way.

Are you sure that you could slow the system down? Maybe you were using some hard-core mind-altering drugs back then....

Re:Not possible!

[eyeye]

He should have bought an external drive, then he could launch apps from different disks with no slowdown.

Perfect!

Ok ok some of us have a rosy dim recollection of Amigas, but can you blame us - at the time the PC (for example) was way behind in multitasking so you didnt even have the normal possibility that two apps are doing disk access at the same time.
Formatting a disk - well you just have to wait and do nothing till its done, not so with an amiga.

I'm a realist though so thats why I'm typing this on my 3ghz athlon PC and amigas are mere memories.

research background for anticipatory scheduling

[Sajma]

The original research for anticipatory disk scheduling was done at Rice University by Sitaram Iyer and Peter Druschel and is described here.

Some benchmarks

[diegocgteleline.es]

Those are some benchmarks posted by the developers, developers, developers....

http://groups.google.com/groups?q=io+scheduler+ben ch+group:fa.linux.kernel&hl=en&lr=&ie=UTF-8&group= fa.linux.kernel&selm=fa.citnj0l.th62p7%40ifi.uio.n o&rnum=7

http://groups.google.com/groups?q=io+scheduler+ben ch+group:fa.linux.kernel&hl=en&lr=&ie=UTF-8&group= fa.linux.kernel&selm=fa.cjtrj8p.vh22h3%40ifi.uio.n o&rnum=1

http://groups.google.com/groups?q=io+scheduler+ben ch+group:fa.linux.kernel&hl=en&lr=&ie=UTF-8&group= fa.linux.kernel&selm=fa.cithjp0.thc0h6%40ifi.uio.n o&rnum=4

Re:_New_ Kernel?

[maxwell demon]

Read that again. It's not 11%, it's 11x. In words: eleven times.

Same thing when kernel went 1.2

[ehack]

When the kernel went 1.2, I think it was RedHat 2.X at the time, I remember discussing that with people on Usenet: Half of us complained about interactivity - the other half said "but the benchmarks are better". There was clearly some subjective issue which the benchmarks missed.

A good example of interactive fluidity is what happens when you resize a browser window when the system is under load - does it move immediately ? Does it move smoothly ? If it "waits" for several seconds before resizing, your user-interface analogy breaks down completely.

This said, the user experience is not really improving as far as I know. Bloatware is killing Linux a Gigabyte at a time. The only way to get faster reactivity seems to be :Upgrade your hardware while freezing the software version :(
------------>
Microsoft firmly believes its system designs are stable and secure. Apple believes its systems are good value for money. Linux people believe their systems are designed for user friendliness.

Re:Same thing when kernel went 1.2

[BlueLightning]

Which browser have you tested with? Both Konqueror and Mozilla have somewhat slow-redrawing renderers which cause this. It is also made much worse by a badly coded or generic X driver. On my system at home with an ATI graphics card, proper drivers, XFree86 4.3 and KDE 3.2, for most non-browser windows tearing and aliasing is barely noticable - in fact I'm not sure that it's any different to Windows on the same machine.

I don't believe Linux is as user friendly as it could be. However I also don't believe the issue you brought up has anything to do with user friendliness - it's more about certain users being picky.

I don't think it's the Kernel or KDE's fault

[MikeCapone]

I have an ancient computer - a K6-2 450mhz with 192 megs of PC100 RAM - and Mandrake 9.1 was a dog on it.

Later I switched to Slackware and it was much faster. Then I upgraded to Kernel 2.6.x (currently 2.6.5) and KDE 3.2.x and things are even faster now.

I suppose that Mandrake is too big/bloated (depending on if you like it or not) for old computers.

2.6 on fedora 1 box in 10 minutes

[bobsalt]

install 2.6 on your rh9 or fed 1 machine in about 10 minutes....

http://people.redhat.com/arjanv/2.5/readme.txt

short version?

put this in your /etc/yum.conf
[2.6testkernels]
name=Test Linux 2.6-test prerelease kernels for RHL9/rawhide
baseurl=http://people.redhat.com/arjanv/2.5/


yum install kernel

Re:2.6 on fedora 1 box in 10 minutes

[Allen Zadr]

I will try this, thank you ... as soon as I figure out how to configure YUM on RedHat Desktop 9 (I still have RH9 on my work computer).

Re:Concrete examples anywhere?

[Kourino]

For your concrete examples, do a google for :anticipatory deadline scheduler: and you'll come up with the Kerneltrap article on Andrew Morton running various benchmarks. The 1000% speedup came in a workload that isn't much like a desktop workload at all (effect of streaming write on many small reads), so it's no surprise that you're not experiencing an 11x speedup on day-to-day apps.

As for the article, it also manages to get Stephen Tweedie's name wrong. For future reference, Yahoo! News articles about Linux (the kernel) probably won't be that great -_^

another I/O speed trick: mount with noatime

[chongo]

While you are waiting to install the new kernel code code, you might try a filesystem mount option called noatime that has been in many *n*x distributions for a while now.

If you don't care about last access times on your files, then you should consider mounting your filesystems with the noatime mount flag as in this /etc/fstab line:

LABEL=/blah /blah ext3 defaults,noatime 1 2

Reading a file under noatime means that the kernel does not need to go back and update the last access time field of that file's inode. Sure, multiple reads over a span of a few seconds will only cause the in-core inode to be modified, but eventually that modified inode must be flushed out to disk. Why cause an extra write to the disk for a feature that you might not care about?

For example: think about those cron jobs / progs that scan the file tree (tmpwatch, updatedb, etc.). Unless you mount with the noatime option, your kernel must at least update the last access time fields of every directory's inode! Think about those /etc files that are frequently read (hosts, hosts.allow, DIR_COLORS, resolv.conf, etc.) or the dynamic shared libs (libc.so.6, ld-linux.so.2, libdl.so.2, etc.) that are frequently used by progs. Why waste write-ops updating their last access time fields?

Yes, the last access time field has some uses. However, the the cost of updating those last access timestamps, IMHO, is seldom worth the extra disk ops.

There are other advantages to using the noatime mount option ... however to wind up this posting I'll just say that I always mount my ext3 filesystems with the noatime mount flag. I recommend that you consider looking into this option if you don't use it already.

Re:Anti-MS Patent

[spectecjr]

Because Samba is network IO bound, not disk IO bound.

Careful when advising /etc to be a partition...

[TheScienceKid]

You may jest, but you should bear in mind that if because of what you've said, someone goes and creates a partition for /etc, they could well right royally SCREW up their system.

A lot of the content of /etc needs to be available early in the boot process such as /etc/inittab. If you WERE to do this you would be advised to use an initrd to mount /etc before init launches.

Otherwise you can have the (fun, fun) headache of synchronising a copy of /etc/inittab and /etc/fstab (not to mention the content of /etc/rc.d and /etc/init.d if you don't use the old monolithic inittab style) from the /etc partition to the / partition at shutdown...

I've done something similar on a machine that was tftp booted with a ramdisk image. It's /etc directory was symlinked into an AFS location where the files were stored on the network, along with a copy of boot-critical files and directories placed in AFS's mountpoint for before afsd is run, although it did end up being VERY messy when I mixed it with large amounts of functionality like kerberos support (which meant adding in time synchronisation)... you get the picture.

In conclusion, I'd say that it's not really worth it. After all, your /etc is just about the only thing on / that DOES change.... your /bin, /sbin and /lib shouldn't change much. Also, /usr /var /tmp /boot /opt (if your distribution has it ) and /root (if it's used at all) should all be seperate partitions.

Likewise, /mnt and /dev rarely change. If you run an AFS server, vicepa, vicepb.... etc until viceiv MUST be partitions in their own right.

Re:_New_ Kernel?

[Allen Zadr]

Same trap everyone else fell into - typing faster than I can think.

I know what I mean. Anyway, I pointed to the right thread, so that the full discussion can be had.

Re:Linux Speed (Or Lack Thereof)

[Ice_Balrog]

I'll take the Linux women over the BSD women any day.

http://www.linuxforum.com/linux_wallpapers_full/53 .php

Re:Anti-MS Patent

[TheNetAvenger]

If Linux is catching up, why does Samba file sharing under Linux run faster than under NT?


Because the easter bunny told you it does?

Geesh...

Samba is a great product, and ran neck and neck with NT4 file sharing services. However, Windows 2K3 is a leap ahead of what Samba offers in features, and also is running circles around Samba performance wise.

There are a couple of benchmarks that give the edge to Samba, but in real world overall use, Win2k3 is considerably faster.

And THIS HAS NOTHING TO DO WITH THE NT Scheduler or predictive disk read ahead techniques that this thread is about.

How can I switch between them?

[ksp]

I know there is a boot-time switch for changing the I/O scheduler, but I still believe you are stuck with one for all devices. How about using different algorithms for different partitions? There is quite a lot of difference between a database device, a filesystem holding binaries, shared libaries, /tmp, spool directories etc. etc. etc. When I/O schedulers are so different in their theoretical foundations, why do you have to choose only one?
This should be a mount option, not a boot option.

Re:How can I switch between them?

[Ziviyr]

Uhh, not /dev/null, /dev/zero.

Re:How can I switch between them?

[MikeBabcock]

NULL is undefined; so the quote is actually funnier the original way, personally.

Re:Linux Speed (Or Lack Thereof)

[bobbozzo]

Your computer will be much more stable.
(Because you won't be able to upgrade it often.)

Believe it!

[Anonymous Coward]

This is all very true. Not moving the disk head is everything.

In fact, my research group discovered this years ago - and precisely because of this we developed a hard drive with a single track. It had 65,536 heads and was very fast.

It was also about two city blocks in diameter. It got torn down because we were violating municipal building ordinances. Shame.

Link

[respite]

Good reading if you are interested in this sort of thing
Linux Kernel Development

FreeBSD runs faster

[Anonymous Coward]

I have tested the Linux kernel 2.6.x series using the fastest Linux distro, Slackware, (I customized it and compiled it) and FreeBSD still runs faster with the defaults settings (no tweaking) !

Free ACM paper here

[fprog26]

Anticipatory scheduling:
A disk scheduling framework to overcome deceptive idleness in synchronous I/O (2001)

Sitaram Iyer, Peter Druschel
18th ACM Symposium on Operating Systems Principles

ACM portal

Using the old citeseer trick, you can read the PDF version here:
Citeseer paper version
PDF version

Don't SLASHDOT citeseer!
There is more than one citeseer mirrors, use google:
Google Citeseer paper search

Enjoy!

Re:FreeBSD

[MikeCapone]

I hear that Slackware is closer to classic Unix and BSD than most of the other big linux distros (Redhat, Debian, etc).

How hard do you think the migration to FreeBSD from Slackware would be?

Re:Anti-MS Patent

[Thundersnatch]

I believe this feature is in NTFS.

Since 1996 at least. As I recall, it was part of the reason a lot of non-MS database vendors at the time were squealing about SQL Server 6 posting such good TPC scores with minimal RAM on NT4. Claims of "Cheating", pre-optimization, special OS hooks, etc. Turned out it was just good caching algorithms and good filesystem performance.

Anyway, disk throughput on our 1996-era NT servers simply blew away our NetWare boxes, even though the hardware was similar. And don't get me started about how bad the I/O was on our HP-UX boxes was back then...

Believe it or not, there are (some number greater than ten) smart people working in Redmond, and they do occasionally get something right. I'm glad to see Linux get similar features, presuming the algorithms are original and not patent-encumbered (that's all we need now.)

Q ld.so?

[4of12]

This kind of reminds me of the arguments about how fast IE comes up vs how fast Mozilla comes. The former, "being part of the Windows OS", gets a head start from preloaded DLLs.

At one point in the past I recall a KDE investigation into why preloading shared libraries might help cut down on slow response that people were seeing with g++.

Do all the mechanisms with ld.so cache help to get shared libraries ready (in a memory buffer) before any program starts, say based on the last accessed or most frequently accessed libraries?

Re:Anti-MS Patent

[TheNetAvenger]

Well, since you didn't give a link to the test, just a report of the test I thought I would share an actual link...

http://www.veritest.com/clients/reports/microsof t/ ms_netbench.pdf

Yes this is a pre Samba 3 comparison.

SAMBA 3 actually performs quite well; however, not as fast as some zealots would like to believe.

Also of note, things that don't seem to get a lot of press is that, on standard (non-RAID) or RAID-1 systems, Windows 2003 Server performs better than a Samba/Linux solution. In RAID 5 solutions, the numbers get closer. (A lot of small businesses don't use RAID 5 on their in house servers) - Which is the IDEAL market for low cost SAMBA solutions.

Additionally, when using a SAMBA file sharing solution, you are giving up many of the features that make Windows 2003 Server and NTFS a better choice.

Like Shadow Copies, Compressed Storage, File Encryption, etc.

I'm sure you mean well, but don't drink all the cool-aid.

There is no reality, just perspective.

Re:Dude

[stecoop]

I was beginning to think either I was such a bad author that I should stop or it was too difficult for people to grasp. It could be the first and I imaging an anonymous joker will say so. But I cant believe I didn't get one Point out of this article while people went on tangents about FAT being a poor file system while they were addressing the article about Disk head movement.

Ehh, all is good. I will post it again in the future.

Re:Anti-MS Patent

[TheNetAvenger]

Then go on to say that samba precludes a bunch of features that are already in linux and thus can apply to any filesystem, samba exported or not.

Really, Linux does Shadow Volumes, Real-Time Compression, Encryption, and Journaling all with the one FS?

So which file system is this? Just curious, cause not one of our Linux gurus could find one.

Re:Anti-MS Patent

[TheNetAvenger]

I never said the benchmark was 'reality', I was making a point about not posting a link to an actual benchmark.

If you look on the internet you can find tons of 'benchmarks' that are non-Redmond and some say SAMBA3 is the fastest and others say it is not. The ITWeek results are not the end all, be all in the benchmark discussion.

Also please note the comments I made about the RAID performances and the loss of functionality when running SAMBA3 instead of Win2k3.

I am not bashing SAMBA, just trying to keep the trolls from jumping on the bridge and eating the goats.

Take Care.

Re:Anti-MS Patent

[TheNetAvenger]

If you do want to cite a benchmark with details that supports W2K3AS being faster than Samba3 that hasn't been torn apart by people reading the details, please do.

I can post reports from our own labs my friend. There ARE times Samba performs better, and there ARE times when Windows 2003 Server performs better. (=Moot Point)

However with Samba, you lose a lot of functionality that many customers think are important.

Not everyone is just looking for a large file storage box, they are looking for a SERVER, one that does more than File and Printer sharing. One that can handle Media and a lot of other things that a Samba solution cannot.

People don't get that this is how Novell got into such a pinch. Microsoft NT server technologies offered more than just 'serving files and printers'. It was able to go head to head with even the high end *nix markets because of its application server capabilites that allow it to extend to whatever a developer wants to throw on it just like *nix.

Samba ia great product, but it is NOT always the BEST option for ALL businesses and Server users.

Re:Anti-MS Patent

[TheNetAvenger]

Filesystem snapshots (what everybody but microsoft calls "shadow volumes"): LVM, works for just about any filesystem, see http://www.sistina.com/products_lvm.htm

You don't get it my friend.. There is a difference between a 'volume snapshot' and Windows Shadow volumes.

Shadow volumes allow snapshots to occur on a file by file basis, not an entire volume at a time. Additionally this information is contained in the same volume, not requiring a 'snapshot point' volume to hold the information.

Microsoft knows what snapshots are (MS Virtual PCs use them), however Windows Shadow Volumes are NOT snapshots.

Take your LVM link as an example, it can't do anythingn like what Shadow Volumes on Win2k3 can.

For example, I can open any folder on my Windows 2003 server (from any client in the world), and right click on a file I have been working on, and see EVERY time I have saved the file, and restore it back to a prevoius version. (Even if I accidentally deleted it)

This is something that is so easy, that it is put in the hands of the users (with permissions) accessing their data on the server.

They don't have to ask an Admin to have the file recovered from a 'Volume Snapshot' and it also doesn't take an entire duplicate volume to hold the snapshot.

It tracks changes to individual files, on a file by file basis. And of course this also applies to Folders.

As for the other features you cite, these are all STANDARD features of NTFS with Volume Shadowing added in Win2K3. There are no other utilities to install, no additional plugins, no additional overhead.

Why install and work with 5 packages just to replicate what NTFS and Win2k3 does out out of the box?

And then you have to consider the 'performance' of these utilties when enabled on the server, and the selected file system.

There is not one set of solutions that fully replicate all the features of Win2k3 and NTFS and can run the same performance, even with SAMBA3 out there.

The SAMABA3 benchmarks everyone here are so happy to cite forget that all these features are turned on and a part of NTFS and Win2k3, and are not installed on the SAMBA3 tests machines being used to acheive their 'stellar' benchmarks.

And these benchmarks are not STILL not always faster than Win2k3.

SAMBA3 is great, but add in the features that Win2k3 is using by default and you will find that it does not have the amazing performance a lot of people seem to think it does.