Ask Slashdot: Smarter Disk Space Monitoring In the Age of Cheap Storage?

relliker writes In the olden days, when monitoring a file system of a few 100 MB, we would be alerted when it topped 90% or more, with 95% a lot of times considered quite critical. Today, however, with a lot of file systems in the Terabyte range, a 90-95% full file system can still have a considerable amount of free space but we still mostly get bugged by the same alerts as in the days of yore when there really isn't a cause for immediate concern. Apart from increasing thresholds and/or starting to monitor actual free space left instead of a percentage, should it be time for monitoring systems to become a bit more intelligent by taking space usage trends and heuristics into account too and only warn about critical usage when projected thresholds are exceeded? I'd like my system to warn me with something like, 'Hey!, you'll be running out of space in a couple of months if you go on like this!' Or is this already the norm and I'm still living in a digital cave? What do you use, on what operating system?

Comments

Performance issues?

[brausch]

How does performance change as the big disks approach full? That was always one reason for the rule of thumb about keeping at least 10% free space on UNIX.

Re:Performance issues?

[gnasher719]

ou want to keep the hard drive at 50% or less to maximize performance. If the hard drive is more than 50% full, the read/write head takes longer to reach the data. If the hard drive is 90% full, most OSes will have performance issues.

Actually, any OS will have performance issues, because the transfer rate (MB/sec) drops from the outside tracks to the inside tracks. That's why for home use, you just buy the biggest hard drive that you can easily afford (if you need 1TB, you buy 3TB), because that way you use only the parts of the drive with the highest transfer speed, and the average head movement time is also a lot less.

Re:Performance issues?

[fuzzyfuzzyfungus]

Unless you are the sort of disconcertingly disciplined and organized person who sets up a monitoring and alerting system for their dinky little desktop, you probably aren't talking about 'the hard drive'. At a minimum, you are probably dealing with some flavor of RAID, or ZFS, or an iSCSI LUN farmed out by some SAN that does its own mysterious thing behind the expensive logo, or some other additional complexity. Flash SSDs are also increasingly likely to be involved, quite possibly along with some RAM caches in various places.

Re:Performance issues?

[kuzb]

That's an interesting idea for the budget-minded, but personally I think if performance is actually an issue I'd use SSDs for things that need to be performant, and store everything else on regular drives.

Re:Performance issues?

[afidel]

Inner tracks have better seek times, which is why high performance applications often "short stroke" drives (ie artificially restrict the percentage of the drive used so that only the inner tracks are utilized, though with modern drives and transparent sector remapping it's unlikely this practices actually works), outer tracks have better streaming performance because more sectors move under the head in a given timeframe.

Re:Performance issues?

[Slashdot Parent]

That was pretty caustic, wasn't it!

Anyway, in today's virtualized world, none of what you ranted about really matters anymore. If disk I/O is important to your application, you're using SSD. If your filesystem needs more space, you just grow it using your platform's volume manager. And yes, real work gets done on Windows servers now. It's not my personal cup of tea, but you might as well just acknowledge it.

And you don't plan 2 years ahead because who knows what your requirements will be in 2 years?

Re:Performance issues?

[RabidReindeer]

If you use Unix on a server, you should have multiple partitions.

I use LVM, you insensitive clod!

Juggling physical partitions is a royal pain.

We have more but we USE more.

[pla]

Today, however, with a lot of file systems in the Terabyte range, a 90-95% full file system can still have a considerable amount of free space but we still mostly get bugged by the same alerts as in the days of yore when there really isn't a cause for immediate concern.

When we had drives in the 100s of MB range, we used a few MB at a time. Now that we have drives in the multi-TB range, we tend to use tens of GB at a time. In my experiences, a 90 percent full drive has as much time left before running out as it did a decade ago.

Perhaps more importantly, running at 90% of capacity kills your performance if you still use spinning glass platters as your primary storage medium (not so much when talking about a SAN of SSDs). In general, when you hit 90% full, you have problems other than just how long you can last before reaching 100%.

Whatever is measured is optimized.

[QuietLagoon]

...when there really isn't a cause for immediate concern.

It all depends what one is concerned about. Is maximizing disk space down to the last possible byte important to you? Or is performance in accessing random data important to you? Or is wanting to keep artificial limits imposed by monitoring systems important to you?

.
Once you determine what is actually important to you, then you monitor for that parameter.

Whatever is measured is optimized.

It's all about the data prouction rate

[aglider]

You insensitive clod! In the age of MBs, we were producing KBs of data. In the age of GBs we were producing MBs of data. And in the age of TBs we are producing GBs of data. And so on. Thus a 90% full filesystem is as bad as 10 year ago. Unless you are still producing KBs of data.