Ask Slashdot: Smarter Disk Space Monitoring In the Age of Cheap Storage?

relliker writes In the olden days, when monitoring a file system of a few 100 MB, we would be alerted when it topped 90% or more, with 95% a lot of times considered quite critical. Today, however, with a lot of file systems in the Terabyte range, a 90-95% full file system can still have a considerable amount of free space but we still mostly get bugged by the same alerts as in the days of yore when there really isn't a cause for immediate concern. Apart from increasing thresholds and/or starting to monitor actual free space left instead of a percentage, should it be time for monitoring systems to become a bit more intelligent by taking space usage trends and heuristics into account too and only warn about critical usage when projected thresholds are exceeded? I'd like my system to warn me with something like, 'Hey!, you'll be running out of space in a couple of months if you go on like this!' Or is this already the norm and I'm still living in a digital cave? What do you use, on what operating system?

Comments

I delete things when I'm done using them

[Anonymous Coward]

I never run out of disk space.

Re:I delete things when I'm done using them

[bobbied]

I'll bet that's not true...

Seems to me that the stuff I work on keeps getting bigger and bigger, as does my collection of digital pictures and videos. Where I attempt to pare down what I keep, some of it stays around...

I expect that most users do the same things and thus data keeps piling up. I don't think it matters how well you are at deleting stuff you don't need anymore.

Re:I delete things when I'm done using them

[dissy]

I delete things when I'm done using them

1) Many of my things I either desire to use for many years to come (a video download I paid for), or am required to keep to cover my ass (taxes, logs, most data at work due to policies, etc)

2a) The cost of more storage space is almost always less than the cost of the time to clean up files that could be deleted. In the context of work this does depend heavily on exactly who made the data and their rate of pay / work load - but I've noted the higher up execs and managers tend to be the worst hoarders as well as of course the highest rates of pay. Most of the lower techs on the shop floor don't even have access above read-only to the network storage here, though that is far from universal everywhere.

2b) Yes there are other people whos time is not as expensive, but no one other than the datas owner/creator can know 100% what needs to stay vs what can go (and sometimes even the owner/creator chooses wrong.)

3) After deleting/archiving data, the chances of you needing it in the future are typically higher to much higher than the chances you are really done with it.

4) For the small number of times you really are done with it (like, totally and fur sure), the amount of data that gets deleted is generally such a small percentage of the whole that, while still a good thing to do, doesn't really help much with the problem at hand - freeing up a lot of space for future needs.

I never run out of disk space.

You either have too much free storage space, not enough data, or possibly both :P

Re:Performance issues?

[Anonymous Coward]

Well, ext4 strives to scatter files around disk to avoid fragmentation. Once the disk begins to approach full, it has to use even smaller and smaller holes to place data into, which causes some fragmentation.

We have more but we USE more.

[pla]

Today, however, with a lot of file systems in the Terabyte range, a 90-95% full file system can still have a considerable amount of free space but we still mostly get bugged by the same alerts as in the days of yore when there really isn't a cause for immediate concern.

When we had drives in the 100s of MB range, we used a few MB at a time. Now that we have drives in the multi-TB range, we tend to use tens of GB at a time. In my experiences, a 90 percent full drive has as much time left before running out as it did a decade ago.

Perhaps more importantly, running at 90% of capacity kills your performance if you still use spinning glass platters as your primary storage medium (not so much when talking about a SAN of SSDs). In general, when you hit 90% full, you have problems other than just how long you can last before reaching 100%.

Re:We have more but we USE more.

[Vellmont]

Exactly. The question is strange (and the attitude of the poster is odd too... 20 years ago is "days of yore", and "olden days"?) Methinks dusting off the word "whippersnapper" might be appropriate here.

Oddly enough, a similar question fell through a wormhole in the space time continuum from Usenet, circa 1994. "Now that we have massive HDs of 100s of megabytes, and not the dinky little ones of several megabytes from the Reagan era, do we still have to worry about having 95% usage alarms?"

The truth being, if you got to 95% usage somehow, what makes you think that you're not going to get to 100% sometime soon? Maybe you won't, but you can't know unless you understand how and why your usage increases. That's not going to be solved by a magic algorithm alone, it involves understanding where your data comes from, and who or what is adding to it. This isn't new. The heuristics and usage question, and estimating when action needs to be taken is just as relevant now as it was 20 years ago.

Re:Performance issues?

[gnasher719]

ou want to keep the hard drive at 50% or less to maximize performance. If the hard drive is more than 50% full, the read/write head takes longer to reach the data. If the hard drive is 90% full, most OSes will have performance issues.

Actually, any OS will have performance issues, because the transfer rate (MB/sec) drops from the outside tracks to the inside tracks. That's why for home use, you just buy the biggest hard drive that you can easily afford (if you need 1TB, you buy 3TB), because that way you use only the parts of the drive with the highest transfer speed, and the average head movement time is also a lot less.

Recommend: Hard Drive Sentinel

[Bomarc]

I install the shareware version of Hard Drive Sentinel on all my Windows systems. It not only will warn you about hard drive usage (%); it will also warn you about errors on the drive -- and in my case I was able to predict that two drives were going to fail (saving data) before they actually failed.

Their support has been very responsive and courteous, their product can work through (see drives behind) most RAID controllers.

And no, I don't have any affiliation with HDS.

Whatever is measured is optimized.

[QuietLagoon]

...when there really isn't a cause for immediate concern.

It all depends what one is concerned about. Is maximizing disk space down to the last possible byte important to you? Or is performance in accessing random data important to you? Or is wanting to keep artificial limits imposed by monitoring systems important to you?

.
Once you determine what is actually important to you, then you monitor for that parameter.

Whatever is measured is optimized.

Re:Bigger question

[Bigbutt]

It's a configuration option when you newfs a file system. Man newfs or mkfs.

[John]

Re:Performance issues?

[kuzb]

That's an interesting idea for the budget-minded, but personally I think if performance is actually an issue I'd use SSDs for things that need to be performant, and store everything else on regular drives.

Re:Performance issues?

[afidel]

Inner tracks have better seek times, which is why high performance applications often "short stroke" drives (ie artificially restrict the percentage of the drive used so that only the inner tracks are utilized, though with modern drives and transparent sector remapping it's unlikely this practices actually works), outer tracks have better streaming performance because more sectors move under the head in a given timeframe.