Slashdot Mirror
Auditing Large Unix File Systems?
jstockdale asks: "The recent article on perpendicular recording hard drive technology brought me, as a unix(tm) admin, to reflect on the management of data systems and file servers of capacities >1TB (which exist today and tomorrow will become commonplace). Since Google for once seems useless, what suggestions does the Slashdot crowd have on methods and software to audit changes, visualize file system usage, and in general to determine the qualitative and quantitative nature of the content of large unix file systems?"
Visualize file system usage: df
Audit changes: diff
Determine the qualitative nature of a large unix file system: How many positive reviews I see of it on googlefight, how shiny the box for the drive looks.
Determine the quantitative nature of a large unix file system: Number of cylinders, number of heads, number of platters, number of sectors, bytes in a sector, weight, physical dimensions.
Thank you. I'll be here all week. This is not funny.
It's the same as visualizing a small file system, except that the files are an order-of-magnitude bigger (DVD rips, databases...)
I like the idea of treemaps.
x .s html
:)
http://www.cs.umd.edu/hcil/treemap-history/inde
Hehe, it was originally made to see what was taking up all the room on an 80MB hard disk
There's various software available based on this concept, most working like "du", except that you get the results graphically. You typically see a large picture on screen of what directories and files are taking up the most space. It looks like a piece of Mondrian artwork, with the size of rectangles corresponding to the size of space taken, so it is easy at a glance to see what is hogging all of the disk space. It can be drilled down, of course, by clicking to zoom in.
A quick Google search revealed SequoiaView:
http://www.win.tue.nl/sequoiaview/
Unfortunately this only runs on Windows, but I'm sure there are similar Linux programs available.
Dr. Demento On The 'Net!
To make a guess, perhaps as storage is growing fast, but read times are not, his drives are getting filled up faster than he can run "du -s -m | sort -n " to figure out whose's filling them up?
Also, whose to say file size increases with storage capacity? Perhaps at his site, number-of-files increases with storage capacity, with the file size staying statistically constant. MB for MB, traversing lots of little files is harder than traversing a few big files.
I always found xdu nice to see where the disk space is burned, but I understand this doesn't scale too well for large filesystems.
:-)
Maybe the quota subsystem can be used for something...
- Hubert
>1TB capacities commonplace tomorrow!? Wow!!
Too bad for me that payday is almost two weeks away...
This user account is inactive account replaced by the PDA
produce a load of products that do all that kinda thing for you. Expensive, but then you could always employ a bunch of monkeys, erm, summer students to do the same stuff manually...
Don't get me wrong here -- in the event that you have a multi TB system keeping track of usage with 'du' really just isn't practical, but do you really even have to ASK the box where the data is?
/oradata there's the database. It gets it's own space. When that gets full he doesn't do an 'rm' to clean things up -- He has to use Oracle tools to do that.
/filestores there's a giant mess of crap that nobody can make heads or tails of. They're hashed filename from a PDM system. He can't do anything to clean that out without -- you guessed it - using the PDM system. The filesystem itself has NO idea what's going in either of it's major usage sections. It's just "stuff" and to rm -rf a directory because you're running out of space would be foolish.
Our backend storage system for my project is 1TB, or at least very close to it. I don't manage the box, but I do work with it. It holds three things:
1) It's OS (small)
2) It's Oracle Database files (300GB on disk, about 200GB used now)
3) Files. Word documents, CAD drawings, TIF, GIF, etc. A whole slew of them.
The admin knows what's using what. Under
Under
When filesystems can actually hold metadata regarding their contents then I'd give this question some though. We could have a whole new set of Unix tools to modify our everything-is-a-file-with-badass-meta-data system. Until then I don't see any way for filesystem maintence to be a huge issue on this multi TB systems. All you can really do with the FS is determine which system needs morespace and order more disks. You can't trim or manage it with the FS.
I'm wrong a lot though, but that's my take on the "issue".
WinFS?
Time flies like an arrow. Fruit flies like a banana.
I have a 8.8TB (raw) EMC Symm here, and another one in Austin TX. Then I have another 600GB or so in Sun jbod. Most hosts are connected to the EMC over a 2Gb SAN using brocades.
I just got this environment a couple weeks ago, and there was NO documentation. So figuring out disk usage over 15 systems has been a nightmare.
As much as I hate to admit it, EMC's ECC and Storage Scope have been a huge help. I could have done it using Veritas as well, but the EMC tools are nice.
And I'm soon going to add another 2TB SATA array over iSCSI so then we'll see if ECC can really manage "other peoples disk"
But welcome to the new exciting field of Storage/SAN architect/admin. With arrays from HDS and EMC coming in the 46TB flavor and more, resorce management is a big job.
For instance my counterpart (he handles windows, I handle *nix) found that we had 1.2TB of unused BCV's (it's a 3rd mirror, weird EMC'izm) that had never been used!
So it's all about documentation. And do yourself a huge favor, and come up with a clean scalable disk/lun/volume/mountpoint naming convention. This can be so critical. Not sure what other people do, but I'll have something like:
host 1: dg001-dg005
host 2: dg006-dg010
host 3: dg011-dg015
the disks are enclosure_name_id_lun#
volumes are v{dg}{v#} so first volume for disk group dg006 would be v1601.
Then mount points are based on either oracle{SID} or port, or app name or something logical.
Keep everything unique! So then you can move luns form host to host without having issues. This also allows you to generate usage reports and know that if v3206 is at 68% exactly what host, disk group, volume, and app are involved.
Ug, sorry, went off, but I'm in this mess right now myself and so I have some very strong feelings about it right now =)
What you want is a simulacrum of Ztree, which is a shareware version of XTree Gold, the version you are looking for would of course run on your preferred OS (which IIRC Ztree does not.)
The arena gets larger, the things you are tracking get larger, but the ability to view branches of your directory structure, treat all the files on the entire set of drives as if they were all a congruent bunch (sort them all by size, date, extension for temporary files, etc...) will be invaluable.
If nothing exists that replicates the functionality of XTree Gold on your OS (ie, Ztree) then learn the functionality and replicate it - all it does is recursively read the directory contents of each dir on the drive and keep them all in memory, then let you work on them irrespective of directory (global access to your files, across all the logged drives.)
www.ztree.com
And no, I don't work for them, but I have been an XTree fan since about 1986 or thereabouts.
Glonoinha the MebiByte Slayer
Just take all the data on your disks, and do a low-level scan. Basically, take each byte and perform a parity check so you get a 1 if there's an odd number of 1's in the binary representation, and a zero otherwise.
Now, here's the secret: take all these zeros and ones, and do a parity check on THEM. BLAM! Your entire array is now down to ONE status bit!!!
Now take a big crayon and write that status bit on a piece of your favorite color paper. Put it up in the machine room for all to see. Or just slip it in your drawer if you think that letting this kind of information out is a security leak. Your call.
Then, repeat the process once an hour or so. Today's arrays are so fast that it shouldn't take long. Each time you get the digit, the zero or the one, compare it to the last output. If it's changed (for example, from 1 to 0 or 0 to 1), than WHAMO, you've got SOMETHING going on, better check it out!!!
This "early warning system" gave me a "heads up" to some serious probablems more than once. You might want to check it out, so-called "storage experts" EMC didn't even have a package to do this so you might do a little coding in VB, but it's worth it!
For example, on one recent project on which I worked, a PeopleSoft/Oracle environment was built on a pSeries system. There were instances for every conceivable piece of architecture which led to 50+ file systems. And we're not talking about 50+ file systems off of /, but file systems within file systems within file systems. This was good for separating data but made df an ugly mess.
Conversely, I worked on another project for with a homebrew app designed to track tickets. Rather than using a database the genius who designed the methodology stored every ticket as a 1024-byte file. This caused the system to eat up inodes even though the NBPI was set to 1024, and it caused an additional fun feature: the ls command could not work. With over 200,000 files per file system (all in one directory), ls could not parse in all of the files. The homebrew guy actually had to write an app to crack open the inode table to list the files. When you set up your environment, first consider what degree of granularity you need.
Next, document everything. Consider this situation: an HP Virtual Array with 100 LUNs each cabled to two brocade switches for redundancy going to ten different systems. Would you know just by popping a cable what effects would happen? Documentation is key for managing large disk/file system environments. This also applies to naming file systems, logical volumes, volume groups, and any other pat of your system.
--Chag
You could check commerical software such as Veritas' new Storage Reporter(formerly Precision Software's product).
Should be available for Linux among other OSes.
I joined two users too late.