Watching Under The Hood Of Tiger's Spotlight

jaketheitguy writes "Over at KernelThread.com, Amit Singh has released a commandline app called FSLogger for looking under the hood of Tiger's Spotlight. You can watch all kinds of filesystem changes going on in realtime. The utility apparently intercepts and displays filesystem change data as it goes out to Spotlight from the kernel. It even tells you which app is making the changes. Looks like Apple has included some pretty powerful API's in Tiger and there may be some othre really interesting uses of this API as mentioned on the app's page. I for one would really like to be able to tell if somebody changed ANY files on my system without my knowledge. I think you can do that with Singh's program, but how do you make sure somebody cannot disable the program?"

Comments

Spotlight changed my life.

[duffbeer703]

I used to be a lonely nerd, but thanks to Spotlight I can:

- Run Faster
- Jump Higher
- Score with the chicks
- Regrow lost hair!

Re:Spotlight changed my life.

[zero_offset]

Disclaimer: If any of these conditions persist for more than four hours, seek medical attention immediately.

Recursion

[dangitman]

Amit Singh has released a commandline app called FSLogger for looking under the hood of Tiger's Spotlight. You can watch all kinds of filesystem changes going on in realtime.

So, this application would shine a spotlight on Spotlight? Is that anything like when you point a video camera at a monitor hooked up to the camera's output?

Tripwire

[@madeus]

Actually you can get this functionality already in a long standing Unix utility called Tripwire.

http://www.tripwire.com/
http://sourceforge.net/projects/tripwire/

There is even a Mac OS X version now it seems:
http://www.macguru.net/~frodo/Tripwire-osx.html

Of course you'd probably then want an OS that implements some form of relevant Mandatory Access Control / POSIX.1e (e.g. LIDS for Linux, Trusted Solaris, or Argus Pitbull (Linux/Solaris)) to help prevent the intruder from interfering with Tripwire itself.

Re:Tripwire

[womby]

I am going to assume you didn't read the article and provide a small description of what fslogger is doing and how it has nothing in common with tripwire.

Fslogger runs continuously and registers itself with the kernel, when a filesystem change event happens details about it are announced to all registered apps and fslogger displays the information it receives in a useful (if verbose) manner.

Tripwire is a fantastically useful app which I run on every one of the servers I admin, and perhaps the OSX version could be extended to make use of the same kernel interfaces that fslogger is using.

Tripwire runs once per day (however often you wish to run it), and scans the filesystem checking each file to see if it matches a checksum calculated at some known time in the past. This is useful on mission critical servers because outside of data / user directories changes should happen very infrequently. Tripwire is a robust way of confirming that a server has the same configuration on a day to day basis.

The startling difference between tripwire and fslogger should be obvious, tripwire has no mechinism to know when a file has changed except by looking at the file directly, fslogger has no mechinism to know if the event is important or not and no mechinism to notify an administrator of the event short of scrolling it by in a terminal.

With tripwire you could delete a file, recreate it from backup and so long as you was careful tripwire would never know. fslogger would display every step you took but would not know if the final step returned the filesystem to its original state.

Different tools with different behaviours for different target users presenting different information in a different manner.

So, to dispute your original assertation,
"Actually you can get this functionality already in a long standing Unix utility called Tripwire."
No, nothing fslogger does is replicated by tripwire and nothing tripwire does is replicated by fslogger.

Tracking changes to the file system

[Simon Spero]

There's a system call that lets user-space programs subscribe to a lot of interesting kernel level events.

Take a look at the kqueue(2) man page.

There are more details available at http://people.freebsd.org/~jlemon/papers/kqueue.pd f

Re:Tracking changes to the file system

[argent]

Yeh, when I heard about this I assumed that Apple would use kqueue and watch changes in the vnodes. It would require some extension to kqueue, because there's no "EVFILT_FS" or "EVFILT_VOP" filter that would monitor VOPs on more than a single file. But they needed to extend HFS+, too, so that's not really a big deal. You do have to be careful with this, because trying to monitor VOP_WRITE would be like drinking from a firehose... but you wouldn't actually need to track file content changes that closely for something like Spotlight that only needs to know that a file has changed recently. The key is that file content changes are reflected in the file modification time, so changes to the stat structure would be good enough if you delayed the actual examination of the file more than the granularity of time_t (one second), so by monitoring a few key VOPs you could get Spotlight working efficiently over any file system.

In fact, you could even just track inode changes and VOP_OPEN, VOP_MMAP, and VOP_CLOSE, and periodically peek at files that are open a long time to see if they're changed. The main thing is to be able to tell where to look without having to regularly traverse the whole file system.

Why they decided to use HFS+ instead of doing it at the vnode layer, I don't know. I can make some pretty good guesses, of course, because after all HFS+ is their baby and they really don't care much about supporting other file systems.

It's a shame. I really don't trust HFS+, and I wish they'd do more to support UFS transparently.

Re:Where's "As Seen on TV" ...

[yanndug]

Interesting. Steve Jobs is out of the office these days... http://www.thinksecret.com/news/0505itunes49.html

Tripwire

[mithran8]

You may be shocked to know how often files change on your system... without a good policy defining the scope of your monitoring, you're asking for a world of hurt. As @madeus mentions, there is an OS X build of Tripwire which gives you a good deal of this functionality. Two caveats, however:

- Tripwire is not a real-time service, it's scheduled to run at specific (user-defined) times.

- Tripwire does not prevent anyone from making changes - it merely ensures that any changes to the OS are recorded and made visible to you.

That said, Tripwire is a very functional tool with excellent scripting and integration potential. Plus, it has a good amount of internal security - unless you know the relevant passphrases, you cannot subvert the product. If you root the box you can always uninstall it, but you can't tamper with the database or policy to hide your changes or trick Tripwire into sending a false 'all clear' message.

As with all such tools, it's best to check it out and evaluate it for yourself to see how it works for you.

Re:Where's "As Seen on TV" ...

[Reaperducer]

I really really really really really really hope that Steve Jobs has something better to do than read Slashdot. If not, then there's no hope for the rest of us.

ASOT is not Steve Jobs

[That's Unpossible!]

Here are three reasons why:

1. ASOT is too familiar with the technical underpinnings of Apple technology. Steve Jobs is smart smart smart, a great businessman, but there is no way he is this familiar with all the technical details. That was what Woz was for, remember? (No I'm not implying this is Woz, since he clearly no longer has this much access to Apple.)

2. There's no way the CEO of a public company would risk the MAJOR, MAJOR, MAJOR lawsuits and trouble that could be caused from the SEC and shareholders by divulging valuable information on Slashdot. There are rules the company officers must strictly follow in regards to how they divulge information previously unknown to the public. The information must reasonably be made publically available, not posted anonymously on Slashdot.

3. Steve Jobs gets more bang for his buck by keeping things top secret until the next time he's doing a keynote.