Slashdot Mirror

← Back to Stories (view on slashdot.org)

Watching Under The Hood Of Tiger's Spotlight

Posted by Hemos on from the looking-at-it dept.

jaketheitguy writes "Over at KernelThread.com, Amit Singh has released a commandline app called FSLogger for looking under the hood of Tiger's Spotlight. You can watch all kinds of filesystem changes going on in realtime. The utility apparently intercepts and displays filesystem change data as it goes out to Spotlight from the kernel. It even tells you which app is making the changes. Looks like Apple has included some pretty powerful API's in Tiger and there may be some othre really interesting uses of this API as mentioned on the app's page. I for one would really like to be able to tell if somebody changed ANY files on my system without my knowledge. I think you can do that with Singh's program, but how do you make sure somebody cannot disable the program?"

32 of 43 comments (clear)

  1. Two in a row? by jack_call · · Score: 1, Offtopic

    Although it's two different areas, isn't two articles on spotlight a little extreme?
    Come on Hemos, lets have a hattrick :-)

    and oh... I for one welcome our new Spotlight overlords

    --
    This is my sig. There are many like it but this one is mine. My sig is my best friend. It is my life.
  2. Spotlight changed my life. by duffbeer703 · · Score: 3, Funny

    I used to be a lonely nerd, but thanks to Spotlight I can:

    - Run Faster
    - Jump Higher
    - Score with the chicks
    - Regrow lost hair!

    --
    Conformity is the jailer of freedom and enemy of growth. -JFK
    1. Re:Spotlight changed my life. by zero_offset · · Score: 2, Funny

      Disclaimer: If any of these conditions persist for more than four hours, seek medical attention immediately.

      --

      Slashdot quality declines as the number of hot grits posts decreases. - Provolt's Law, Apr-09-2005

    2. Re:Spotlight changed my life. by valkraider · · Score: 1

      Ask you doctor if "Spotlight" may be right for you!

      Possible side effects may include but are not limited to data loss, computer malfunction, loss of electricity, rugburn, high phone bills, cataracts, auto repossession, and in rare cases death and/or dismemberment and eternal damnation. Use only as directed.

  3. Two stories in 20 minutes? by Mensa+Babe · · Score: 1

    It seems that the Spotlight is in the spotlight. Contrary to what I said before, the AC might have been right about Spotlight being overhyped in the extremes. It is overhyped to the max.

    --
    Karma: Positive (probably because of superiour intellect)
    1. Re:Two stories in 20 minutes? by Anonymous Coward · · Score: 1, Funny

      Heu's froum Canadau. Theuy speull wourds wiuth louts ouf euxtra U's iun theum theureu.

  4. Recursion by dangitman · · Score: 2, Funny
    Amit Singh has released a commandline app called FSLogger for looking under the hood of Tiger's Spotlight. You can watch all kinds of filesystem changes going on in realtime.

    So, this application would shine a spotlight on Spotlight? Is that anything like when you point a video camera at a monitor hooked up to the camera's output?

    --
    ... and then they built the supercollider.
    1. Re:Recursion by wealthychef · · Score: 1

      Honestly, if you were to redirect the output of FSLogger to a file... wouldn't it then bog the system down?

      --
      Currently hooked on AMP
  5. This story seems utterly confused. by Anonymous Coward · · Score: 1, Informative

    First there always has been a program called /usr/bin/fs_usage which monitors file system access. Second is the story writer worried about someone altering his files or about spotlight. How spotlight functions has been the subject of many detailed articles. Any time you change a file, spotlight calls the appropriate indexing program and collects and stores the metadata. It is not going to alter the data fork of your document but the data has to get stored somewhere. So relax. this story is paranoid delusional.

  6. IDS Potential by Sentry21 · · Score: 1

    This has a lot of potential in the server market. Imagine an IDS that monitors certain files for changes and notifies the sysadmin immediately whenever a static file is updated. The system could have scheduled periods for upgrades, during which it doesn't send a thousand warnings to you, but other than that, it could monitor all disk activity at a low level without being subverted by e.g. changing the IDS's file hashes before it does its next check.

    Interesting idea.

  7. Where's "As Seen on TV" ... by GaryPatterson · · Score: 1

    ... when you need him?

    He was very vocal about this sort of thing, and now he's gone very quiet. Almost as if he was an Apple employee who was given The Warning (tm) or... (obligatory Star Wars reference being used in shameless Karma whoring) ... as if a million of his posts were made, and then suddenly silenced. Hmm...

    When I get some time, I'll read the article (thus breaking a long-running streak for me) and compare to ASoT's statements.

    1. Re:Where's "As Seen on TV" ... by As+Seen+On+TV · · Score: 1, Informative

      No, I'm just out of the office this week.

    2. Re:Where's "As Seen on TV" ... by Elwood+P+Dowd · · Score: 1

      So, not only do you talk sense about unreleased Apple products while being an Apple employee, you do so from your office at Apple?

      Awesome.

      --

      There are no trails. There are no trees out here.
    3. Re:Where's "As Seen on TV" ... by yanndug · · Score: 3, Funny

      Interesting. Steve Jobs is out of the office these days... http://www.thinksecret.com/news/0505itunes49.html

    4. Re:Where's "As Seen on TV" ... by SYFer · · Score: 1

      Although I cannot believe these posts are from Jobs himself (as the poster below and others have suggested), he's either astroturfing /. in a provocative way on behalf of Apple--probably with Jobs' express consent, or he's truly doing it on the sly. Given Apple's secretive nature, I'd say that if it's the latter, he'd be a fool to post from the office as he would be easily identified via logs and, I can only assume, be shown the door. And he does not strike me as a fool.

      If the AC's observation is correct and he is surfing from 1 Infinite Loop, it's clear to me that he's doing it with the blessings of the company. On the other hand, the appearance of Apple IPs in the logs could be from anyone at Apple--/. is a major tech site and I'm sure it's read by a great many of the employees. Hell, even a security detail assigned to rooting him out could generate the hits.

      I guess you'd have to build a sort of ASOTV honeypot (or "AsottyPot") to track clicks, but I'm not exactly sure how you would set it up or sift the results. You'd have to post compelling honeypot links in response to his comments then maybe look for hits that closely match his posting times and look for consistent patterns. Joe Apple Lunch Hour Surfer would not consistently know when every ASOTV post appeared, so if a pattern emerged, you might have something.

      Bottom line though, it would be very easy to trace him internally if he were posting from campus. He's also given a few specific personal details over time (the island thing, for example), but those could be clever red herrings, of course. I'm leaning toward his being an authorized insider poster and frankly think it's a smart thing. Apple is always concerned about the online zeitgeist, so why not join in the fray. Are they bound by the Prime Directive?

      --
      "...all the labours of the ages, all the devotion, all the inspiration, all the noonday brightness..." yada yada
    5. Re:Where's "As Seen on TV" ... by Reaperducer · · Score: 2, Funny

      I really really really really really really hope that Steve Jobs has something better to do than read Slashdot. If not, then there's no hope for the rest of us.

      --
      -- I'm old enough to have lived through six different meanings of the word "hacker."
    6. Re:Where's "As Seen on TV" ... by GaryPatterson · · Score: 1

      Cool bananas. You've been quiet lately, so I wondered...

    7. Re:Where's "As Seen on TV" ... by paco_loco · · Score: 1

      ASoTV doesn't seem too bothered about giving out details of his movements does he? If anyone at Apple really wanted to identify him it wouldn't be too hard if you read all his posts...?

    8. Re:Where's "As Seen on TV" ... by That's+Unpossible! · · Score: 1

      I do like the insider's look at Apple, also, but I think Apple would be opening itself up to shareholder lawsuits and SEC troubles if it has officially told an employee it is OK to post "insider info." For example, the information ASOT posted in the discussion of the video capabilities in iTunes could be considered vital information that should not be divulged to only a small group of people.

      --
      Ironically, the word ironically is often used incorrectly.
  8. Tripwire by @madeus · · Score: 3, Informative

    Actually you can get this functionality already in a long standing Unix utility called Tripwire.

    http://www.tripwire.com/
    http://sourceforge.net/projects/tripwire/

    There is even a Mac OS X version now it seems:
    http://www.macguru.net/~frodo/Tripwire-osx.html

    Of course you'd probably then want an OS that implements some form of relevant Mandatory Access Control / POSIX.1e (e.g. LIDS for Linux, Trusted Solaris, or Argus Pitbull (Linux/Solaris)) to help prevent the intruder from interfering with Tripwire itself.

    1. Re:Tripwire by womby · · Score: 2, Informative

      I am going to assume you didn't read the article and provide a small description of what fslogger is doing and how it has nothing in common with tripwire.

      Fslogger runs continuously and registers itself with the kernel, when a filesystem change event happens details about it are announced to all registered apps and fslogger displays the information it receives in a useful (if verbose) manner.

      Tripwire is a fantastically useful app which I run on every one of the servers I admin, and perhaps the OSX version could be extended to make use of the same kernel interfaces that fslogger is using.

      Tripwire runs once per day (however often you wish to run it), and scans the filesystem checking each file to see if it matches a checksum calculated at some known time in the past. This is useful on mission critical servers because outside of data / user directories changes should happen very infrequently. Tripwire is a robust way of confirming that a server has the same configuration on a day to day basis.

      The startling difference between tripwire and fslogger should be obvious, tripwire has no mechinism to know when a file has changed except by looking at the file directly, fslogger has no mechinism to know if the event is important or not and no mechinism to notify an administrator of the event short of scrolling it by in a terminal.

      With tripwire you could delete a file, recreate it from backup and so long as you was careful tripwire would never know. fslogger would display every step you took but would not know if the final step returned the filesystem to its original state.

      Different tools with different behaviours for different target users presenting different information in a different manner.

      So, to dispute your original assertation,
      "Actually you can get this functionality already in a long standing Unix utility called Tripwire."
      No, nothing fslogger does is replicated by tripwire and nothing tripwire does is replicated by fslogger.

      --
      **** lying is wrong even for sleeping dogs
    2. Re:Tripwire by womby · · Score: 1

      I didn't notice that you was replying to somebody who suggested creating tripwire, I thought you had posted a comment specifically about flogger.

      Sorry if it the reply came across as harsh.

      --
      **** lying is wrong even for sleeping dogs
    3. Re:Tripwire by @madeus · · Score: 1

      Hehe, easily done. :)

      I think the idea of having Tripwire hooks so that it's automatically informed of changes real time when on Mac OS X is certainly interesting and I'd think eminently doable.

      I think true real time updates may actually have been a feature of a commercial implimentation (for Solaris), but that would be going back 7-8 years ago now, so I'm not certain (it could have been just a daemon that periodically checked for changes, or I may have remembered wrongly).

      PS: I hadn't heard the name 'fslogger' used for the real time file monitoring feature in the Dawrin kernel - though I've not actually had a name for it all (it seems weird that it wouldn't have one, given it impacts not just things like Spotlight, but Automator and even the Finder, which now updates Windows contents in real time) - but is it perhaps just the name for a utility that lets you view the change history?

  9. Tracking changes to the file system by Simon+Spero · · Score: 3, Informative
    There's a system call that lets user-space programs subscribe to a lot of interesting kernel level events.

    Take a look at the kqueue(2) man page.

    There are more details available at http://people.freebsd.org/~jlemon/papers/kqueue.pd f

    1. Re:Tracking changes to the file system by argent · · Score: 2, Interesting

      Yeh, when I heard about this I assumed that Apple would use kqueue and watch changes in the vnodes. It would require some extension to kqueue, because there's no "EVFILT_FS" or "EVFILT_VOP" filter that would monitor VOPs on more than a single file. But they needed to extend HFS+, too, so that's not really a big deal. You do have to be careful with this, because trying to monitor VOP_WRITE would be like drinking from a firehose... but you wouldn't actually need to track file content changes that closely for something like Spotlight that only needs to know that a file has changed recently. The key is that file content changes are reflected in the file modification time, so changes to the stat structure would be good enough if you delayed the actual examination of the file more than the granularity of time_t (one second), so by monitoring a few key VOPs you could get Spotlight working efficiently over any file system.

      In fact, you could even just track inode changes and VOP_OPEN, VOP_MMAP, and VOP_CLOSE, and periodically peek at files that are open a long time to see if they're changed. The main thing is to be able to tell where to look without having to regularly traverse the whole file system.

      Why they decided to use HFS+ instead of doing it at the vnode layer, I don't know. I can make some pretty good guesses, of course, because after all HFS+ is their baby and they really don't care much about supporting other file systems.

      It's a shame. I really don't trust HFS+, and I wish they'd do more to support UFS transparently.

  10. Physical security essential by davidwr · · Score: 1

    "how do you make sure somebody cannot disable the program?"

    You can't, not withint guarenteeing physical security to the box.

    If someone can pull your hard disk OR boot with their own media, all is lost.

    Short of that, your question amounts to "how do I keep from getting rootkitted."

    --
    Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
  11. Tripwire by mithran8 · · Score: 2, Informative

    You may be shocked to know how often files change on your system... without a good policy defining the scope of your monitoring, you're asking for a world of hurt. As @madeus mentions, there is an OS X build of Tripwire which gives you a good deal of this functionality. Two caveats, however:

    - Tripwire is not a real-time service, it's scheduled to run at specific (user-defined) times.

    - Tripwire does not prevent anyone from making changes - it merely ensures that any changes to the OS are recorded and made visible to you.

    That said, Tripwire is a very functional tool with excellent scripting and integration potential. Plus, it has a good amount of internal security - unless you know the relevant passphrases, you cannot subvert the product. If you root the box you can always uninstall it, but you can't tamper with the database or policy to hide your changes or trick Tripwire into sending a false 'all clear' message.

    As with all such tools, it's best to check it out and evaluate it for yourself to see how it works for you.

    --
    An object at rest cannot be stopped!
  12. ASOT is not Steve Jobs by That's+Unpossible! · · Score: 2, Insightful

    Here are three reasons why:

    1. ASOT is too familiar with the technical underpinnings of Apple technology. Steve Jobs is smart smart smart, a great businessman, but there is no way he is this familiar with all the technical details. That was what Woz was for, remember? (No I'm not implying this is Woz, since he clearly no longer has this much access to Apple.)

    2. There's no way the CEO of a public company would risk the MAJOR, MAJOR, MAJOR lawsuits and trouble that could be caused from the SEC and shareholders by divulging valuable information on Slashdot. There are rules the company officers must strictly follow in regards to how they divulge information previously unknown to the public. The information must reasonably be made publically available, not posted anonymously on Slashdot.

    3. Steve Jobs gets more bang for his buck by keeping things top secret until the next time he's doing a keynote.

    --
    Ironically, the word ironically is often used incorrectly.
    1. Re:ASOT is not Steve Jobs by Decipherer's+Dream · · Score: 1

      Has anyone noticed yet that As Seen On TV is an anagram for Steve's Anon? (Anon could mean Anonymous).

      I don't know of any Steve at Apple except Jobs, so what should we conclude?

    2. Re:ASOT is not Steve Jobs by cvdwl · · Score: 1
      I don't know of any Steve at Apple except Jobs, so what should we conclude?

      That ASOT is a witch!!

      --
      ... grumble, grumble, grumble, mutter, mutter, Millenium... Hand... Shrimp, I tol' 'em, I tol' 'em.
    3. Re:ASOT is not Steve Jobs by CableModemSniper · · Score: 1

      Burn her! Burn her!

      --
      Why not fork?
  13. Re:Just use fs_usage by Lucractius · · Score: 1

    On a side note this kind of thing is (though i cant check exactly im just going from the info i have on what spotlight does cause Well. I dont have a mac :P ) also available for windows machines through programs available at www.sysinternals.com

    These guys are utter LORDS of the nt OS by any definiton. ( read their "About us" section and see just how A class it is. A Microsoft Most Valued Proffesional no less )

    Anyway. There are filesystem access and notification tools around for nearly any os and its good to see OS X realy making a push with them instead of the way theyer usualy swept under the rug in most OSes publicity stuff (not that many oses have publicity to speak of lol )

    --
    XML - A clever joke would be here if /. didn't mangle tag brackets.