Slashdot Mirror
Examining Mac OS X 10.4's Spotlight
Ton writes "Apple has published a discussion of Spotlight, the radical systemwide search technology that will be part of Mac OS X 10.4 'Tiger'. The really interesting part is that metadata will be playing a big role in Spotlight while just a few years ago people were afraid metadata in Mac OS X was going the way of the dodo."
Spotlight is basically a SQLite db that holds data about documents and files on your system. Metadata is gathered by a sort of 'plug-in' for each different file type.
A Typical use will be making query's such as: Show me everything agent dero sent me between tuesday and thursday last week. Mails, IM transfered images, you name it... Best of all, since this is metadata based, it's supposed to be lightning fast
You could envision a plugin that would Spotlightify slashdot threads you read, in theory, and apply the power of a database to it.
but really, you should RTFA
Exercise caution when modding this message up: the author acts like a jerk when his karma is excellent.
You must have a different version of locate to me. I can't get mine to index my emails, it has no idea about the metadata entries in common document types and can't tell the difference between an image and a movie file.
Could you send me the source for the version you have installed that does that?
The linked article is shit.
. htmlYou want this one instead, its got loads more info on what it does and how it works, plus some code examples for the gimps.
http://developer.apple.com/macosx/tiger/spotlight
Shitram Brown, PhD
Professor of Mathematics
um...
no
Perhaps you don't have the imagination to see how it is different. Imagine being able to type in "dog" and get everything on your drive that has dog in it anywhere. In the title, the text, the metadata, within pdfs......
another goose
People who have used it report no performance degredation. And no, its nothing like Windows search, which Mac OS has also had since System 8 or earlier.
For one, it doesn't take half an hour, it shows you the results as you type, instantaneously.
Secondly, via plugins it can understand *any* file, such as an image metadata importer that uses OCR so you can search for words, or a Flesh-tone detector so you can search for all your porn that way.
Shitram Brown, PhD
Professor of Mathematics
>You must have a different version of locate to me.
That might be the case.
There are no atheists when recovering from tape backup.
makewhatis.cron can be a pain on Linux as well, if it is on a workstation which is mostly switched off.
Unfortunately for windows boxes, they do tend to be left shut down a lot of the time, so more of their runtime is spent rebuilding the search database when the machine is being used for something, rather than in the middle of the night, which is the preferred way
http://michaelsmith.id.au
The post links to the Apple Spotlight page that has been there for months. Is THIS the "discussion" that is being referred to in the post?
>>> "Apple has published a discussion of Spotlight, the radical systemwide search technology that will be part of Mac OS X 10.4 'Tiger'.
. html
What's really funny is that there's no link to the actual published discussion... but anyway...
http://developer.apple.com/macosx/tiger/spotlight
The radical difference is that Spotlight generates the metadata itself rather than you having to tag stuff yourself. It has content handlers to intelligently tag all kinds of different "stuff" so it "knows" what a Word document is and what a web page is and what a .png file is etc etc.
Bad analogies are like waxing a monkey with a rainbow.
Already the differences in Fat32/NTFS versus HFS+ (the mac filesystem) yield significantly faster searches before spotlight is introduced. Sit down on an OSX apple and notice that an entire search of the HD is actually a fast operation, not the waiting many-minute exercise that it is on windows.
Now since spotlight is built into the core of the system, and isn't just a tack-on service like the windows indexer is, there are significant speed advantages, updating the SQL database when files are modified, added, etc is incredibly light on the CPU, and is equivalent to doing something like changing the file name.
What spotlight isn't, and this might be where you are getting confused, spotlight isn't a spider that crawls from folder to folder cataloguing information about each file, which is what the windows indexer was doing, hence why it was resource intensive, as it was busy checking files and folders that you have possibly not made any changes to.
As a counter to the 'Filesystem metadata is great, but "instantly" updated search indexes sounds like a solution to a problem that doesn't really exist.' Microsoft, google and apple would disagree. Having an up-to-date catalogue without the CPU strain is a must have, go figure MS have been trying to implement it since NT4.0.
As mentioned, I think it's the plugin architecture that makes it special. That makes it possible to search for anything that you can imagine. For example, you could write plugins for your logfiles, movie subtitles, internet cache, etc. It's basically your imagination that sets the limit.
To my knowledge, other metadata-based search systems have not had a similar degree of extensibility. Please correct me if I'm wrong.
Apple are well known for optimising their software to be significantly faster with each pre-release build. Having had the opportunity to test the developer tester of 10.4 with spotlight on a 12" powerbook (which was bogged down with various applications at the time) I can assure you that spotlight remained snappy, and definitely true to the 'instant' claim (I've noticed apple are quite careful on not over advertising their products, as it cause more problems than sales and a bad image). After using microsoft products we become very used to how slow a process can be. Apple's advantage is clear, they know their target hardware, like video-card driver writers they can optimise any part of their OS to fit their hardware for optimum speed. Additionally the g4/g5 chipsets have some quite useful registers for performing these sorts of searches (think sort of like MMX for x86, except with developers actually utilising them outside of games)
I still have to be convinced that full-content indexing is a good idea. I very rarely need to search for something in the contents of a group of files, and when I do it's usually such a small group that the time saved would not outweigh the disk space used by such large indexes. On the other hand, this problem should get better over time, since the largest files are usually video, and have little indexable content, meaning that the index is likely to get relatively smaller over time (until someone writes a plug-in that can interpret objects in images, and applies this to every frame in a movie. Fortunately, I think this is still a long way off).
I am TheRaven on Soylent News
Okay - I'll bite
* Desktop-metaphor based GUI for a personal computer
* WYSIWYG publishing with a laser printer
* PDAs via Newton
* AppleLink (err, AOL now)
* QuickTime (movies, QTVR, 3D, etc)
We could go on and on. Give Apple props where due, huh?
And please consider modding the troll down...
Just a small info. The brain behind Spotlight is Dominic Giampaolo, the same guru that wrote the fantastic BeFS for BeOS.
uhm. No. It is not continually indexing the data, if you read the article you'll see it only updates the meta-data for items when they're saved - you can write custom plug-ins for new data types, or just go with the bundles ones for standard file types like images, text etc.
:
:
:
Filesystem metadata is great, but "instantly" updated search indexes sounds like a solution to a problem that doesn't really exist.
On the contrary, this is a *better* solution to a very basic problem that has plagued computers since they were invented.
The problem
How do I organise and access the data I use every day (emails, letters, images, music etc)?
The old solution
You can put your files in folders (one per file). You can name the files with a short description, ending with a cryptic 3 letter code to denote the file type. Files *must* be in one category/folder only at a time. Limited meta-data (date modified, file-type etc) may be stored.
The new solution
You add meta-data to files (often automatically) saying who created them, what project it's under, whether it's 'to do' or 'unfinished' or whatever. You'd do this in a save dialog for the application, as you saved the file. All other applications which use searchlight will update their view of this stuff for free, in real time.
When you want to work on a project, you click on the live project folder, and immediately you see all the files, emails, images etc for that project, no more, no less, regardless of where they are on the disk and what other projects they're shared with.
Want to see all the stuff to do with John, 5 months ago? On this project? Containing the word gizmo? That sort of query will be easy to make.
If you have an image editing application, it can show you all the images taken in Paris in 2002, without having to build a database application into it. This makes adding this kind of feature to applications trivial.
Ideally adding meta-data tags like 'project-1', and 'To do' should be as easy as choosing them in the save dialog or applying them like a label in the Finder. It's not quite at that stage yet, but that should come later. Some of these ideas are quite old (Be), but they are long overdue in a desktop operating system.
The Both links say quite a bit. I guess the kernel gurus know better, but i think the sql plugin for a FS would be a cool thing to show off with at the very least.
n .net/Articles/100148/
http://kerneltrap.org/node/view/3727
http://lw
The reason Windows XP does not do full text search correctly is because it uses a specific registry handler entry for each type of file (*.txt, *.rtf etc). It uses a different handler for different types of files.
; EN-US;Q309173
.TXT flat text handler is identified by using a registry key:
{ 5e941d80-bf96-11cd-b579-08002b30bfeb}"
.ASP place
{ 5e941d80-bf96-11cd-b579-08002b30bfeb}"
However it only comes with a few configured filetypes settings, and no way to set a default "When no searchFilter available, treat as plain text" setting.
I stressed and strained about this when XP came out initially. The only way I found to do it so I got expected results was to build myself a scanner.
It searched through a drive, and identifies EVERY file extension.
It then looks through the registry to see which Extensions have linked Handlers.
It generates a reg file containing stub links for every unmatched filetype.
Its a bit shotgun, but allowed me to continue using the Text search for XP.
Microsoft have released their own shotgun registry pack, for more info see here:
http://support.microsoft.com/default.aspx?scid=kb
(I have since moved myself into using my own full search tool, but at least the XP search doesn't miss files which are clearly within visible range).
[Now for the science part..]
Take a file, something like "PunchTheMonkey.asp".
Make sure you have it open in notepad, and make sure there is a certain text string - for instance "spyware".
Open the windows XP search in that folder, tell it to search *.ASP, and give it the phrase "spyware".
Windows XP will NOT find this file.
-----
The Windows
[HKEY_CLASSES_ROOT\.txt\PersistentHandler]
@="
Adding an entry like the one above for each required filetype will restore the full text search functionality.
So, I add the following entry into the correct
[HKEY_CLASSES_ROOT\.ASP\PersistentHandler]
@="
After I have logged off/rebooted, I try the same again, and XP will now identify the file.
liqbase
Clicking the 'X' doesnt actually close the application. This annoyed me to start with, but ive slowly gotton used to it.
o se/
If you want to quickly quit a load of apps or switch application, hit cmd-Tab, and then cycle through the apps with the tab key.
However you have one gig of RAM on the system. You have no need to quit the programs when switching between them. They'll be paged out to disk as necessary if you manage to fill the available RAM. Multi-tasking works very well as processes aren't in general allowed to hog the processor.
I think this is a common thing amongst people who're used to windows - the windows in OS X represent documents, not applications, so that's why they can be closed without quitting the application. You will find Apple managed to balls this up by being inconsistent though - some applications DO quit on closing the window, but in theory they're applications which only have one window, and are utilities, like the Address Book.
Be sure to try expose as well, though I doubt it'd work well on that older system.
http://www.apple.com/macosx/features/exp
Probably very similar to Search Kit which currently does the same thing, but has to be manually set up. You can choose the type of index it creates, inverted, vector or both together.
This has already been done to some extent in Quicksilver.
/.
http://quicksilver.blacktree.com/
It's an app that indexes parts of your file system and supports plugins to to index application data. The best part is that it is keyboard based. For example. type command-space "slash" enter and it fires off Safari opening
I'm not sure how Apple will improve on this.
The difference between Canada and the USA is that in Canada healthcare is a right and gun ownership is a privilege.
Beagle uses a new kernel feature called inotify which triggers an event any time a file is created or moved etc. Beagle watches for these events and reindexes files straight away, if needed.
In terms of the actual indexing, Beagle uses a C# version of Lucene which is designed for speedy searching.
Quicksilver is a pretty nifty program, and I used to use it as a free alternative to Launchbar (which used to perform roughly the same tasks). Both programs learn what you want certain shortcuts to mean, and both use Command-Space to activate them. For me, entering 'FX' is Firefox, 'PS' is photoshop, and so on.
However Launchbar has since updated to 4.0 beta release, and in doing so has pre-empted spotlight, as it does (right now, in 10.3) index system-wide metadata. So now you can cue up songs by entering MP3 names, open any kind of files by entering keywords for filename or type, open websites, perform google searches,Google image searches and so on.
It's worth trying out as an alternative to Quicksilver.
Having to select the application window before I can quit it using the application menu. Or I have to right click on the dock icon to quit. Annoying still.
OK, use Splat-Tab (Apple/Command/Cloverleaf, call it what you will) to switch between apps. When you get to the one you want, hold down Splat and press Q. It quits the application. Press H instead and it Hides it. There's more of these...
Hope this helps.. It seems this is OS X 10.3 only, so you might want to check out LiteSwitch X which does the same thing.
Mark
Liked this comment? Why not buy me something nice
Apple has had this type of search engine before, they called it V Twin and it was a basic part of Copland. This is what Sherlock used in Classic and why it was so fast. The idea is even older, it's from a conceptual computer interface Apple dubbed the Knowledge Navigator. All this appears to be is V Twin running on SQLite instead of a proprietary method.
The interesting part to me is the focus on metadata. I loved this feature in BFS that metadata was king. This is going to lead the way to better file management. Hopefully the Finder will integrate it.
Not since Marie-Antoinette played milkmaid has looking simple and honest been so fake and complicated.
This article provides a little bit more to the discussion:
http://www.appleinsider.com/article.php?id=733/
I think this is what will be most interesting, is that you can save your query for use anytime. All of a sudden it becomes easier to mange projects that are constantly being updated.
* Desktop-metaphor based GUI for a personal computer
Xerox invented that one.
* WYSIWYG publishing with a laser printer
Xerox invented that one too.
* PDAs via Newton
Invented by Psion in 1984 with the Psion 1.
* AppleLink (err, AOL now)
Applelink was built on AOL, not the other way around.
* QuickTime (movies, QTVR, 3D, etc)
Quicktime is a collection of other people's codecs with Apple extentions. QTVR was also invented outside of Apple.
Don't get me wrong, Apple does cool stuff but their strongsuit is marketing, not "invention".
Clicking the 'X' doesnt actually close the application.
That depends on the app. If it only has a single window, like the System Preferences app, it will typically terminate when you close that window. If it's a document-style app like a spreadsheet or a word processor, then it generally won't quit when you close the last window. Many apps have a user preference setting for whether to terminate when the last window is closed.
Every time I boot the Mac, my TFT display is 'wavey' until i have the monitor do an autoadjust
What kind of display are you talking about here? Is it one of Apple's very old VGA-input LCDs?
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
As someone replied earlier, this is a new paradigm in app management: the top menu controls the application, and the window menu controls the window. More importantly, OSX apps are designed to be left open -- keep them open, close or hide their windows, and they'll use virtually no resources, but will start significantly faster the next time you use them.
Having to select the application window before I can quit it using the application menu. Or I have to right click on the dock icon to quit. Annoying still.
Learn your keyboard shortcuts. Take the ten minutes to learn them, and you'll regain hours of your time. Cmd-Q is the shortcut for quit, for example. If you're used to Windows machines, you can switch the cmd key with the Windows key.
Love the dock. Its just ..... right.
Check out Quicksilver, from http://quicksilver.blacktree.com . Once you get used to it [and once it gets used to you], it's phenominally faster than the Dock.
The ability to access the underlying BSD OS easily. Love it.
iTerm, from http://iterm.sourceforge.net , is a great OSX terminal app.
Here's a list of favorite OSX apps I posted a while back. Most are free/OSS, and they're all some of the best apps for any platform.
http://unxmaal.com
it handles pdfs! Yippee!
-- I speak only for myself
Documents are indexed as files are saved. The performance hit is during document saving. There is no need for "background indexing".
Apps need to be made "Spotlight-aware" in order to invoke the Spotlight indexing on save.
Apple looked like it was abandoning, or at least deprecating the concept in OS X.
Well, Apple *did* deprecate the old file type and creator tags, and resource forks in files that the Mac file system had always had since 1984. There were lot of problems with the metadata in the original MFS, not the least of which was that each file on the Mac was actually two files.
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
Learn your keyboard shortcuts. Take the ten minutes to learn them, and you'll regain hours of your time. Cmd-Q is the shortcut for quit, for example. If you're used to Windows machines, you can switch the cmd key with the Windows key.
Another useful keyboard shortcut for quitting apps:
When you are using Command-Tab to switch through the running apps, you can hit 'q' to quit the selected app, or 'h' to hide it.
This is useful for quickly quitting twenty or so apps that have been running for months and are no longer useful, while leaving the twenty useful ones still open..
I often have data on my machine that I don't want others looking at, financial info, a book I'm writing that is embarrassingly bad, etc. I just stick them in encrypted disk images. It also helps me keep work and personal info separate. Typing in a password once to access each directory is not too much of an inconvenience.
The Amiga's "DataTypes" system was around before Quicktime.
Matt. Want XML + Apache + Stylesheets? Get AxKit.
Avie Tevanian deprecated Metadata. The Technote that recommended removing all metadata, resource forks, type/creator etc (since removed after developer backlash) was written by none other than Tevanian. Obviously he carries a lot of weight, but hardly "Apple" did it. This was very much NeXT imposing its view of computing.
Meanwhile, which filesystem is better - one that can handle named forks or one that can't? I agree that they cause portability problems (and bundles are far more elegant), but the filesystem support is a positive thing (for more on the rationale behind a lot of this, see the "Grand Unified Model" on folklore.org).
I don't know what kind of crack I was on, but I suspect it was decaf.
Which explains why it's tied to the filesystem rather than using a general hook at the vnode layer to allow the same functionality to be implemented regardless of the filesystem in use.
Wow. Check it out. Everything you said here is completely 100% wrong.
Spotlight is filesystem-independent. It runs as a set of daemons and stores its metadata database in a hidden directory called ".Metadata" at the root level of the volume.
All your "could be" talk is basically a summary of how Spotlight works.
I write in my journal
I have a volume with nothing on it except 60 GB of AAC files. The metadata folder for that volume is 14 MB.
I write in my journal
"Didn't Apple hire the BeOS filesystem guys?" Yes.
From Dominic Giampaolo's homepage:
"My biggest claim to fame is writing the Be File System [...] At Apple I work in the file system and Spotlight groups. Lately though I've been spending most of my time on Spotlight (which is a lot of fun)."
Yes. System Preferences > Keyboard & Mouse > Keyboard Shortcuts. Check the box marked "Turn on full keyboard access".
This allows you to tab between gui elements. Ctrl-F2 activates the menus for keyboard access. And you can edit shortcuts for every application you have.
Veteran, Bermuda Triangle Expeditionary Force, 1992-1951
"As someone replied earlier, this is a new paradigm in app management: the top menu controls the application, and the window menu controls the window."
Actually, this behavior is not a new paradigm as it has been a feature of the Mac OS back before it was Mac OS -- all the way back to The Beginning.
There are a few reasons for this behavior, but the most important one is that in good UI design, each widget should serve a clear purpose. On a Mac, the "close window" widget closes windows and that's it (unless the app has only one possible window). Aside from making the app appear to "launch" faster, it's a cleaner UI implementation that leaves little room for ambiguity. Plus, lets say you are downloading a big file in your web browser but don't want the display or the Dock cluttered with windows. On the Mac, you can close all the windows but still not quit the browser and keep the download active and out of sight.
Per Square Mile, a blog about density
What's radical is that it does all the above, plus some. The way I rememver Jobs introducing it is something like this.
You have a program called iTunes that creates a database of your music so you can search for a song by any one of a number of tags, including genre, play time, title, author, etc plus any of the keywords the user adds and how they rated it.
You have another program called iPhoto that does the same for image files because iPhoto understands the internal tags in a jpg (or other image) file.
You have another program called Finder that indexes based on file data. It knows what size the mp3 is, but not how long the song is -which iTunes does know.
You have all this separate programs for dealing with different kinds of files because they all contain different kinds of metadata and internal tags.
Spotlight puts all these kinds of searches in one place, and allows you to combine them. So with the appropriate plug-in filter, it can search any file type and take advantage of any internal tags in the file to speed up the search. Its much faster and more accurate than searching based on the entire contents of the file.
So Spotlight combines metadata it generates itself (file content), with basic file metadata (file size, creation date...) and file type specific metadata (image dimensions or song duration).
Then, IIRC, you can save your search and the results will be updated in real time as files are added or deleted.
Eat Shit and Die, Fuck Bag!
In Mac OS X Panther, I hear fast-user switching is a boon for this type of illicit activity. Create a separate user, say, "batman", with no admin rights, turn-on file vault for that specific user to make sure everything gets encrypted. When the urge comes ... fast-user-switch, do thy bidding, fast user-switch-back ... lah lah lah.
Extraordinary Vacations. Exceptional Prices
You don't have to worry as long as you and your wife have separate accounts and you set your permissions right on your porn directory under your user directory.
Jesus was a compassionate social conservative who called individuals to sin no more.