Slashdot Mirror
Examining Mac OS X 10.4's Spotlight
Ton writes "Apple has published a discussion of Spotlight, the radical systemwide search technology that will be part of Mac OS X 10.4 'Tiger'. The really interesting part is that metadata will be playing a big role in Spotlight while just a few years ago people were afraid metadata in Mac OS X was going the way of the dodo."
Anyone who has used the instantly updated searches in Mail.app or iTunes will have a feel for how useful a system-wide approach could be. However I too am concerned about resource usage. I think I'll wait and see how big the metadata index tends to get and how big the CPU/memory hit is.
I believe though that the indexing is done during saves, so you'll not notice a general system slow down. What you will notice is a slow down on file saves.
From reading the article, I think Hans Reiser has been right about the need for reiser4 on mainstream linux.
He saw all this stuff comming from way back. If you read the LKML, you will remember that he warned us.
Its a pity no one listens to him.
I read about beagle for linux it seems to be very similar in functionality. http://www.gnome.org/projects/beagle/
-- My site
I'm waiting for Tiger so that I can try out Automator. This promises to be a point-n-click version of scripting. Hopefully this will be easy enough to use even my parents and maybe even my boss will be able to use it.
The first thing I'll do is try making an Automator to create thumbnails. Currently I'm using a bash script I wrote on my Linux box to do this. This will be the first time I've paid for an OS upgrade since Win98, so I hope it's worth it.
Vote for global prefs bug
Spotlight's datastore isn't SQLite.
The DB for it was custom designed for fast unicode text searches. As far as i know Apple isn't going to document the DB format but will be providing a C based API to search it.
Does the world need another DB file format? We'll see....
Coming from a WindowsXP background, some things Ive noticed so far:
- Clicking the 'X' doesnt actually close the application. This annoyed me to start with, but ive slowly gotton used to it.
- Having to select the application window before I can quit it using the application menu. Or I have to right click on the dock icon to quit. Annoying still.
- Love the dock. Its just
..... right.
- Most of the file system is hidden from you, which I like. Put my data where I want it and ignore the rest.
- The ability to access the underlying BSD OS easily. Love it.
- Everything looks and feels 'polished'. THats what I always hated about KDE/Gnome when I tried them, the features were there, but noone had taken the time to step back and polish the entire thing off so it all looks and feels together.
- Every time I boot the Mac, my TFT display is 'wavey' until i have the monitor do an autoadjust. Dont really know whoes fault this is, tho its fine under windows and linux.
So, final conclusion? I love it, so much that I have already placed an order for a G5 Imac. And in the meantime, Ive purchased a G4 upgrade for this little baby, just to help it alongUnless you used BeOS in the past!
This really is a big deal, much bigger than Microsoft's feeble attempts at full text search, or Google's desktop search. In many way's this much, much more useful than full-text search, especially for developers.
At home I have about 6,000 MP3s, a 1000 photos, 500 scientific articles in PDF format and hundreds of words files that I need to juggle. Each one has its own metadata database, and none of them are updated in real time.
Databases:
MP3 - WinAmp & AudioTron
Photos - Photoshop
PDFs - Acrobat Indexer
Word files - MS Indexer
That doesn't include any of the other data that is stored completely databases and would have been easier to store in the file system - like email, guitar tab files and god knows what else.
A properly implemented global meta-data store (that works at the filesystem level, not as an iterative service) profoundly changes how one uses the system, making sorting and finding data actually almost pleasurable.
+--------------------- You idiot! I told you we were facing the wrong way!
Devon
Check out Mor Naaman at Stanford who is working on adding GPS metadata to photographs. Once he has the GPS coordinates he uses that to get information such at time of day, lighting, weather, elevation, temperature, etc... This allows you to create metadata searches for "All early morning images in clear weather in Las Vegas, etc..."
YOu can try the system out here with a collection of almost 4k images.
Metadata is gathered by a sort of 'plug-in' for each different file type.
Apple has had a few developer kitchens on writing Spotlight importers. The idea is that any given app developer might have his own ideas as to what constitutes the interesting searching criteria for his file types. Apple has importers for common image formats, plain text, rich text, mail messages, etc.
If you were a photographer, for example, and you have a fancy camera that puts a lot of info into the EXIF tags of the image files it generates, you could search for "all images I made using this particular lens with a f-stop setting between 2.5 and 3", or if you're looking through files from a music notation program, you could search for "all files in 5/8 time in the key of G minor".
-jcr
The only title of honor that a tyrant can grant is "Enemy of the State."
I am the poster and the link was included in the post. Actually, the whole post was about the specific link to the Apple Developer site. Why the editors removed that link is absolutely beyond me...
How well this system works will in part depend upon how many data format plug-ins are provided. For example, take something like the SID audio format. It's relatively unknown, but has an officially registered MIME type with IANA giving it a status above many other file format types, and it is used to provide background sounds on some web sites. Will it make the cut?
This is just one file format chosen at random. There are thousands out there, some of which are used pretty heavily for documentation in certain circles. How about all of the OpenOffice file formats, or the AbiWord format?
I can see this feature being hugely useful if Apple does a good job of providing plug-ins, and making it easy for third-parties to add more.
I'm not convinced yet apple is going to get Spotlight right, i.e. truely revolutionary. It has potential (smart Finder folders is on the right path) but at the moment, it seems they are more interested in simply trying to duplicate Quicksilver/Launchbar technology, which is the wrong way to do this.
I'm tired of apple ripping off ideas from developers without (A) Giving them credit or (B) developing something equivalent so the new as at least as feature-full as the old. Based on apple's history, the first version of Spotlight will likely be a horribly dumbed down version of Launchbar in terms of tech, since apple is obsessed with "ease of use": i.e. a three year old has to be able to work it.
Rant aside, there are a few key pieces I think apple is missing:
(1) User-created metadata. I should be able to tag anything I want with any metadata I want so the organization system follows ME and MY preferences, instead of the system determining it for me. Apple should be thinking about taking the insanely wonderful metadata system they created in iTunes and applying that to the finder. It is essential you be able to tag metadata in, because you don't always access the same objects for the same purposes.
(2) Flexible file system. This is a concept I've developed which basically says that the file system should be dynamic and adaptable to match the thought flow of the user (only possible with a good metadata file system). If you've ever seen this app on the PC, think: "The Brain". What that means is that if apple does #(2) right, it should be easy as hell to tag things, and then basically I can create relationships which let me "flow" through my files by navigating CONCEPTS instead of folder heirarchy. A good app that does this is Devonthink. Devonthink will grab the contents out of your files, and when you do a search, you can not only see your search term but "related" search terms. Click on a new search term and you get a new listing. So as you come up with ideas about what you want to do, you can easily and naturally branch off into other parts of your file system. This methodology models the way the human brain actually works- thinking in concepts and spacial organization, rather then structure. (The "flexible" comes because the system takes your tags and adapts the search around them, allowing you to change how the "flow" works, depending upon what topics are most important to you.)
(3) The next level after metadata search is a new way of visually interpreting the metadata and relationships between. Which means a NEW FINDER. I can't believe Steve actually threw this comment out after demoing Spotlight: "With this, you probably won't even need to use the finder any more." Well then why even have the Finder at all, Steve?! There IS a reason for the finder, which is why it's stayed around all these years, and that is that people think SPACIALLY. People are creatures of habit, and one way we remember where things are is if we know where to look for it and it's always in the same place. Which means there needs to be a visual grounding to the above dynamic files system, to give people a sure footing to all of this. I'm talking about things like a window that always stays in the same spot and always performs the same task, like showing you what new files have been added to the system, or actively updating your list of word documents wherever they are. Right now in the finder, a window is a window is a window. That shouldn't be. If a search is applied to a window, then that window isn't just showing you files, it's performing an active function. The finder needs to evolve to take on the new roles and responsiblities it should have in the context of a metadata files system. Spotlight should replace the finder: the two should work together seamlessly.
The good news is that Spotlight is built into the system, so even if apple screws up the implimentation (likely), the next generation of 3rd party apps will hopefully be able to fill in the gaps.
I'm very fuzzy on the details, but I know that Apple played a leadership role, back in the mid-90s, in lobbying the FCC for the radio spectrum allocations for what we now call WiFi.
Spotlight can support arbitrary file types, entirely dependant on what an application developer decides to supply, and you decide to install. Google is limited to the file types Google implements.
WinFS is an overly complicated pile of steaming pooh, that Microsoft are having trouble delivering.
AFAIK they use some kind of search on the lexicon for the inverted index. For instance, the string "nut" is matched to "nutmeg", "donut", etc., and the document lists for those terms are merged together. Phrase search would also be done using all matching words, eg "nut hol" would expand to phrase searches like "donut hole", "peanut holder", etc.
The exact method for matching the search string to the lexicon isn't clear. It could be a suffix tree, but it may be as simple as grep-like scanning of the words, since there aren't that many relative to the text size.
Looking at mail.app it seems to do this process on each keystroke. It's not terribly fast, but it gets the job done.
---- "If we have to go on with these damned quantum jumps, then I'm sorry that I ever got involved" - Erwin Schrodinger
Of course, this metadata will be so much cooler when something like spotlight is there to take advantage of it...
Die Menschen verhoehnen was sie nicht verstehen. -- Goethe.