Examining Mac OS X 10.4's Spotlight

Ton writes "Apple has published a discussion of Spotlight, the radical systemwide search technology that will be part of Mac OS X 10.4 'Tiger'. The really interesting part is that metadata will be playing a big role in Spotlight while just a few years ago people were afraid metadata in Mac OS X was going the way of the dodo."

Comments

Radical

[agent dero]

Can someone please explain a little more as to how Spotlight using metadata is a "radical" new thing?

I haven't seen any mainstream implementations (WinFS?) of it, but I didn't know it was a brand new concept.

Re:Radical

[dJOEK]

because for every result you get, you here Steve say 'BOOM!' ;-)

Apple is not known for doing things 'radically new', but more for
'Taking a good concept/idea that no-one managed to implement in a useful way, and then doing it right'

MP3 players weren't new when the iPod appeared.

Re:Radical

[Meredeth]

MetaData is not new. Its not radical. But MS aparently can't make it work. So Apple gets to use it first, 5 percent of the computer population go wow! 95 percent ask why can't we have this, and Longhorn SP1 will get it and proclaim it as a great new radical technology.

Re:Radical

[mabinogi]

Nowhere does anyone say that Spotlight using metadata is radical, or that metadata itself is radical.

The metadata part was noteworthy because MacOS has always had metadata, but Apple looked like it was abandoning, or at least deprecating the concept in OS X. The fact that Spotlight will use it shows that metadata on MacOS still has a future.

Re:Radical

[the quick brown fox]

Don't get me wrong, Apple does cool stuff but their strongsuit is marketing, not "invention".

You've got to give them credit for product design as well. Nobody makes more desirable-looking software and hardware. Is it any wonder that Apple's fiercest supporters are graphic designers?

Re:Radical

[BlowChunx]

QuickTime player may not have been ground breaking, but the entirety of the framework was.

Name one other multimedia framework that has been around as long as Quicktime. And don't mention Video for Windows...I'll take your response off the air.

Re:Radical

[reso]

wouldn't they have already brought this up by marketing the hell out of it? even if it's only to take some of the wind out of apple's tiger before it starts getting promoted heavily.

I hate to admit it, but from what i've read, i'm not going to have to fiddle with spotlight a whole lot. it's just going to work well, like most of apple's stuff.

Re:Radical

[jxyama]

>'Taking a good concept/idea that no-one managed to implement in a useful way, and then doing it right'

you can make a darn good argument that *that* is precisely the definition of 'radical.' if it wasn't implemented in a useful way and didn't see the light of the day, we wouldn't be talking about it as being 'radical' because we wouldn't be talking about it at all to begin with.

Re:Radical

[womby]

I will do you one better.

* Desktop-metaphor based GUI for a personal computer
Xerox invented that one.
Xerox did not invent a desktop-metaphore, they invented windows-icons-menu-pointer, look back at grandparent, apple took existing ideas and did them right.
* WYSIWYG publishing with a laser printer
Xerox invented that one too.
Xerox invented laser printing, Apple invented Publishing.
* PDAs via Newton
Invented by Psion in 1984 with the Psion 1.
Psion 1 was a digital diary, the Newton was a digital assistant.
* AppleLink (err, AOL now)
Applelink was built on AOL, not the other way around.
Correct.
* QuickTime (movies, QTVR, 3D, etc)
Quicktime is a collection of other people's codecs with Apple extensions. QTVR was also invented outside of Apple.
No, Quicktime is a video framework, a video container format AND a collection of codecs, iTunes, iMovie and Final Cut Pro all leverage Quicktime in ways totally unrelated to to the codecs and in ways impossible with any other video technology, and apple created this over 10 years ago.

The poster was listing technologies, not that apple had invented outright, but that apple had taken a base inspiration and created a market defining product.

Re:Radical

[dgatwood]

Nope. QuickTime predated DataTypes by a year. QT: 1991, DataTypes 1992.

What's so special about searching

[Anonymous Coward]

I've been using computers through my whole childhood, school, work, etc. I'we been doing all sorts, from playing games to hardware design and real time data analysis on the computer. And never, ever have I had the need to search for my files.

What is with this search thing that everybody is so hot about?

Not that it doesn't look cool though. Anything on a Mac looks cool ;)

Re:What's so special about searching

[thulsey]

Try to pretend that you're managing 2 or 3 or more major projects that can change or be passed along to someone else every few months with mails, im's, files, reports you don't look at, media submitted by other people in different countries, to-do lists and other project management data...

Now imagine someone asks you, the project manager (or just the last person still around) on a project from 3 years ago, what the initial proposal from that guy in japan who did the Flash files was versus what we paid him and what the VP's said about that....

People *will* have copies of these files still floating around *somewhere* in e-mail or im history, at least. You may not, I may not, but that's where this will come in handy.

A few years ago, hd space was not large enough to think that you'd keep all that data around, but gmail's new 1Gb e-mail storage just showcases the lack of a need to dump all that crap off your media if you can just organize it well, and who needs that when you can keyword search, anyway?

Re:What's so special about searching

[donert]

I suggest the value of search depends on what you use your computer for and maybe how good your memory is. To me it is tremendously important. Most of my job is spent managing information. Searching for things and adding things to my knowledge base. I love Google's Desktop search for this reason. I also use blinkx, but prefer the google UI implementation. Months or years after something was written, I need to go find it. I may not know if it was ppt, xls, doc, pdf, or xml. I may not have written. I may not have even read it, but I need to find it.

Increasingly I find that mult-media files matter. MP3 recordings of meetings, images of whiteboards, videos of presentatons are all fair game. (My hobbies include photography and genealogy. So findings pictures of people and places, correlated with GPS tracklogs is also of interest.)

I also find that, although I structure my file system according to something that makes sense initially, it won't be the way I want to search for it later. I usually file things accordning to client and project. But later I may need to 'find all system specifications where the DRP recovery time requirement was longer than 30 minutes'. This kind of search would require a lot of my time.

I need way better search, way better meta-data (which means system created because people don't), and more disk space ;-).

When I first saw the spotlight demo, I was thrilled. A very small step to help me with my job.

linux - OSX coexistence with spotlight

[e**(i pi)-1]

It will be interesting to know how this will work together with other OS, for example with linux or solaris. Some of the metadata look similar to what has with the file system status accessible by

stat file.jpg

in linux. Would be nice in linux to beef up on metadata too.

I hope that spotlight will work also, if you have a linux partition exported to the Mac via NFS. Will file information of NFS mounted systems also stored in the database?

Having linux and OS X working together is already now not without issues. If you have a file Test.jpg and test.jpg in your Linux partition and you copy both to the same place in OSX, the finder (on the mac) complains, because the two files are considered the same.

except

[Anonymous Coward]

it won't work properly in Longhorn and you have to wait three more OS revisions for Microsoft to get it right.

Sounds funny, but it's more likely to be true.

Mac OS Metadata Is Just One Kind Of Metadata

[DLWormwood]

The really interesting part is that metadata will be playing a big role in Spotlight while just a few years ago people were afraid metadata in Mac OS X was going the way of the dodo.

The kind of metadata that was almost deprecated by Apple isn't quite same thing as the "modern" concept of metadata. The classical HFS metadata covered concepts like file type, file creator, and "Finder bits" that aren't handled at the file system level in other OSes. This, combined, with the Mac OS's historical use of resource forks for storing developer defined data records, made perserving such data difficult or impossible in heterogenous environments like the Internet. It's really a shame; I've always thought this concept was the most elegant attempt to solve the problem of "rich data" associated with data files without requiring the data in the file itself to have some form of universal container format.

The metadata concept used by Spotlight is going to be based in part on a plug-in system that allows the Mac OS to reconstruct metadata information from the data within files themselves, rather than just using the metadata facilities provided by HFS and Mac OS resource forks. That means that each different kind of file, from Word documents to PDFs to Postscript jobs, needs its own special kind of processing to read its own format of storing such data. It's less elegant and more processor intensive that just using the historical HFS system, but it's more likely to to be useful for extracting metadata from files provided by Windows and other Unix variant users.

Re:Sounds like Windows, actually

[BasilBrush]

The difference? Spotlight works - it does find data in a large variety of files and emails, and my bet is that it won't doesn't eat up the huge amount of resources that you say Windows does.

Filesystem metadata is great, but "instantly" updated search indexes sounds like a solution to a problem that doesn't really exist.

Doesn't exist *for you* perhaps. Perhaps you don't have a lots of user data, or you have taken time to sort it into useful folders. I'd say it's about as useful as the incremental seach in iTunes is. Sure I could remember what artist did a track, and access a track by scrolling down to that artist, then finding the track. Or I could scroll down the list of thousands of track names, remembering my alphabet ordering, and locate the track that way. Assuming I've remembered the exact wording of track name. But I've always found it easier to type whatever word comes to mind first from artist or track into the search box.

And so it is with documents. Even if I do remember the file name and folder that a particular piece of information is stored in, I still need to navigate there. Most times it will be quicker just to type in whatever it is you remember about the data you want into a search box - even if you know where the data is stored.

Spotlight vs. Quicksilver/Launchbar

[JonBob]

The technologies are barely related; Apple is not ripping off QS/LB in the least here. Spotlight is a technology for searching through files based on their conent and metadata. QS/LB are utilities for finding files based on easily typed mnemonics. You are looking at one aspect of Spotlights appearance (the dropdown search pane in the corner) and assuming it's a ripoff based on some similarity to the appearance of the other utilities.

In fact, the Spotlight indexing technology will be a boon to the utilities, as they will be able to leverage this newly available metadata to execute even more powerful searches. Quicksilver is already invaluable to me, and I expect it to just get better.

Re:Im very interested...

[shotfeel]

Clicking the 'X' doesnt actually close the application. This annoyed me to start with, but ive slowly gotton used to it.

LOL. Quitting the app just because I closed a window is one of the things that annoys me the most about Windows. If I'm done working with one document in Word, I have to be sure to open up the next one before I close the first or I have to wait for Word to start up again.

Extrinsic vs intrinsic metadata

[gidds]

I think we're talking about different things here. What's being discouraged is the sort of metadata that's extrinsic -- separate from the file. On the Mac's file system, each file has a 4-character type code and a 4-character creator code, and also any amount of data in the resource fork, all of which is separate from the normal datastream. This is all at risk when moving files to or from different file systems or machines, and can be a pain to maintain and use. I think OS X is right not to need it (though it still handles it well).

However, from what I've seen, that's not the sort of thing Spotlight is about. The plugins we're talking about make use of intrinsic metadata - information extracted from the datastream itself. Many common file types include some descriptive information: EXIF data in pictures, MP3 tags in audio files, meta tags in HTML files, and so on. Spotlight is a way of extracting and using that data.

The practical differences include, OTTOMH:

• Spotlight's information won't be lost when files get stored on other file systems, sent over email, processed on other platforms, &c.
• Spotlight uses information that's already in the files - you won't have to set it up manually.
• You can use existing tools to see and edit the metadata - MP3 taggers, photo editors, whatever. And you can do so on any machine and OS.

This is probably one of those rare cases when that foul word 'leverage' might be appropriate -- Spotlight should allow you to make much better use of an existing resource. As such, it sounds like a jolly neat idea!

Search Ontology

[ElitistWhiner]

There's been reference from the beginning of the computer revolution to this solution we've all been waiting for... and credit to evolutionary steps taken by apps such as,Quicksilver, Launchbar, BeOS, etc... but one application that predates AND which most closely matches the feature set is:
Simson Garfinkle's "Sbook.app" from NeXT in the 90's.

The usefulness of Sbook.app ability to add tokens in a flat file for instantaneous searches enabled people to apply Sbook.app outside its realm of address book that it originally was designed.

Abstracting its functionality and interoperating at the kernel level is pure Apple polish on the brand. Until people start using "Spotlight", the verdict will be out on adoption across the platform.

I will venture it will be one of the defining characteristics of the Mac platform into the future.

Re:disk space

[PhunkySchtuff]

Well, let's just throw some figures up into the air. You've got a 200GB hard drive. The index is taking up 1GB. This is half of one percent of the drive space, a couple of dollars worth.
I'd say that 1GB is a lot larger than it will ever be, so it's not a concern for me at all
I'll happily spend a couple of dollars on drive space for instant searches on my local machine.
Kai

Practical examples of what spotlight does...

[MrMeCee]

...using smart folders.

If you have a mac with a ton of files, various "Previous System Folders" etc...follow along :)

I have smart folders for pdfs, avis, mpgs, and wmvs

I have these sorts of files *all over the place*...movie clips, test files, you name it.

I go to the finder, "open" the Windows Media Files folder, and they are all "there"

Or all the "archive" files (zip, rar, sit/sitx etc) i've collected and not erased in the last year...

or all of the emails i've received from japanese users...

it goes on and on.

To me, its like the whole star trek "Computer..find all of the blah blah blah for sector Whatever"

It concentrates on the "what you want" as opposed to the current paradigm of where did i pit it/what app did i use, etc