Haystack: A More Compelling View Of Your Data

Peristaltic writes "MIT's Haystack project has released the source for it's "Universal Information Client", Haystack. In their words: 'Haystack looks into the use of artificial intelligence techniques for analyzing unstructured information and providing more accurate retrieval.' Unlike some attempts I've seen in the past to pull it all together on my desktop, Haystack shows some promise -- One of it's more useful features allows you to take the information you've been wallowing through, and have Haystack continually refine a 'dynamic hierarchy' until you get what you need. Haystack also performs some neat tricks such as combining Email, IM, web pages, etc. into a single inbox."

Comments

That's great and all, but..

[notque]

Haystack also performs some neat tricks such as combining Email, IM, web pages, etc. into a single inbox

It may just be me, but this is a feature I never want.

I do not want 1 large program to run all of my applications. I do not want to get my email, from where I get my web pages, and my IM. I don't want any of this.

I am quite happy with seperate programs which I can use at my pleasure. I'm happy with the lack of bulk, and the fact I can change an email client without changing a web choice. (although I only use pine anyway.)

Is this just me? Do all of you want your programs shoved together in one large application?

I didn't get any options on my cell phone (like text messaging) because I purchased a cell phone. I wanted a cell phone. To make calls. Nothing else.

Re:That's great and all, but..

[RevMike]

So should I assume you don't want it embedded within Emacs.

Re:That's great and all, but..

[Anonymous Coward]

Oh, man, no text messaging? Text messaging's even more useful (and cheaper) than making calls!

Re:That's great and all, but..

[Xerithane]

Is this just me? Do all of you want your programs shoved together in one large application?

You mean like a Window Manager? That's how I see this thing... it's like a Window Manager with applications embedded inside of it (think of a forced dock type thing.) It just handles whatever data you present it with (or the computer presents it with) automatically.

I didn't get any options on my cell phone (like text messaging) because I purchased a cell phone. I wanted a cell phone. To make calls. Nothing else.

My cell-phone has bluetooth, PDA functions, games, voice recording, voice dialing... that's the great thing about choice. You, nor I, are the entire market.

Re:That's great and all, but..

[Transient0]

Yeah.. I have to say I'm not sure how radically different it is compared to just creating yourself a local web-site with a bunch of links to useful files and programs on it, particularly if you include cgi links to shell scripts and what not.

Perhaps even more significantly... how does this really differ from a decent OS or Desktop-Env? I mean in GNOME I can have a Mozilla window open in a quarter of my screen, a terminal window running Mutt in another corner, OpenOffice in another part and nautilus file browser for the rest, etc. The only thing that it seems to include that isn't easily accomplished by an intelligent custom set-up of Linux (or Windows or Mac for that matter) is the arbitrary associations between files and you could do that with a few clever applications of symlinks.

What I'm saying basically is that this package looks like it does some neat things, but it takes up a gig of space and a ton of memory when you could hack together something that works just as well for your individual purposes with almost no memory or storage overhead in a couple of hours if you know a scripting language.

Re:That's great and all, but..

[ChrisDolan]

I agree that monolithic apps cause problems (I gave up on Mozilla mail for example), but the particular example of IM and email consolidation is appealing to me.

I like to document conversations, so I keep and file a lot of email. The IM discussions are often lost from the conceptual thread, unless I manually digest them into an email. If IM transcripts showed up with my email, it would be easier to keep a uniform record of workflow.

A more interesting solution to me would be that iChat or the like would email the IM transcript at the end of the conversation, instead of being integrated into the mail client.

Re:That's great and all, but..

[Anonymous Coward]

Your signature should be properly attributed to Mitch Hedberg.

Re:That's great and all, but..

[a8f11t18]

Well, I think that some things do work well together,
if done correctly. Like Opera and M2. Now, M2 is the
coolest mail client I've ever used, but it wouldn't
have been so if it wasn't integrated with opera.

Some interesting bit of text on a site? I mark it and
send it in seconds to whomever I want to in the contact
list thanks to the integrated M2. Have some notes stored
in Opera Notes I want to send someone? Well, I can
quickly and directly do this thanks to M2. Want to just
check my mail very very quickly? Well, a M2 window
or 2 or 3 opens up instantly like all other windows inside
Opera..

An external mail client would be less integrated, and take
more time to load / "get ready". As it is, M2 is fully
integrated, lightning fast, and very very cool.. and comes
in a small 200 KB dll.. no thanks to external mail clients
for me at least :)

Re:That's great and all, but..

[SWroclawski]

Actually- it's like an Emacs.

It takes different information inputs and presents them in one interface.

It doesn't appear to be as flexible though.

- Serge

Re:That's great and all, but..

[donkiemaster]

the problem with having everything separate is that relevant communications cannot get to you fast enough. you may forget to check your email, or your voicemail, or not log into aim. I think you don't want it because you haven't seen it work well, or to your benefit. oh yeah, is the girl that would be upset with you your mom?

Re:That's great and all, but..

[notque]

And that's exactly my problem, the flexability.

I can create, and manage my own set of windows, programs, etc for the functions I want to do.

That doesn't mean that others won't find this program useful, but I certainly wouldn't want to use it.

And even if the program was incredibly flexible, I still would rather do it myself. I don't need any more programs controlling all of my data in a centralized place. I do not own a tin foil hat, but I'd rather just manage it all on my own.

Re:That's great and all, but..

[ichimunki]

Doesn't iChat have a transcript function? Isn't Mac OS X a Unix that has the ability to run a cron daemon (with the ability to send an email as part of a cron job)? Of course, if iChat were Free Software, you could add a function to the Quit or Close Window subroutines that sent transcripts of any sessions in the chat log folder to a specified user.

Now what I'd like is a web proxy that saves all of my submitted form information, so that I have local copies of things like Slashdot posts without having to copy and paste them in or out of the browser (for example)... kind of like how most email clients save a copy of my sent mail.

Re:That's great and all, but..

[a8f11t18]

But I'd like to point out I wouldn't like ie. mozilla browser
+ mozilla mail inseperatible.. that doesn't work well because
mozilla mail is a bit like outlook express.. i would never want something like that integrated with opera, bad idea.. but.. for those who haven't tried m2.. part of the reason it works
so well integrated is that it organizes and display information
in a completely different way than mozilla mail / OE etc.. plus it's incredibly light in comparison, and is more like "mail windows" than "mail client". If that made any sense :D Blah.

Re:That's great and all, but..

[notque]

the problem with having everything separate is that relevant communications cannot get to you fast enough. you may forget to check your email, or your voicemail, or not log into aim.

I could only hope to forget to check my voicemail, or email. That blasted voicemail light sends out a beacon to all who come by to say, "Looks like someone has a voicemail." Grrr, as if I didn't know.

I think you don't want it because you haven't seen it work well, or to your benefit.

And that's true. But let's say that it's the bee's knees. All that I could hope for. I'm happy with what I have, and have no reason or want to change. It's not that I fear change, or anything like that, I am just quite content with my current solution. If I feel like changing it up, maybe I'll add a skin to something. That's as dramatic of a change as I need.

oh yeah, is the girl that would be upset with you your mom?

That was funnier the first 50 times I heard it. :)

Re:That's great and all, but..

[Anonymous Coward]

Yeah you are right. We should just quit researching new methods to present information or enhance our experiences. Everyone should just go back to pine. We should limit all progress to stay in synch with your idea of how things should be. Additionally, just like you, we should pass summary judgment on any new idea and dismiss before actually trying it or looking into the details of how it works or the possible benefits.

Also, since the earth is probably flat there is no need to try to sail across the world since we would probably just fall off the edge.

Re:That's great and all, but..

[notque]

Really? I have a horrible time trying to type out a message at any time. If there is a need for me to get someone a message while I am not at an internet connection, I can call. If they are not there, I can leave a message.

Text messaging is enabled on recieve on my phone, and people send me messages, and I have never seen a single one that was at all needed aside from, "Hey, call me in 5 minutes."

Most of them are, "R U Free 2nite?!"

Just call.

Re:That's great and all, but..

[notque]

Yeah you are right.

That's really the only part I need to quote to sate my ego for the day.

Re:That's great and all, but..

[notque]

Thank you, I will update that now.

Re:That's great and all, but..

[rdeadman]

I think what Haystack is trying to solve is the data management issue. For thirty years we have been living with application-centric computers. So much so that we think in terms of best-of-bread point-tools. Do we know where Mozilla stores our email folders? No, its hidden by the application. (Okay, I do, but that is because I'm a bit geeky and share my Mozilla email folders from a File Server across my intranet...) How about Outlook, Netscape, Eclipse, etc.

In my inbox I have folders for home, each client project I am working on, future leads, charitable organizations I am involved in. A similar parallel hierarchy is repeated in my file system for documents. My IM tools have their own way of tracking contacts that is unrelated to my email or projects. I store my Eclipse projects in yet another place. Mozilla organizes my bookmarks in yet another hierarchy. It's all a real mess and makes working on a project a job of mentally mapping all the pieces together.

Now, what would be real nice would be if Haystack could define a plugin API (a la Eclipse) so that my email client could be wrapped and plugged in to Haystack. Same for IM clients, web browsers, etc. The point tool then only has to worry about its job and hands off data persistence to haystack. Then I can choose the best app and let Haystack worry about tying the data together. As someone else mentioned, this sounds more like a replacement for the file system. But it could be more, if each plugin could define how it interacts with other plugins and defines its own responsibilities.

I'm sure there is a lot of refinement needed, but it is an interesting new paradigm. Activity-centred desktop insteaed of a tool-centred desktop.

Re:That's great and all, but..

[LostCluster]

I do not want 1 large program to run all of my applications. I do not want to get my email, from where I get my web pages, and my IM. I don't want any of this.

So I take it you're not running Windows, Internet Explorer, and MSN Messenger?

Well, even if you're running Linux, Mozilla, and AOL Instant Messenger, they're still running on the same physical hardware and using the same window manager software in order to keep the interface consistant and organized.

And that's the point of this project and several other next-gen file systems in development now... Presenting users with a unified and organized interface that shows them their data in a way they can find it easily. From a user perspective, it makes more sense to store information as "messages that came in from Bonnie" rather than have a seperate file storage device for e-mail, IMs, voicemails, etc.

You might think it's simpler to have a physical device manage each communications protocol you use, and I'm sure product manufacturers will continue to support you with products based on that concept. However, most users would rather have their computers keep the difference between protocols to itself.

It doesn't matter how the information gets to the computer as much as what the information is and which person or organization is credited as the author. That's the best way to present information to a user who doesn't care about tech stuff.

Re:That's great and all, but..

[sharekk]

The idea is not just to have your mail client, calendar, browser etc. combined, the idea is that it has some built in intelligence to understand how things relate. If you have a meeting with someone it could understand the relationship between the people invited to the meeting, messaging logs and emails so you could easily brush up on the topic, see the slides attached in an email etc. without having to figure out where everything is. For us absentminded people this is a really neat thing though I see haystack more as a prototype than the bright shiny apps I'm sure we'll see attempting this some time soon.

Re:That's great and all, but..

[GeekWade]

I am not sure I would use this particular feature, but I can see that it might be of use to support personnel who might get alerted to some type of incident via the methods listed. I know that in the past I have been tasked with aggregating data to be inserted into an incident tracking system, and coding for these things (yahoo, aim, icq, yada yada yada...) can be like hitting a moving target blindfolded...

Just my $.02

Re:That's great and all, but..

[Sigurd_Fafnersbane]

And that's the point of this project and several other next-gen file systems in development now... Presenting users with a unified and organized interface that shows them their data in a way they can find it easily.

Exactly what everybody wants. I have a unified and organized interface. I call it the filesystem. It has a root directory and under this root I have files and sub-directories. It is clear and intuitive and it gives great flexibility in how to store and organize your data.

Re:That's great and all, but..

[jorleif]

And even if the program was incredibly flexible, I still would rather do it myself. I don't need any more programs controlling all of my data in a centralized place.

I think you are overreacting a bit. This is a research project after all so it's hardly perfect. What about your current OS, doesn't it "control all your data in a centralized place"? How is this different except for being more convenient? Do you actually care if the programs operating on your data are different processes, or plugin threads? Do you actually care what the underlying data representation is as long as its fairly efficient and allows for convenient operations=

I personally think contextual right-click menus are very convenient because they group functionality not by application, but by task. E.g. When you manage files you might also want to rename one, so that can be put in a contextual menu. The way computers work today is not an expression of great freedom that projects like Haystack endanger, but the result of systems evolution. This means that some rough points aren't being fixed even though there underlying reasons have gone away with newer hardware ages ago. Haystack-like things just offer a different views, if they are any good they might survive, otherwise they will die without causing any harm.

Re:That's great and all, but..

[Isaac-Lew]

It's actually cheaper for me to text message (unlimited messages on my plan) than it is to send/receive calls before 9 pm (at least to non-customers of my particular service). Also, text messaging is less annoying in restaurants, movies, houses of worship, etc. than sending/receiving voice calls (IMO).

Re:That's great and all, but..

[Ewan]

/opt is intuitive now? :)
and /etc for configuration files?

The unix filesystem is very powerful and flexible, but it is hardly clear and intuitive.

Re:That's great and all, but..

[Captain Rotundo]

if /etc and /opt aren't 'intuitive' for you you could just type this at a prompt:

# ln -s /etc /configuration

then:

# ln -s /opt /optional_programs

hows that for intuitive?

Re:That's great and all, but..

[Sigurd_Fafnersbane]

I will actually claim that it is both clear and intuitive if you consider the scope.

/etc, /bin,..etc. are for the system admin to worry about and in his/my scope it is a fairly straight-forward way to organize things. For a system user, the home-dir is the main reference point. I have still to meet a person I cannot explain the concept of files and directories and the file-system tree structure in the matter of minutes.

Re:That's great and all, but..

[Ewan]

Intuitive means:

Knowing, or perceiving, by intuition; capable of knowing without deduction or reasoning.

So needing to explain the filesystem means it is not intuitive.

Re:That's great and all, but..

[mcrbids]

Hmmm... That's fine for you, but most things don't work that way. Let's take your logic to its extreme:

If I want a gun, I'll go to a gunshop...

Oh, so you wouldn't run to a "Sporting goods" store, where you can get your gun, some ammo, maybe a camo jacket, and some wax for your skis while you are there?

Integration can get crazy - you wouldn't want to buy a gallon of milk at the sporting goods store, but in some cases (such as the Walmart Megastores) even that can make sense.

I didn't get any options on my cell phone (like text messaging)

So, you carry a cellphone AND a pager? My cellphone is an addressbook, cellphone, pager, and text messager all in one. That fact does not stop me from using it (very effectively) as a cell phone.

Is this just me? Do all of you want your programs shoved together in one large application?

KDE comes with Kmail and Konqueror. They are integrated and use the same libraries. I *love* the combination.

Re:That's great and all, but..

[s88]

Thats just rediculous. Why is sorting information by user any better than by protocol? While i agree by protocol isnt usually the best way, what you need is a sort by X. If I am having a mutliple person email discussion on some technical topic I dont want to have to remember the 17 different people that are in the discussion, i want to sort by topic. Outlook lets me do this. If this can extend to include other protocols... cool. But don't tell me there is a new single way to arrange data. That is worthless.

Re:That's great and all, but..

[jorleif]

Ok, so can you find all emails and IMs from "Bonnie" without using the email-client or IM-client? Is it convenient? Could non-technical users use it?

Ye olde tree-based filesystem is great, but I still believe there are better ways to organize certain information. Relational databases are also pretty popular so they must solve something the filesystem does not.

Emails are usually not files, neither are contacts or calenders. They are in files in some specific format and not accessible to all operations that could benefit from them. If we are to build a better integrated system we either need to go back to all plain-text data in files in a hierarchical tree, or we need a use a more expressive yet standardized data model.

Re:That's great and all, but..

[Sigurd_Fafnersbane]

So needing to explain the filesystem means it is not intuitive.

True, but in that case I doubt you will be able to design anything on a computer that can be used without prior knowledge. That the little plastic thingie besides the larger plastic thingie in front of the glass plate with images on, can be used to manipulate a group of dots on the glass plate - and that you, if you move the group of dots over another group of dots that looks like a small picture and presses an area of the little plastic thingie can make new images appear and that this has any relevance to the user, is also not self-evident.

A computer is a tool and in general you need some kind of introduction to an effective tool in order to use it. If you have never seen a wheel before, I doubt you would guess the usefullness of a wheel-barrow or a bicycle or a pair of roller-skates without a helping hand from a by-stander. The process of introducing one-another to new concepts is what we call culture.

Re:That's great and all, but..

[Anonymous Coward]

then you better stay away from the remberance agent.

Re:That's great and all, but..

[Kaki Nix Sain]

Ok, so can you find all emails and IMs from "Bonnie" without using the email-client or IM-client? Is it convenient? Could non-technical users use it?

Sure I can find 'em.

prompttext user$ ls mail/*bonnie* .Gabber/*bonnie* .Gabber/*/*bonnie*

Pretty convenient. If the gabber logs aren't encrypted you can even grep that lot for whatever you are looking for. But you've got me on the non-technical users bit. They will need a widget to do it for them.

Re:That's great and all, but..

[greenrd]

Why is sorting information by user any better than by protocol? While i agree by protocol isnt usually the best way, what you need is a sort by X.

You missed the point. That was just an example. Sort by X is the ultimate goal here.

Re:That's great and all, but..

[xigxag]

Perhaps even more significantly... how does this really differ from a decent OS or Desktop-Env? I mean in GNOME I can have a Mozilla window open in a quarter of my screen, a terminal window running Mutt in another corner, OpenOffice in another part and nautilus file browser for the rest, etc.

That's an interesting point. The need or desire for a product such as Haystack, with all your information visible at once, really demonstrates something we all know intuitively but rarely say out loud:

The paradigm of the overlapping windowing system has been a failure.

We are forced to resort to tabs, to multiple desktops, to transparent windows, to menu bars...all kludges which exist mainly because overlapping windows tend to obscure more of our information than they reveal. We're constantly forced to move, resize, minimize and maximize. Particularly if you are stuck with MS Windows...the OS out of the box comes with no way to tell a window: Don't let other windows overlap you. Or to say to a programs: Always start your window in that one corner of my screen. Or to tell two windows: Lock together on your edge so I can use you as a unit.

Re:That's great and all, but..

[CaptnMArk]

>The paradigm of the overlapping windowing system has been a failure.

Probably true.

But MDI, tabs, multiple desktops, transparent menus and menu bars only make things worse.

Re:That's great and all, but..

[donkiemaster]

I am of the opinion that until I know everything I need to know the instant I need to know it then communication and information retrieval can improve. But as you say, you are happy with your rate of information absorption, so I guess ignorance is bliss. And oh, by the way, I am sorry that I didn't read all the other responses to your signature, I really should have known that many other people had found it just as dorky as I did and made fun of you for it. My mistake.

Re:That's great and all, but..

[Anonymous Coward]

well, the reason we're all using linux is because we think that the "unix way of doing things" is superior. Concepts like files, processes, pipes, and utilities have proven more stable, versatile, robust, secure, yada yada, than other programming models including threading, meta-databases, registries, etc.

Re:That's great and all, but..

[Anonymous Coward]

The answer to data management issues is to standardize the format. And there's only one standard that everyone agrees on: plain character text. The same problem existed in 1969, and Ken Thompson realized it when he designed his operating system. It's the reason Unix has succeeded time and again where other, better funded and marketed ideas have come and gone.

There are many answers to cataloging, searching, sorting, and retrieving data, but as long as it's all plain text, it doesn't matter. A few good examples off the top of my head are the slocate utility, vFolders in Ximian evolution, and LDAP.

Runtime overhead

[PureFiction]

Beware the load on your system if you wish to try this out. It eats RAM and CPU with gleeful abandon.

From the system requirements:

• 
  - Pentium III 700mhz-based computer or better (Pentium 4 2ghz strongly recommended)
  - 12 megabytes of RAM (768 megabytes strongly recommended)
  

s/strongly recommended/REQUIRED/

Re:Runtime overhead

[3.5 stripes]

Big difference between 12 and 768, damn.

Re:Runtime overhead

[PureFiction]

Arg, cut-n-paste errors. Should read 512M

Please take note of the following system requirements for Haystack:

* Pentium III 700mhz-based computer or better (Pentium 4 2ghz strongly recommended)
* 512 megabytes of RAM (768 megabytes strongly recommended)
* Windows 2000, Windows XP, or Linux (Linux build requires GTK+ 2.0 libraries)
* At least 1 gigabyte of disk space (or more, as your repository grows)
* Java 2 Development Kit (JDK) 1.4 or later note that JDK 1.4.1 does not work with Haystack; use JDK 1.4.1_02 instead)

Re:Runtime overhead

[3.5 stripes]

Ahhhhh. That makes a lot more sense :)

Re:Runtime overhead

[Anonymous Coward]

* Pentium III 700mhz-based computer or better (Pentium 4 2ghz strongly recommended)
* 512 megabytes of RAM (768 megabytes strongly recommended)
* Java 2 Development Kit (JDK) 1.4 or later note that JDK 1.4.1 does not work with Haystack; use JDK 1.4.1_02 instead)


I think we've found your problem son!

Re:Runtime overhead

[meshko]

Isn't 768 memory requirement a little bit outrageous? Can anyone really expect people to have more than 512 in a normal PC? I have never had a problem with 256 even running Visual Studio and 10 tabs in Mozilla.
Not to mention my FreeBSD box -- it's pretty happy with 128 (yeah, that's because I don't use Gnome/KDE in case you are wondering)
Well, any new computer I'd buy today would have 512 of course, but 768???

Re:Runtime overhead

[Anonymous Coward]

Java.
'Nuff said.

Re:Runtime overhead

[Unkle]

This is rather new software, and is coming out of an educational source - if a company were to put this out, it would most likely not even be alpha software. I'm sure this is not ready for any type of wide-spread, regular user deployment, and thus does not have anywhere near the optimization that is present in Visual Studio or Mozilla.

Re:Runtime overhead

[LostCluster]

- Pentium III 700mhz-based computer or better (Pentium 4 2ghz strongly recommended)
- 12 megabytes of RAM (768 megabytes strongly recommended)

Thing is, the average $500 eMachines being sold today meets those requirements.

Re:Runtime overhead

[Anonymous Coward]

This is worse than playing Battlefield 1942. Imagine trying to play them at the same time!!

Re:Runtime overhead

[MrWa]

- 12 megabytes of RAM (768 megabytes strongly recommended)

It can run with only 12MB by they recommend, strongly, that you have 768!?! Does that not seem like a very large range to everyone else?

Re:Runtime overhead

[interiot]

Some geek use doesn't even require 512mb, but some do. Geeks who play around with Photoshop semi-seriously like to work with images that are several times larger than what the final copy will be, and if you have more than a couple layers on a 3000x2000 image, you're going to need more than 512mb.

Re:Runtime overhead

[thom2000]

This is worse than playing Battlefield 1942. Imagine trying to play them at the same time!!

Haystack isn't a game, stupid!

Re:Runtime overhead

[Anonymous Coward]

I realize im under the requirements but it took 10 minutes to start on my XP 1600+ under XP with 256 megs of ram damn! Kewl but man even windows doesnt take that long to start!

Awesome Mozilla effect.

[GraZZ]

Wow. Looking at the Haystack site with Mozilla looks awesome! I don't know if it's my version (1.4rc1) or some weird image setting, but the main image on the page stays stationary as I scroll around, but the clipping of the image changes. It's really hard to describe, but looks awesome.

Of course, IE just renders it properly. BOOOORING.

Re:Awesome Mozilla effect.

[ergonal]

Happens in Opera 7.02 too.

Re:Awesome Mozilla effect.

[tuffy]

This is the nifty bit of code that generates that effect:

<div style="BACKGROUND-ATTACHMENT: fixed; BACKGROUND-IMAGE: url(images/cover.png); WIDTH: 520px; height:370px; BACKGROUND-REPEAT: no-repeat"></div>

Fun with Cascading Style Sheets :) It might've been more effective, however, to stick the big image in an iframe so people can scroll around in it easier and have a look.

Re:Awesome Mozilla effect.

[Xerithane]

Wow. Looking at the Haystack site with Mozilla looks awesome! I don't know if it's my version (1.4rc1) or some weird image setting, but the main image on the page stays stationary as I scroll around, but the clipping of the image changes. It's really hard to describe, but looks awesome.

That just annoyed me. It's just a div layer clipping hack, part of the CSS2 spec. I hate myself for knowing that (I am not a web-*) but I do...

IE can't handle alpha layers in PNGs yet, I'm not holding my breath for decent CSS2 support.

Re:Awesome Mozilla effect.

[arvindn]

No, mozilla renders it properly. The relevant code is this:

<div style="background-attachment: fixed; background-image: url(http://haystack.lcs.mit.edu/images/cover.png); width: 520px; height: 370px; background-repeat: no-repeat;"></div>

So it is supposed to be stationary. Also notice that you don't see the whole image in IE.

Whoever designed the page must be really geeky if they don't care about it working correctly in MSIE :-)

Re:Awesome Mozilla effect.

[SkArcher]

and Opera 6

Re:Awesome Mozilla effect.

[GraZZ]

I agree; the only way to see the left side of the image is to resize your browser narrower...

I'm sure this isn't what the site's creator intended, as it makes it hard to look around such a pretty interface. :)

Re:Awesome Mozilla effect.

[OrangeGoo]

Actually, it's IE that has it wrong, not Mozilla. IE has yet to do CSS properly (funny that they can take the time to invent their own CSS, but can't be bothered to implement the standardized stuff). IE also doesn't support the alpha channel on PNGs, which makes them all but useless from a web-design standpoint. Since IE dominates, we have to design to them... hooray... Nuts to IE.

Re:Awesome Mozilla effect.

[a8f11t18]

*shakes fist* I wanted to say in opera 7.11 as
well but with that darn slashdotting and everything
I can't.. ;)

Re:Awesome Mozilla effect.

[Anonymous Coward]

FWIW: Konqueror (3.0.3) renders it correctly too.

Re:Awesome Mozilla effect.

[Jugalator]

I'm personally getting the Awesome Slashdot Effect right now. :-P

Re:Awesome Mozilla effect.

[fignuts]

*changes screen resolution*

WHOA! THANKS FOR THE TIP!

wait... now i wont get any work done today...

damn you mozilla and your proper CSS!

*regrets not just using IE*

Re:Awesome Mozilla effect.

[_xeno_]

Heh, I've got another neat effect going on with that image. The screen shot uses ClearType (I'm assuming), so it's optimized to be displayed on an LCD screen based on subpixel rendering. But that's OK, I've got an LCD screen!

Except that the screen I'm looking at has an BGR striping order (while most screens have an RGB striping order) - making the image look funky.

If you're on an LCD, compare these two images of ClearType in the two striping orders (courtesy - or stolen from, or whatever, Microsoft):

• RGB
• BGR

One of them will look wrong, and you should be able to see what I'm talking about.

But whatever :)

Re:Awesome Mozilla effect.

[arlow]

Yeah -- everybody at MIT pretty much has to use Mozilla because all the other browsers don't get certificates quite right, and you need certificates for a lot of MIT stuff.

also, this guy was my recitation teacher for 6.001 -- a few months back I checked out his research and thought to myself, "hmm, this seems cool..." and then it's on slashdot. maybe I should try insider trading. martha, give me a hand...? :/

Re:Awesome Mozilla effect.

[noda132]

After extensively developing and testing in both Mozilla and IE, I think I can generalize this phenomenon:

If IE and Mozilla render a page differently, Mozilla is always right and IE is always wrong. Always.

Re:Awesome Mozilla effect.

[Anonymous Coward]

> (Score:4, Offtopic)

Whoa dude, you r000l!

efficient?

[DougMackensie]

like finding that pesky needle?

THe real test.

[Unknown Poltroon]

Can it organize 3 gigs of random pr0n?

Re:THe real test.

[frodo from middle ea]

No , it uses "artificial intelligence" to sort "information".

Re:THe real test.

[Anonymous Coward]

Probably not, however, I'm sure I could help out.

I'll email you my FTP address, you upload your 3 gigs, and I'll do the rest.

yeah, I promise.

Re:THe real test.

[Rick.C]

Can it organize 3 gigs of random pr0n?

Yes, but that will require some optional hardware: eye-tracking camera and moisture-sensing drool-cup attachment.

Re:THe real test.

[beef3k]

Write a Haystack plugin that performs image analysis on the images/selected movie frames and categorizes them, and you could probably make a fortune with the pr0n site people.

That would mean they could spend their entire days drinking coctails at the beach, instead of wasting their current 1 hour day of labour.

Re:THe real test.

[Anonymous Coward]

You only have 3?? Get to work man! You've got a ways to go to get to 30 gigs compressed. ;)

Re:THe real test.

[Anonymous Coward]

Oh God I hope it's only used for drool.

Re:THe real test.

[Lord_Dweomer]

" Can it organize 3 gigs of random pr0n?"

You know, I'm probably going to be laughed at openly for this.....but deep down you all know you want something that could do that. How sick are you of searching around TGP sites when you know EXACTLY the types of girls you like. What I want is software that can recognize that I like brunettes......and based on the um...length of time I have various images open.....or based on a thumbs up or thumbs down learning system it could learn which facial features and body types I prefer. Then it could sort pr0n for me either on my harddrive, or add this to a massive online pr0n database and BAM, you have porn that is not only filtered by type (which ANYBODY can do now...) but filtered based on your personal views as to what is attractive. Saves you a lot of time, and um......energy.

Haystack?

[m0rningstar]

I see Haystack and I think of one of the old, old IDS systems/companys. Stalker, right?

On the other hand, it's a damn fine name for both types of product, and I haven't seen anything on the Haystack IDS in a long while...

Whoop de doo

[onthefenceman]

As much as I hate to say it, doesn't this look exactly like the "Outlook Today" page from MS Outlook?

Or maybe that's just because the A/B comparison is too easy because they're both open on my PC at work now...

Re:Whoop de doo

[DigitalCH]

I so agree. Although adding in IM is a little more sophiticated but several of the digital dashboard products could do it.
Bottom line, this is nothing new.

nefarious purposes?

[Anonymous Coward]

I'm worried that this could be used for data mining/viewing by the federal gov't. Having this software know all about you is a bad idea.

Re:nefarious purposes?

[OrangeGoo]

The government really isn't as interested in what you're doing as some people might lead you to believe. There is some constant monitoring going on behind the scenes, no doubt, but there's not anyone sitting around all day reading your email - except you. "Big Brother" doesn't care one iota about your email, unless you're planning some kind of terrorist activity, in which case - at least from where I sit - I want them to read your email. They may also snatch your email if you steal a few million dollars from your investors, but I'd be happier if they snatched your bank account and shared the wealth. ;)

WHAT???

[Justin205]

What does this mean? I'm confuzzled... Hmmm... E-mail, webpages and IM into one inbox... Wait a second, IM in an INBOX? IM is INSTANT MESSAGING. How will this work?

Re:WHAT???

[gmuslera]

So you IM client don't have history?

I just want a relational filesystem...

[Xerithane]

Not too much to ask, it doesn't even need to be truly a filesystem. Just overload all the file access commands (At this point, probably easier to just write a new filesystem)...

Group data by category, content, whatever. "Symlink" to the inodes, and you're off. We don't need AI for that and I think it would be a more complete solution. I don't see an AI engine that can correctly categorize my mp3's, I don't think I'd trust it for all of my data yet. Let's start small and get usable systems.

Spiffy program though, wish it weren't in Java... wish it weren't 42MB... wish it ran smoothly under Linux. I'll stop complaining now.

On a side note, Did anybody else find that scrolling image annoying and mentally confusing. Er, I'll really stop complaining now.

Re:I just want a relational filesystem...

[stratjakt]

Good news! You're getting one, the successor to Windows XP will sport WinFS, a brand spanking new filesystem based on SQL Server's engine.

Now, of course linux trolls will whine and whatnot, but SQL Server is a killer DMBS, and this filesystem will be cool. Imagine how fast apps will start when they dont have to scour a half dozen directories for .dll files, but instead "SELECT location FROM files WHERE filename = 'msvcrt.dll' AND version = '7.8.29'"

Anyways, in a few decades someone will write a free-as-in-no-money version for lunix. So hold tight.

Re:I just want a relational filesystem...

[truthsearch]

I've been thinking about something like this too. It would be great to have a system automatically attach metadata to any file. There could be a database that catches calls to create files and asks the user to enter metadata if desired, but that may be too annoying. Having some form of AI such as in Haystack combined with a good view of categories and other metadata would be very useful in organizing data. Ideally there would be very little effort by the user to perform the organizing. The AI would do almost all the work and user could just browse.

Time to download the code and get the wheels turning...

Re:I just want a relational filesystem...

[Xerithane]

Now, of course linux trolls will whine and whatnot, but SQL Server is a killer DMBS, and this filesystem will be cool. Imagine how fast apps will start when they dont have to scour a half dozen directories for .dll files, but instead "SELECT location FROM files WHERE filename = 'msvcrt.dll' AND version = '7.8.29'"

A file system built on an SQL engine doesn't work... It's like putting a Viper engine in a Ford Focus. A simple meta-dbm attached to each node (and visa versa, an index on the meta-dbms... similar to how the iPod works) would work just fine. I've never saw the point of allowing an entire SQL engine on a filesystem.

Anyways, in a few decades someone will write a free-as-in-no-money version for lunix. So hold tight.

That was really damn funny, thank you.

Re:I just want a relational filesystem...

[73939133]

Good news! You're getting one, the successor to Windows XP will sport WinFS

Yes, that is great news--the attempt will break so many things that it will seriously hurt Microsoft.

Anyways, in a few decades someone will write a free-as-in-no-money version for lunix. So hold tight.

DBMS-based file systems have been around for decades; there are good reasons why people aren't using them.

Linux has several file systems using database technologies (as well as change notification). However, what Linux doesn't have is a file system that lets you perform arbitrary relational operations. That's because such a "file system" would simply not conform to the interface and semantics expected of a file system, and lots of things would break.

But, of course, if you are Microsoft, you don't have to worry about standards, you just merrily break things and redefine APIs whenever you please.

Re:I just want a relational filesystem...

[Xerithane]

Having some form of AI such as in Haystack combined with a good view of categories and other metadata would be very useful in organizing data. Ideally there would be very little effort by the user to perform the organizing. The AI would do almost all the work and user could just browse.

I don't even think you need AI. Just fuzzy logic. We already know (for the most part) the filetypes, and all standard files will have a standard filetype identifier. All you need to do is find out what type of filetype you are working with and call a parser to try to collect key words (then score against a keyword matrix, like a vector search engine or something) and store the result.

Google algorithms would work better for data categorization than AI would. I've just never been a big AI-is-the-goal type of coder though. AI is a tool, it's great and all, but not necessary for a lot of tasks.

Re:I just want a relational filesystem...

[Anonymous Coward]

Well, DBMS-based filesystems are often SQL-like I.e. they're "statically typed" and rigid. What I'd like is basically a persistent "dynamically typed" relational database.

Re:I just want a relational filesystem...

[krb]

You say "Group data by category, content, whatever" and then say "we don't need AI for that". Well, you're almost right, but you need some intelligence in order to make decisions about what the content of file X really is. You could say, "well, yeah, that's me..." but the point of this and other Knowledge Management systems is that it takes the responsbility of categorization off of the user, because we are often inconsistent, or, at least, incomplete. Let's say I have a document that pertains to two or more general topics, lets say, Pollution, Energy Use and Windmills. Let's also say that right now i'm using it for a school report on alternative energy, so i classify it, quite sensibly for now, by year, course number, and assignment. That's totally useless in a few years when i'm looking for the information. I *could* have been smarter and manually attached some meta data to the file describing the kinds of topics it relates to, but i may miss one, and plus, that's extra work for me. Projects like this use complicated statistical (usually) analysis to determine the content for you automatically, and maintain a persistent database of all files realted to particular topics/content items, etc. Haystack and many others do this categorization with an ontologie which predefines the topic groups or elements they care about. Some systems derive the content groups dynamically, and include fuzzy searching to allow you to find documents and files related to some keywords (or if they're real good, natural language query) you enter.

What you mentioned is not that different from what they're doing, except they're not making it transparent -- they're making into a workspace.

I'll note also that categorization of text into topics or genres, while difficult, is easier than doing the same with music. The kinds of statistical analysis you can do on text doesn't lend itself to fourier decompisitions. To properly categorize music (in my opinion at least, which admittedly counts for little) the best technique would be to separate and identify the individual instruments (voices) in the song. This makes categorization a bit easier because now you can get data for tempo, rhythm, sohpistication of note progression, etc. on a per instrument basis. I'm not sure it's possible tho.

My 57 yen.

Re:I just want a relational filesystem...

[Glock27]

Imagine how fast apps will start when they dont have to scour a half dozen directories for .dll files, but instead "SELECT location FROM files WHERE filename = 'msvcrt.dll' AND version = '7.8.29'"

Which will of course be slower than "\\\\c\\windows\\system32\\msvcrt-7.8.29.dll". Enjoy the pig-like characteristics of your new "filesystem". ;-)

I'm pretty sure Apple or the Unix community will do a better job, long before Microsoft arrives at version 3 that actually works.

OK, that was my anti-MS rant for the day. Now I feel better! ;-)

Honestly, I just have no respect for Microsoft "engineering". Just about anything would be better than the monstrosity that is Windows. Seriously.

Microsoft's business practices are just the icing on the cake. :-P

Re:I just want a relational filesystem...

[jorleif]

The problem with the term AI is that it usually when something "AI" starts to work well it's being called something else. Remember that compilers and information retrieval (google) used to be AI once. This begs the question: What did you mean when you said fuzzy logic, but not AI?

Is AI as in symbolic AI (search-based etc.)? It seems to me that all sensible information categorization systems, even those built on top of fuzzy logic could reasonably be called AI. Google uses very sophisticated data-categorization algorithms, or at least it seems so based on my search results. Those are probably based on statistical classifiers and other such AI techniques.

Re:I just want a relational filesystem...

[jorleif]

Very insightful post I must admit.

To properly categorize music (in my opinion at least, which admittedly counts for little) the best technique would be to separate and identify the individual instruments (voices) in the song.

This is probably the "correct" way to categorize music, but since even humans have a hard time categorizing music I believe an automatic analyzer would have a hard time generating good results. Have you noticed that every band is X-Y with a touch of Z and W, where {X,Y,Z,W} are music styles? Then when you actually hear the music it's nowhere near what you expected based on the description.

A more useful (though boring) way to categorize music would be by doing statistical inference based on which songs the user listens to a lot. Then compare that with what people with similar preferences listen to (by some non-orwellian means) using for instance clustering or similar methods.

Re:I just want a relational filesystem...

[Xerithane]

Is AI as in symbolic AI (search-based etc.)? It seems to me that all sensible information categorization systems, even those built on top of fuzzy logic could reasonably be called AI. Google uses very sophisticated data-categorization algorithms, or at least it seems so based on my search results. Those are probably based on statistical classifiers and other such AI techniques.

I'm using Fuzzy Logic as just a way of branching true-false trees. Not so much a full-blown AI system, just (as you said) statistical qualifiers.

I don't view AI as "AI" -- it's mostly types of AI. To me, AI is something that is entirely abstract enough to handle tasks (Think self-configuring Universal Turing Machine) -- otherwise it's just statistical programs over very broad data sets.

I've done some work with neural networks (indirectly) and that was the general consensus there, as well... the mindset just stuck.

Re:I just want a relational filesystem...

[jorleif]

I don't view AI as "AI" -- it's mostly types of AI. To me, AI is something that is entirely abstract enough to handle tasks (Think self-configuring Universal Turing Machine) -- otherwise it's just statistical programs over very broad data sets.

I might misunderstand you, but does this not mean that the "AI"-methods used by Haystack would not be AI? I also have some Neural Network / Statistical processing background so I tend to share this same view that the intelligence is in the designer not in the program. However, the term AI is often used to describe both neural and statistical methods. Actually neural networks could also be perceived as statistical methods. I don't know about fuzzy logic but my intuition tends hint that they arent fundamentally very different either, just based on different mathematical theory.

To return to your relational filesystem idea, I would prefer having both relational search capabilities and google-like utilization of implicit information. I don't know about you, but I certainly am not very good at being consistent, so search and selection methods allowing some kind of fuzziness criteria would certainly be nice. The basic OS could still work with files, but my personal information should be organized in some more practical way, and should preferably be available all the time from diverse clients (home computer, work computer, PDA, cellphone...).

Re:I just want a relational filesystem...

[Xerithane]

I might misunderstand you, but does this not mean that the "AI"-methods used by Haystack would not be AI?

Using AI methods does not constitute AI. I haven't looked at their design papers so I can't say for sure. I have yet to see a real AI implementation, so I would say they use AI-Methods, but not AI.

I also have some Neural Network / Statistical processing background so I tend to share this same view that the intelligence is in the designer not in the program. However, the term AI is often used to describe both neural and statistical methods.

Most people view AI as some ambigious concept that they can't understand, so they just slap the label AI on anything that involves automated decision making... *shrug*

Actually neural networks could also be perceived as statistical methods. I don't know about fuzzy logic but my intuition tends hint that they arent fundamentally very different either, just based on different mathematical theory.

Fuzzy Logic is massively statistical, just a bunch of true-false statistics.

To return to your relational filesystem idea, I would prefer having both relational search capabilities and google-like utilization of implicit information. I don't know about you, but I certainly am not very good at being consistent, so search and selection methods allowing some kind of fuzziness criteria would certainly be nice. The basic OS could still work with files, but my personal information should be organized in some more practical way, and should preferably be available all the time from diverse clients (home computer, work computer, PDA, cellphone...).

I'm consistent by putting them in directories... that's about as far as it goes. I try to organize it like $DEVROOT/(Commercial|Contract|OpenSource)/$Categor y
That never really goes through... being able to
store everything and pull it up using different methods would kick ass. Sometimes I write a perl script for Project "Oops, I forgot" that would be great in my current project. Digging through all of my source takes ages, but having something to figure out A) It's a perl script (easy), B) roughly what it's purpose is (hard) would be awesome.

Re:I just want a relational filesystem...

[2TecTom]

Actually, why don't you just ask the artist to decide which genre a song belongs to. The category (ies) that songs (data) belong to are often most evident to the originator.

As well, Audioscrobbler and freedb do a reasonable job of collecting stats on the media itself and about the uses it is put to.

What I think was required was a way to attach custom attributes (meta data) to file system objects. Then sophisticated queries could be used in scripts. Maybe there's a file system or app that does this already? Imho, this is inevitable. Perhaps we just aren't there yet?

Re:I just want a relational filesystem...

[jorleif]

When I said "personal information" I meant email, web-pages, text files and documents rather than code. Sure it would be cool with tools for dealing with code as well, but I suspect you would need to use different methods than for the natural language material to be succesful.

What I would like to have would be something that could integrate all information related to some "project" into a single, powerful interface. Something that would allow me to have information in different formats (txt, html, pdf, ps, doc, spreadsheets, images and so on) searchable from the same locations. Also it would need to have a convenient way of "commenting" material written by others and linking these commments to the right place in the right document, without me having to place e.g. HTML links in the right locations. Then discussion material (mailing-lists, forums) should also be integrated in the same interface.

I guess my main point is that different media should not be artificially classified by its media type but rather by its contents (or more specifically which project it belongs to). My secondary point would then be that everything should be searchable, rather than found using a URL. More like the web and a relational database and less like a filesystem tree.

Re:I just want a relational filesystem...

[Xerithane]

What I would like to have would be something that could integrate all information related to some "project" into a single, powerful interface. Something that would allow me to have information in different formats (txt, html, pdf, ps, doc, spreadsheets, images and so on) searchable from the same locations. Also it would need to have a convenient way of "commenting" material written by others and linking these commments to the right place in the right document, without me having to place e.g. HTML links in the right locations. Then discussion material (mailing-lists, forums) should also be integrated in the same interface.

For me, this would be the ultimate window manager. When I'm on a computer, I'm generally working... or playing games but then I'm on Windows... so anyway. Having a truly integrated desktop would be awesome, but unfortunately too messy (especially with Linux) because there is no standard. If people were to actually come up with and adhere to a standard application model, something like this would be possible. Until that day, you'd have to re-invent so many wheels it wouldn't be feasible.

I guess my main point is that different media should not be artificially classified by its media type but rather by its contents (or more specifically which project it belongs to). My secondary point would then be that everything should be searchable, rather than found using a URL. More like the web and a relational database and less like a filesystem tree.
Exactly - and that's what I would like. Even if I have to at first "train it" what things look like.

In regards to plain text/personal information -- have you thought about looking at Bayesian filtering for a solution to that? I haven't (yet) but the idea is festering in my brain.

Re:I just want a relational filesystem...

[jorleif]

Actually, why don't you just ask the artist to decide which genre a song belongs to. The category (ies) that songs (data) belong to are often most evident to the originator.

Because different people have very different perceptions. Of course if the artist decides it's numetal-folk-country, then he certainly is "right" about that. But then bands playing similar music might call their music neoclassical funkcore just because they have a different background and the usefulness of the classification will be about as low as that of the current classifications. Therefore it's a lot more useful to know about similarity of bands (or songs) than of some "orthodox" classification.

Re:I just want a relational filesystem...

[Anonymous Coward]

Can you believe it? Something that you want will be created by Microsoft. Just do a search for Windows File System / WinFS.

Re:I just want a relational filesystem...

[jorleif]

In regards to plain text/personal information -- have you thought about looking at Bayesian filtering for a solution to that? I haven't (yet) but the idea is festering in my brain.

Bayesian filtering is another of those words that mean a lot of things ;)
Nowadays it is usually used in reference to the techniques used for spam-filtering, which is a very specific task. Classification: Spam / Not Spam. Basically everything that uses the bayesian view on statistics can be considered a bayesian method, without considering the underlying model. In other words for many statistical models it is possible to derive bayesian optimization schemes (or "learning rules").
A widely used set of language models are the Hidden Markov Models. I'm planning to use them on an information extraction problem (populating database tables from free-text descriptions), and that's about the closest to the problem we're discussing that I've been. You could probably use them as a partial solution here as well, but I can't think of any really clever scheme at the moment.

For personal information one would like to have something that clusters the data into different categories. There are lots of methods for this. One I'm familiar with is Self Organizing Maps (an example paper about them).

And finally, sorry to be boring, but I'm not currently working on anything that would create something like the system we've discussed =)

Re:I just want a relational filesystem...

[IntlHarvester]

What I would like to have would be something that could integrate all information related to some "project" into a single, powerful interface.

The product you are looking has existed for decades -- it's called Lotus Notes, and is widely used in large companies for document tracking and project management and so on.

However, few people understand the "powerful interface", and it requires putting all your data into a proprietary database with clunky code interfaces.

Re:I just want a relational filesystem...

[jorleif]

I'm not entirely surprised, usually when I get a good idea someone else has already implemented it before. The proprietary database is a bit of show-stopper though... I guess I have to check it out.

Re:I just want a relational filesystem...

[IntlHarvester]

I think you can download a free-trial at lotus.com.

The basic idea behind Notes is an "unstructured document database" as the "haystack". Each document has a "form" which describes an arbitrary set of "fields" in the document. You can then setup queries against the fields as "views". Documents can be hyperlinked together, full-text indexed, and so on. So, unlike RDBMS products, it provides a very flexible way to organize data.

I haven't used it for a few years, but the big downside was that that the coding was in a spreadsheet-like "formula language" that was annoying as hell. The other huge downside is that Notes is effectively an island -- it only works well if you are committed to put all of your interesting data into Notes. It would be a much better product if it integrated with your filesystem, other databases, MS Office, and so on.

One reason I'm looking forward to WinFS (etc), is that this is really 80s-level technology and should really be universal by now.

Re:I just want a relational filesystem...

[Xerithane]

Bayesian filtering is another of those words that mean a lot of things ;)

Isn't it just...

A widely used set of language models are the Hidden Markov Models. I'm planning to use them on an information extraction problem (populating database tables from free-text descriptions), and that's about the closest to the problem we're discussing that I've been. You could probably use them as a partial solution here as well, but I can't think of any really clever scheme at the moment.

Do you have a website or anything for this? It seems pretty cool, we should talk. If you need webspace or a server for it I can provide it.

And finally, sorry to be boring, but I'm not currently working on anything that would create something like the system we've discussed =)

I'm enjoying the conversation, I wish I had more time to dedicate to it but I am working on something... that isn't like what we're discussing at all, and is ultimately boring. As the saying goes, "Character is doing something that needs done with the interest in doing it fades away." At least we get to put in some cool magic in the backend.

Re:I just want a relational filesystem...

[krb]

why thank you ;)

i have about 60 gigs of (legal) MP3s so it's something i've put some thought into.

I agree with you that the best way to categorize music is probably not to start with an ontology of fixed genres but to try and just group music with similar stylistic elements with each other, and let the user infer what the genre is from there. I.e., i don't care what the grouping is called, as long as it helps identify the style. Still pretty hard i imagine...

-kerry
apogee404@hotmail.com

Re:I just want a relational filesystem...

[jorleif]

No website, no code just a problem and a few ideas...

I'm not sure it will work very well. In fact I doubt it will, information extraction works best for huge corpuses of text with some fairly simple regularity in their structure. It can still be natural language, but one might make the observation that A is usually before B, and bias the system for such a special case.
There was one project where they mined medical journal articles for which genes were found to affect which properties. I don't have a link to the paper right now but if you're interested I can look it up.

I might put up a website at some point. The project is both school work and commercial work so I'll get around doing it at some point. The interesting part is school work and the useful part (the application) is commercial.

Re:I just want a relational filesystem...

[Xerithane]

I'm not sure it will work very well. In fact I doubt it will, information extraction works best for huge corpuses of text with some fairly simple regularity in their structure. It can still be natural language, but one might make the observation that A is usually before B, and bias the system for such a special case.

Just for an algorithm idea, if you scan a huge glob of text (web-pages, etc) and form patterns based upon word relevance and relation, and then categorize the webpage (manually or through automated tools... screenscarping google, etc.) and have a final result with loose attachments to category. This is just for building the statistics engine, however. I was working on an idea for this using IRC text, but considering most of the people on IRC type with the acumen of autistic lepars, it didn't turn out to well.

There was one project where they mined medical journal articles for which genes were found to affect which properties. I don't have a link to the paper right now but if you're interested I can look it up.

Ironic... after my NN work I went to work writing genetic sequence analysers.. Mostly finding patterns in the contigs to find A/T stops (much more challenging than it seems, especially when a 0% false-positive rate has to be there)

I might put up a website at some point. The project is both school work and commercial work so I'll get around doing it at some point. The interesting part is school work and the useful part (the application) is commercial.

We should talk more, if you have the time/desire I am pretty sure I'd love to bring you into nerdfarm. Just a general mish-mash of open source work, some of it is theoretical. We're currently building the website framework right now so there isn't anything to see on the page.

Re:I just want a relational filesystem...

[jorleif]

I'll check nerdfarm every now and then so I can join if I have something to contribute.

What kind of language model did you use for the IRC analyser? It would probably possible to create a model resilient to bad spelling.

Re:I just want a relational filesystem...

[Xerithane]

What kind of language model did you use for the IRC analyser? It would probably possible to create a model resilient to bad spelling.

Just statistics based upon word relations, ignoring certain prepositions. I was testing an idea that I had to determine if you could find the difference between languages (like SVO/SOV syntaxes) through statistical analysis.

You can also check my JE, I post in there quite a bit.

Re:I just want a relational filesystem...

[2TecTom]

Yes, I agree, everyone is able and freely does use the genre field (or not) but what of cases where a person doesn't know which genre to assign?

As well, wouldn't it be better if there were tags for multiple or meta genres. Doesn't the depth, and consequently the power, of a system increase as you increase the number of meaningful and useful connections?

I think people forget that if you don't put much thought into it, you really shouldn't expect much intellegence out of it.

Re:I just want a relational filesystem...

[2TecTom]

Yes but who would know better than the artist which musical influences were the basis for a composition? As it is now, the labels do most of the classifying or it isn't done and then one has to assign them manually. I'm all for people using their own categories, however, I'd also like to know what the composer thought.

Besides this is a moot point if you read my initial post, as I was advocating multiple attribute fields which could be used to reference multiple generes or sort songs by other characterstics, style, tempo, key, happy, romantic, etc, etc. In fact, I should be able to specify I want to hear all fast tempo'd, upbeat, happy songs and my OS should be able to provide just such a filtered view. Of course, this assumes that the people who write software begin to notice that most apps are rather shallow and redundant.

If software was ships, we'd still be sailing.

Bad news.....

[oh2]

Can you say SKYNET ?

Re:Bad news.....

[Anonymous Coward]

no, But I can say ECHELON.

Skynet was a networked defence system with AI, Echelon is a (set of) program(s) which sifts through electronic data sources looking for like material and associating it in a non-structured way; faxes, emails, CCTV, phonecalls, voiceprints - it's all in there.

Haystack is much more like the latter, the AI is used to sift data, not make rational decisions with missiles as end effectors.

The ultimate test

[jabbadabbadoo]

The ultimate test for such a system is putting my inbox into the information stream. At the end of the day, 99% of it better be trashed automagically.

another tool

[Anonymous Coward]

I've heard about another tool much like this one, which have been developed through eXtreme Programming. It is a hybride between assembler, aspectoriented Java and Perl, and is platform independent as long as the platform is Linux or FreeBSD or, perhaps, an obscure Java-based realtime-appletkernel. It is used to produce a new kind of buzzwords.

Re:another tool

[Anonymous Coward]

what license is it released under?

Re:another tool

[Anonymous Coward]

It's released under a new license that forces you to release source code and does not allow binaries to be produced or distributed.

Bloatware anyone?

[henbane]

Isn't this just the equivalent of the usual attempts by designers to make applications to be everything to everyone instead of just doing one frigging job well?

"Information in One Place" - I already have access to all my personal information and files in appropriate windows. So does anyone using a basic OS.

The Allinwonder Pro File System?

[istartedi]

combining Email, IM, web pages, etc. into a single inbox

Whatever happened to the "does one thing, and does it very well" philosophy? If I sorta remember that I got something in an e-mail, I look in my e-mail. What's the advantage of throwing away that piece of information (where it came from)?

Yes, it's nice to use the computer to do grunt work for us, but there are some things that are better left to the user. Some of us like to come up with little "systems" for organizing things that are unique to us. We've all heard stories of the receptionist who files contacts under 'D' because new contacts are always invited for Drinks. An AI is not going to be any more rational than that, and the kooky system it devises won't be in our heads--it'll be in some obfuscated format that nobody will understand, not even the ditzy receptionist.

Re:The Allinwonder Pro File System?

[gmuslera]

You are not forced to use it. And many people could find this kind of things useful, Sometimes I can't remember when I read about something, and i.e. checking in the web, or all my mailboxes, or all my IM contacts history could be hard, specially if I want to specifically find my first and original reference.

I think that this is the main focus of the program, to improve our memory not searching all the internet, but specifically what you saw earlier and want to remember.

Of course, saying that this could be useful don't means that it actually is :) is the search database is full of spam, web popups, sites that don't exist anymore (at least in the original URL) or with questionable content, or boring IM conversations, not only will be very big but also relevant results could be hard to find.

But anyway, a tool (not need to be haystack, could be simpler) where you can tag information as important, interesting, or needed to be remembered later, and maybe annotate it (a web page, an image, a mail message, or a IM session) have a big potential.

Re:The Allinwonder Pro File System?

[Anonymous Coward]

Who says you have to throw away that information. Just make one of your attributes "WhereItCameFrom", and you're all set.

And the whole idea here is that you can come up with your own system for organizing - the website says user-defined predicates get treated just the same as the built-ins. All the self-organizing stuff is just to speed things up - you can make corrections (no harder than doing it yourself in the first place) and it gets better at doing things the way you want.

Re:The Allinwonder Pro File System?

[jorleif]

The "does one thing, and does it very well" philosophy was based on systems where you could integrate these utilities easily with each other, the UNIX command-line. Haystack is actually similar after a fashion, it makes all information processable within the same framework. With desktop applications you have a separated information into different applications that work on different information. What you would actually want to do is separate the different tasks into specialized interfaces optimized for that task, keeping the information processable by other programs when needed.

Receptionists might be stupid and AIs are not that bright either, but you must admit that a local google probably would allow finding the contact people anyway just based on contextual information.

Re:The Allinwonder Pro File System?

[istartedi]

Just make one of your attributes "WhereItCameFrom

If something like that is setup by default, fine. After I posted, I realized that this reminds me a lot of the DB-oriented file systems which have been discussed before on Slashdot. I think something like this would be cool at a low-level. Having a powerful database engine built into the file system is great. Having one table in a relational database is not great. In other words, this strikes me as something to be concerned about at the lower level, and not something that should be used to change the user interface.

Being able to search my entire system for "stuff Martha said" and get results in OlogN time. Good. Having the UI tell me that everything is in "the file". Bad. Having the UI default to a conventional HFS/Explorer type view, and offering me an alternative of "browse by date", and doing it schnell. Good. Having a new DB configuration chore on top of all the other computer related chores. Bad.

In other words, I think that if this Haystack thing was implemented properly, most users wouldn't notice the difference and the powerusers would be pleased.

Haystack and OSAF PIM

[dughutch]

Perhaps these teams should share some notes. I'm not sure about the web page info inclusion, but I'd love to use a top notch PIM that could data mine MY data.

As for the resources... perhaps you could tell it when you wanted it to data mine, so the mining would ot interfere with other work.

Correction...

[botzi]

It's not 12 or 128 but

# 512 megabytes of RAM (768 megabytes strongly recommended)

Damn.........!!!!!

Neat approach to the chaos problem.

[Meat Blaster]

I was looking at this sort of concept about four years ago, but was stymied by issues that were only recently addressed by artifical intelligence. One of the techniques that showed promise is ruminant nomination -- being able to abstract everything to a set of individual but similar (comparable) characteristics is a must. Unfortunately, such an algorithm was nearly uncomputable at the time, and bubblesort just won't cut it.

What's interesting about Haystack is that it seems to take advantage of such an algorithm, and in such a manner as to permit multiple feeds simultaneously on harvested data. Hopefully, the source is available somewhere to check out, because I'm curious about how they managed it.

Re:Neat approach to the chaos problem.

[lucifuge31337]

Hopefully, the source is available somewhere to check out

Jesus christ. It's bad enough when people don't read the article, but the damn story is about the SOURCE TO HAYSTACK being released.

Maybe you can write an algorithm to make yorself more observant.

So is this another search engine ?

[Rosco P. Coltrane]

I'll believe in their AI when I can type "X free" as a search query and it returns a link to www.xfree86.org instead of a million links to pr0n sites. Does this AI learn what people search for usually ? is it able to determine over time that capital-"X" and "free" in my particular searches are about opensource graphical software, unlike the same query by the dirty old man next door ?

By the way Haystack people, when you use titles and phrases containing "universal", "seeks to bring [...] to the average user", "artificial intelligence" , it trips my PR bullshit meter. I was about to bail out when I noticed the download link.

Re:So is this another search engine ?

[Anonymous Coward]

Or maybe YOU could learn to search for XFree86, or just plain XFree, which comes up with a full page of non-pr0n in google.

Re:So is this another search engine ?

[Anonymous Coward]

Believe

Google, now with believable AI.

Just have to know the right ...

[Anonymous Coward]

I thought Emacs already had that!

Wonder if..

[bigattichouse]

You could replace a desktop with this sort of interface, where apps you run would integrate into the one tool.. publishing information about their progress, etc

Isn�t this named for the problem?

[Anonymous Coward]

Isn't haystack the problem that this tries to fix? I think this project should have been called 'needle' or possibly 'findy.'

Re:Isn�t this named for the problem?

[sirius_bbr]

Isn't haystack the problem that this tries to fix? I think this project should have been called 'needle' or possibly 'findy.

Nooo, it refers to the HUGE requirements on your system to run it ;)

Heureka

[Anonymous Coward]

Haystack looks into the use of artificial intelligence techniques for analyzing unstructured information...

Does this mean that we finally got a tool that can help us understand our politicians?!

Performences...

[botzi]

# Haystack does not run as well under Linux in this build; certain features, such as the embedded Web browser, do not function, and the system is much slower.

I didn't see an explanation on why is that????
Suppose it has something to do with the JVM or is it a source code issue???
As far as I could see they don't provide an answer....Any guesses??

Re:Performences...

[IamTheRealMike]

I suspect they embed IE. We do that at work for one of our Java apps, it's very easy if you have the right tools (we use neva coroutine, see nevaobject.com).

It does reduce your portability somewhat of course :) I've been getting our app to run using Wine. Internet Explorer in a JVM in Wine on Linux is a bit bizarre, but we haven't seen any major speed problems with it so far.....

Re:Performences...

[Glock27]

Suppose it has something to do with the JVM or is it a source code issue???

The JVM(s) on Linux are fine, so that's not it.

Sounds like a case of IE-dependence to me... :-/

Screenshots

[ergonal]

There was only one measly screenshot in the overview section, and NO screenshots in the screenshot section, so here's another one.

Re:Screenshots

[botzi]

Dude, I totally support the guy that moderated as "Informative";o))))))))

Re:Screenshots

[Anonymous Coward]

I cracked up when I saw that moderation. He was either trying to be funny, or didn't FTFCL (follow the fucking comment link, akin to RTFA ;).

Attempting to find their server...

[bovilexics]

up and running is now like trying to find a needle in a haystack.

*rimshot*

Thanks /.

ITS ITS IT'S IT'S

[Anonymous Coward]

Honest to fuck, I'm going to add to my resume:

- Knows the difference between "its" and "it's" and will wield them correctly at the slightest provocation.

But...

[Anonymous Coward]

But can it find an apostrophe in the possessive third person neuter pronoun? No, because there isn't one.

bah

[SkArcher]

Opera does all of that - well, okay, it doesn't do Instant messaging, but i don't use that anyway.

But it does have a download agent, a web browser, a mail client and a newsreader all in one.

And its only a 3.7Mb download.

Re:bah

[Anonymous Coward]

in fact there is
it uses ICQ, I have the 6.0 version. I dont know about 7.0.

Re:bah

[a8f11t18]

I would like to point out Opera 7.11 is more
like 3.2 MB than 3.7 but oh well :)

Anyway.. Opera 6 used to have an IM client as
well!! think it was icq, never really used it..
but love the combination opera+M2 though..
never really liked the old mail client.

Anyway.. that's the first thing I thought when
I read the article.. that sounds like old Opera :D

Re:bah

[a8f11t18]

http://www.opera.com/graphics/docs/screenshots/ope ra-711-win-m2.jpg

That's how it looks like btw :) the thing on the left is the
"hotlist" (sidebars), not part of the mail window at the right.

Re:ITS ITS IT'S IT'S - Mod Parent Up!

[Anonymous Coward]

Its about time someone pointed this out!

When will this site be able to correct it's own grammatical mistakes?

Hahhahaha suckers!

[Anonymous Coward]

Nothing like slashdotting MIT to make you feel like you've accomplished something! How's your precious class-A IP registry now?

Sincerely

Bunker Hill Community College

Re:Hahhahaha suckers!

[Anonymous Coward]

There's slashdotting MIT, and there's slashdotting a little server at MIT. BIG difference.

(The big difference being that the main CS computer at BHCC is probably the size of the average desktop at MIT AI Lab)

Re:Hahhahaha suckers!

[Eric Savage]

There's being able to laugh at yourself, and then there's having such a large pole stuck up your ass that you need to post a reply.

WTF? No Mac OS X version?!!!

[alchemist68]

Sure would be nice if this ran fluently on other platforms.

Does this remind anyone of the digital dashboard?

[DigitalCH]

This reminds me of the digital dashboard concept from MSFT and others a while back. Basically the dashboard aggregated email, stock feeds, etc... and displayed it all together... Reuters, Avaya, and some other companies even came up with slick features to display IM, and Voicemail inside the email discussion thread as well. I really don't see this as anything new or noteworthy... Seems like they got a long way to go.

Archive.org's recent cache

[s88]

http://web.archive.org/web/20020210150513/http://h aystack.lcs.mit.edu/index.html

Re:WTF? No Mac OS X version?!!!

[Anonymous Coward]

It's assumed that if you don't run windows you are inteligent enough to organize your own info.

Re:ITS ITS IT'S IT'S - Mod Parent Up!

[Anonymous Coward]

Its a grass-roots movement that will have to act anonymously.

Haystack + ReiserFS

[Anonymous Coward]

Seems to me the features planned for ReiserFS would be ideal for storing RDF triples for efficient retrieval.

So imagine a Linux distribution with the future ReiserFS as the filesystem, and something like Haystack (rewritten in C) as the desktop, with a nice API...

Gnome and KDE have done a great job at catching up to the Microsoft desktop, maybe even improving it incrementally...but something like this would make Windows and OSX look as primitive as Win 3.0.

Re:Haystack + ReiserFS

[Glock27]

So imagine a Linux distribution with the future ReiserFS as the filesystem, and something like Haystack (rewritten in C) as the desktop, with a nice API...

Why re-write in C? Use gcj. If the current UI stuff is Swing, either use SWT, or re-write it using something like the Lightweight Java Game Library for fast graphics and cool special effects. :-)

Re:Haystack + ReiserFS

[KingRamsis]

right on !!!...
I was thinking how can such a beautiful UI be made with the crappy Swing, and I just read about some *SWT* issues with GTK+ on their site, now all makes sense...and a big light bulb is hovering over my head.

Re:Slashdotted, Google cache link

[beef3k]

Or use someone elses bandwidth at any rate. /.ing an .edu site is quite impressive.

I want the larger version...

[Anonymous Coward]

Gaint haystacks

Stop and think a moment

[jabber01]

And if, after thinking, you still hold the same view, please promptly uninstall your Operating System of choice.

Needle in the Haystack?

[OrangeGoo]

So now instead of finding the needle in the haystack, we have to figure out how to put the needle in the haystack?

Doesn't sound so hot to me...

Re:WTF? No Mac OS X version?!!!

[Anonymous Coward]

It's assumed that if you don't run windows you are inteligent enough to organize your own info.

That would be much funnier if it didn't run on Linux.

Wait a minute .....

re: neat approach to the chaos problem

[ed.han]

"and bubblesort just won't cut it"

well, see; that was your problem right there: radix sort! :D

What's so revolutionary?

[javaaddikt]

What's the big deal? Yahoo and other portals have had customizable news/weather/mail/todo/calendar pages for years.

google cache

[Alpha_Nerd]

http://216.239.33.100/search?q=cache:KtKSHcwESUQJ: haystack.lcs.mit.edu/+&hl=en&ie=UTF-8

Re:google cache

[Alpha_Nerd]

Google cache

Sorry about the first one, I'm an idiot =/

Haystack

[Anonymous Coward]

A single all powerfull programme to do everything ... hmm sounds like Windows ... or Sauron's ring

I was a usability tester...

[Anonymous Coward]

for Haystack at LCS recently, and was not that impressed. It is designed to do certain kinds of tasks very well (e.g., editing things that are embedded in other types of information - the tests given were things like "edit this picture that's a part of this entry in your Outlook address book"). Unfortunately, at the expense of making these tasks as close to one-click as possible, other things (versatility the most, but also common sense design) have failed.

I find it easy enough to edit information of the "My Documents" variety without worrying about how it is integrated into other information on my computer, and I'm sure other readers here do, as well.

The best way to actually use this software would be in the case where John Q. has a specific task to do over and over again but isn't ready to tackle a batch process.

Looks like someone forgot the basic rule...

[Anonymous Coward]

KISS.

Agents...

[orn]

Haystack is an interesting idea, but I have a hard time distinguishing what it does from what, say, Lotus Notes does. And Lotus is _terrible_.

I like the idea of bringing all my information together in one place. I don't like the idea of only having it in that one place. What I would like would be an application that can watch how I use the computer, then bring those applications together to make it more seemless.

For example, I have about four different calendars in my life: the work calendar, the one on the cell phone that I use for stuff that I can't miss, the calendar that schedules airplane rentals, and (of coursE) my girlfriend's calendar. So how do I bring those all together, and yet still be making entries in them separately?

The same is true for information. I have a primitive blogging system (really just a bunch of text files that are date coded), I have work documents that I use regularly, I have web pages that I monitor (sometimes a little too often) and I have textbooks that I'm reading (instrument flying at the moment). So how do I get all these forms of information - or at least an index into them - together in one place? But again, without changing the current organization scheme.

This is the tool that will make the computer a lot more useful - an actual organizational tool.

Rudy

Re:Agents...

[gobbo]

Wholehearted Agreement with the parent. Lotus Notes is shoved into our laps at work, and it's been a struggle to part out its functionality into the proper parts: Mail.app, Safari/Camino, Address Book (waiting for propr LDAP support, grr), iCal, and other 'business' tools, on my machine. [Not that I'm an Apple Software Fanatic, but they work and fit into the budget.]

L.Notes had a whole wing on the now-MIA Interface Hall of Shame. It reinvents the conventions found on other platforms (it tries to be a platform unto itself) and does so badly; it's buggy, slow, and designed for administration [decent encrypted document database scheme].

Plus, it centralizes, for better or worse, all my information on servers controlled by I.T..

Now I'd love to have a central app that takes feeds from my favourite info management apps, sorts/ranks/prioritizes/interrelates the items for me according to my usage and prefs, and lets me 'zoom in' to a task by switching to the preferred stand-alone app at will. Haystack has only part of the picture, the model is still gather-control, rather than sift-sort-go.

One item I've found intriguing is StickyBrain, a sticky-on-steriods app, by Chronos LC, which takes info in many categories and allows for quick index searching, plus offers system-wide info-archiving services and some alarm and word-processing features. I had the same kind of thing running with BBEdit, a notes directory, and grep, but it was like hammering nails with a wrench.

I want all my info hotlinked to lists of related items, dynamically: make every significant word a keyword, realtime. After all, what are multi-GHz and piles'o'RAM for, anyway, when not rendering?

Re:Agents...

[CynicTheHedgehog]

iCal for MacOS X allows multiple calendars, each overlayed in a different translucent color. You can activate or deactivate the display of each independently. It has exactly the same flexibility as the Outlook calendar when it comes to recurring events as well. I thought that was pretty inventive...comes in handy when juggling work, school, and my personal life (red, blue, and green, respectively).

Whether or not that helps you, I dunno, but applications like that certainly do exist.

Re:Agents...

[bobbozzo]

So how do I get all these forms of information - or at least an index into them - together in one place? But again, without changing the current organization scheme.

Something like Google should work, except for your calendars.

Re:Agents...

[puck13]

Check out Boswell for the Mac. You enter, import, copy/paste whatever text you like (including entire directories, email archives, etc). It's got an amazing search engine that can manually or automatically filter any entry into any number of virtual folders based on any number of key words. The interface needs some work, but the concept is magnificent.

I don't (and you probably don't either)

[73939133]

A file system with the power and flexibility of a relational database ceases to be a file system. What are things like "cp" supposed to mean? How do you transfer "a row" through a serial connection? What kind of transactional guarantees is it going to make; if it's going to make DBMS guarantees, it's too slow for many file system applications, and if it's not going to do that, is it really a DBMS?

If you want a database, just use a database. MySQL and various embedded databases are widely available on Linux now; no need to clutter up the kernel.

Re:I don't (and you probably don't either)

[Xerithane]

A file system with the power and flexibility of a relational database ceases to be a file system. What are things like "cp" supposed to mean? How do you transfer "a row" through a serial connection? What kind of transactional guarantees is it going to make; if it's going to make DBMS guarantees, it's too slow for many file system applications, and if it's not going to do that, is it really a DBMS?

I didn't say "relational database" -- I said "relational filesystem." As in, finding documents that are related to some other entity. I enjoy messing around in the Gimp. Sometimes I do work related images, other times it's just for fun. I'd like to put every image under $HOME/gimpwork. However, I like to find out which ones are for work and which job, for fun, etc.

I'd like to be able to say "ls --category=work $HOME/gimpwork" and show only those files. This doesn't require a database, it requires a few meta flags.

File copying is the same, ls is the same, everything is the same. Maybe just a wee bit slower.

If you want a database, just use a database. MySQL and various embedded databases are widely available on Linux now; no need to clutter up the kernel.

You wouldn't have to clutter the kernel. A system that I am envisioning could reside purely ontop of any existing filesystem. It could have a DB backend (but that would be overkill)

There are some logistics problems that would make it easier to be in a kernel module -- but assuming everybody would use the proper set of commands, it could keep everything in sync just fine without mucking in kernel space.

Re:I don't (and you probably don't either)

[73939133]

I said "relational filesystem." As in, finding documents that are related to some other entity. I enjoy messing around in the Gimp. Sometimes I do work related images, other times it's just for fun. I'd like to put every image under $HOME/gimpwork. However, I like to find out which ones are for work and which job, for fun, etc.

Perhaps you should just familiarize yourself a bit more with UNIX command line utilities; the ease and power with which skilled UNIX users can find and organize things is one of the major selling points of UNIX.

If you want to find all the tex files containing the word "business" and having been modified recently, use:

$ find ~ -mtime -2 | xargs fgrep -il business

If you have a command line program "similar-to" that outputs a numerical similarity score to a given target file, you can do something like the following to get all the files under your home dirctory that are similar to ~/target.tex, sorted by similarity

$ locate .tex | fgrep ~ | xargs similar-to ~/target.tex | sort -rn | more

If you want to set up some tagging system, try using symlinks or hard links:

$ mkdir -p ~/categories/{home,work,fun}
$ cd ~/some-project
$ ln -s *.jpg ~/categories/fun

Organizing data is hard, but chances are that whatever you want to do is already pretty easy with existing tools in UNIX/Linux; otherwise, people would have already added tools for addressing the problem long ago.

Re:I don't (and you probably don't either)

[73939133]

Oops, typo, that should have been...

If you want to find all the tex files containing the word "business" and having been modified recently, use:

$ find ~ -mtime -2 -name \*.tex | xargs fgrep -il business

Re:I don't (and you probably don't either)

[Xerithane]

Perhaps you should just familiarize yourself a bit more with UNIX command line utilities; the ease and power with which skilled UNIX users can find and organize things is one of the major selling points of UNIX.


Uhm, typical "I'm a l33t l33n0x" response. Thanks for wasting your time on an attempt to try to make yourself seem more elite. If you had actually read what I was going for, you would understand that your system of symlinking and fgrep lines makes you look like a fucking retard. Like this:

Organizing data is hard, but chances are that whatever you want to do is already pretty easy with existing tools in UNIX/Linux; otherwise, people would have already added tools for addressing the problem long ago.


Because Haystack is doing something different than what is done, and Mac OSX is working towards the same goal. Linux just clones what the competition is doing, a few years later.

Thanks for playing, save your elitism for high school.

this is site is done

[Anonymous Coward]

tried to download it and it looks like it is /.

Site slow, google cache:

[Anonymous Coward]

I like clickable links

However,

[cyt0plas]

Representing data is all fine and good, but how about a program that actually _derives_ data worth representing?. For me, being able to represent my sock drawer in a nice, organized manner doesn't do me much good.

From their "Overview" page

[Zaxor]

"Ever start a conversation in an instant messenger, switch to e-mail, then switch back to instant messaging just to realize that you had forgotten what the conversation was originally about?"

No.

I'm with those who prefer their apps separate.

Re:Slashdotted, Google cache link

[Anonymous Coward]

Not quite as impressive as it might seem. Research groups running inside LCS all run their own servers, so we're probably just banging on some poor Dell box. Also, lots of groups use their workgroup server for everything like nfs/smb/print/web and even number crunching. Maybe they're testing Haystack on that server too... ouch. When we've slashdotted the main MIT web site (web.mit.edu) then I'll be impressed.

Re:Awesome "Mozilla" effect.

[Anonymous Coward]

I've seen it on other pages using IE

Big whoop

Very funny, a haystack is what I have *now*!

[Fefe]

I need something against my haystack, not a software that does more sedimentary data storage.

You mean vi

[A nonymous Coward]

Emacs already has all that.

Emacs!

Vi!

Come on come on, one and all, joint the flamefest!

Mr. Godwin, meet the nazis of the unix world.

PDAs?

[Dynamo247]

The sys reqs for Haystack do seem a bit absurd. Wouldn't Haystack really have it's place on a PDA where you'd want fewer programs sucking up your hard disk space without giving up functionality? I'd freaking love it if my iPAQ had one interface for all my needs. All they have to do is "shrink it" down req wise...but i guess if it were as simple a job as putting a word in quotes then it'd be done already.

Wow this program SUCKS

[Anonymous Coward]

I am attempting to run it for the first time, it's been going for 10 minutes and still running (I hope, damn Java app).. I'm on a machine with a 2 GHz CPU, and 1 GB RAM.

Six Degrees

[mblase]

Six Degrees by Creo is another attempt to do this same sort of thing, except that it's commercial and it's been available for Mac OS X and Windows for several months.

Re:Wow this program SUCKS--Me Again

[Anonymous Coward]

OK, it finally finished.. then... CRASHED! I'm done. Deleting this, and Java from my system :-)

Dying for a torrent here.

[Sanga]

(no text)

Have you tried it?

[jamie(really)]

I've been waiting for this for a few weeks now. I've been looking for a PIM that has email, calendar, and tasks. Apart from Outlook, what product has that? I have recently tried:

Outlook
OSAF's Chandler PIM
Haystack
Pogomail (not a PIM)
Eudora (not a PIM)
Mozilla

I am now using Mozilla because it has bayesian spam filtering built in and because it has a calendar plug in.

I have decided not to use Haystack. It is simply not production ready, and I'm sure the guys at MIT wont mind me saying so. It crashes. It locks up. It doesnt have undo!!!! I cant tell you how many times I screwed up one of the panels and couldnt get it back. I also couldnt figure out how to delete spam. I get about 200 emails per day, of which 8 arent spam. I could use a pop filter, but I have an emap client too.

However, I am very impressed by this software and it is absolutely the way forward. I *want* my information integrated. I want my tasks to automatically reference the people I need to do them with and the web pages I used for reference and the dates in my calendar. I want my contacts to appear in many different categories, instead of as a different copy in each category all of which I'd have to update.

I want email and calendar and tasks to be like a light switch or a tv. I want to just turn it on and it all be there. This software is fabulous and you would all benefit from giving it a test drive, even if you ultimately uninstall it.

This morning some Lizard Lawyer has ....

[Anonymous Coward]

included Haystack into a pending patent, which they keep open for this purposes, with enough buzzwords to make Haystack appear to part of the original patent application, which could be several years old.
The USTPO will approve it no doubt, and then the lizard will start a crusade against those who 'stole' his 'Intellectual Property', siting the original application date of the patent as proof that he had developed Haystack much earlier than MIT.
I haven't met a lawyer yet who didn't think he wasn't an expert in every other discipline on the planet.

READ THIS!

[Anonymous Coward]

"One of it's more useful features allows you to..."

Please people, engineers, and programmers--English grammar is no different from any science, there are rules and you should learn them!

it's == it is

The correct rendering of this sentence would be:

"One of [its] more useful features allows you to..."

Even a second grader knows the difference, please educate yourselves and others, don't use the excuse "but I can calculate quadruple integrals, spelling and grammar don't matter." They do, it's the bare minimum. I can also calculate quadruple integrals, but at the same time write grammatically correct sentences, so what's your claim now? Don't be ignorant, broaden your knowledge or you will pass on these and other fine mistakes to your peers and children, that in and of itself should be considered a crime.

Slowwwwww!!!!!

[Anonymous Coward]

No wonder that the app is slow, it was made with java. I can't understand why people still use that slow framework.

Scopeware Vision

[www!!!1]

Scopeware Vision is similar but better than this. It only requires 128 megs of ram!

Try the 30 day free trial. It rulz!

ultimate fat app

[Anonymous Coward]

So is this the emacs or the microsoft.bob of net apps? Or maybe the microsoft.bob.emacs of net apps, heheheh.

In the Brave New World your computer. . .

[kfg]

operates you!

No more nasty "thinking." It will tell you "where you want to go today," how you want to get there, how you want to see it and how you store it for retrieval.

Choice is good, so choose not to be bothered with making your own choices. That's the ultimate choice. Brainless sheep for diversity unite!

Of course this is just a preliminary step along the way to the ultimate goal. In the Brave Newer World of the future you will turn your computer on, stick it in the closet, and it will simply do everything for you on it's own. Read and answer your mail, conduct your business, play your games, etc., all without any interaction with you at all. Think of the time and productivity savings!

Once a year it will print out a summary of how your life is going on a 3X5 card in easy to read large type which it will place in a location it deems to be most suitable for such an item.

Bend over. It won't hurt your computer a bit.

KFG

EMACS SUX0RZ

[Anonymous Coward]

Vi rulezzz! James Joyce rulezzz also!

I forgot to tell you, IT SUPPORTS OGGS!

[www!!!1]

I know because I added that feature myself :)

Re:Yay, Java is Fun

[oblom]

After 15 minutes of bootup/installation process with 100% CPU and 150 MB memory usage it just disapeared from process list.

Not good enough for usability experts.

what it sounds like you want...

[Ayanami Rei]

is extended ext2 attributes (google for the kernel patch and userland utils)
It allows you to give attributes to files, (like "category")
I understand there's also gnome support.

Re:what it sounds like you want...

[Xerithane]

is extended ext2 attributes (google for the kernel patch and userland utils)
It allows you to give attributes to files, (like "category")
I understand there's also gnome support.

Index the attributes and put limited statistical information checks on it (also access count, would be nice) and yeah... pretty much.

Another Iteration of the OpenDoc

[jonbrewer]

From the Design Principles:

"...provides a single, uniform interface for manipulation of e-mail, instant messages, addresses, web pages, documents, news, bibliographies, annotations, music, images, etc."

"...attempts to match a user's own focus on objects in view and what can be done with them. An operation (such as spellchecking, sending an e-mail message, or rotating an image) can be invoked at any time on any object for which the operation "makes sense" (i.e. a blob of text, a person, or an image respectively)."

Back in the heady days of the PPC 601 and the Newton, one of Apple's software groups was working on this problem exactly. While I don't think OpenDoc could organize your information, it was certainly a uniform interface for manipulating stuff, with the focus on the stuff, and not the application in use. At that point, about seven years ago, I naively believed that one day OpenDoc would provide an environment in which I could edit a web page and all elements (including raster and vector images) without having six applications loaded. Ha!

C4n u say g00gl3

[Anonymous Coward]

idd, all of the above and learn to use quotes "X free"

however i must say i prefer "XXX FREE" ;p

or better yet, an 'i feel lucky' on

"* my big black * you * *!" (very funny :)

seriously, go to searlores.org and learn to go00ogle

Understandable, I have unlimited calling

[notque]

With Unlimited calling, versus a pay per text message, of course I would choose a call.

I seem to forget that not everyone has that. So obviously my reasoning is different than everyone elses.

Understand your point of view very much now.

Ugh...

[Lovgren]

Just tried it. The UI response time is so slow that I uninstalled it ten minutes later out of frustration.

hehehaystack

[Anonymous Coward]

All you need do to find a needle in a haystack is to sit down in the haystack. Murphy was an optimist.

Re:hehehaystack

[c64cryptoboy]

1) Burn the haystack
2) Get out a metal detector and find the needle.

This reduces the problem from NP to P complexity.

Re:WTF? No Mac OS X version?!!!

[Anonymous Coward]

Mac OS anything? You a Mac weenie buddy? A ONE button mouse idiot? Go home, and do not come back without a note from your mother.

The most innovative thing is...

[clard11]

..the idea of storing away a "UI continuation" as an object, which can be used to restart an operation from some pending point.

Integration of email and IM is already available in commercial products like WebSphere Portal Server.

Re:The most innovative thing is...

[jorleif]

Yes this is certainly very cool, although the picture rotation example seems kind of lame from an extendability perspective =)

Continuations in the UI are really a significant idea and have the potential to make sessions much more useful. It would also solve the problem with UI-events "stealing" the focus from what you are trying to do. Instead of popping up som idiotic dialog when one is trying to type the event would wait nicely until the user can bothered to answer the question.

I wonder if it would be possible for the continuation not to contain only the state that it currently is in, but also the commands that got it there. If that would be possible then creating macros would become a lot more convenient. You would just perform the operations once, and then click the continuation and say "Make Macro", instead of repeating the steps with an operation recorder toggled.

extensions?

[hitmark]

i just wonder about one thing, can one build extensions to it? like say instead of building everything from a webreader to whatever the haystack system just act as a kinda glue and whoever can maybe write a better web part or mail or maybe some games to it? if so then i see this as the next gen UI! context sensitive action menus (rightclick menus), the ability to gather names for anything from meeting chedules to mass mail sendout from anywhere, having the ability to not only bookmark pages but allso actions, and i must say i realy loved the idea that any action you want to do and that needs more feedback puts a box to the side so that you can go around until you are satisfied. the worst idea every for the desktop methafor was the part about stacking windows! im downloading, maybe it will chug badly on my pc but i belvie for the most part that its the "database" system that is the reason for this as its digesting every mail, im and whatsnot that passes tru the system! hmm, someone commented that all this was stuffed into emacs. i wonder what FSF/stallman was aiming for with that one, having it run as the UI on top of the hurd directly? i wonder why they dont just merge the 2 projects and realease it as GNU...

Microsoft BOB

[Anonymous Coward]

So, is this similar to Microsoft BOB?

Used to be called...

[Lulu of the Lotus-Ea]

Lotus Notes.

I used this product back in 1998 on my then job. I liked it pretty well then, and I understand they've made some improved versions even.

Re:Used to be called...

[slim]

Are you kidding me?

Finding stuff from Notes is like... well.. finding a needle in a haystack.

I haven't read all the threads.

[Steveftoth]

of this discussion and I'm posting late so this might be a little redundant.

After reading the information at their web site I think that they are trying to do the same thing that Apple is doing with tthe iApps, just that Apple is slower because they chose to integrate them differently and Apple also has to produce something that people can use now.

They want to basically integrate the different types of data that we all use on a daily basis, email, web, IM, etc in to one data repository so that any program can use them. Instead of having to write a seperate address book for your word processor and use that for mailing labels, just have one large address book for your entire computer. Apple's address book does the same thing. Instead of having a different spelling checker for every single application, use one global one that has a custom dictionary, that way you don't have to enter your last name into like 5 dictionarys.

They want to provide a unified interface to your data instead of 5 disparate ones. Apple is trying to do the same thing. Trying to seperate the data on the computer from the interface. So that the user only has to deal with the interface and not the details of files.

Yes ,you can just use the file system. You could just think about a computer in terms of the HFS that everyone has. But does everyone actually want to? A Tree based file system works for some people, but others like lists, and want everything to be in huge (really huge) lists. Instead of having to drill down into their deep, nested file system, some people like to have all their files in one folder. (Usually the desktop)

Some people want to deal with their data in tree format, some in list format, maybe some remember everything by name and others like lots of pretty colors and icons to represent their data. People who have used the mac might know about Finder Labels that let you assign colors to files and folders. So you can then make say a red word document called 'resume' and that way you can remember that it was for emergency use only. Or some other such scheme that one might invent for their own data.

The point is that everyone is different and today's programs don't allow you to interface with your data in many ways. Since every program has it's own data store, it makes it hard to move data from one application to the other. Word maybe your favorite word processor, but when you want to use features like the mail merge, you have to massage your data from whatever program you are using to keep track of all your friends. You could use the outlook integration features, but even then it's using 2 seperate programs to access your data, since Outlook is storing it. Really, shouldn't there be one source for all your personal data on a computer and then outlook draws from that when it needs addresses or Word draws from it when it needs addresses.

ITs called usenet.

[Anonymous Coward]

alt.binaries.erotica.brunettes.

Re:ITs called usenet.

[Lord_Dweomer]

If you had read my post in its entirety you would have seen that what I am looking for is more than just a category of "brunettes". Read the post before you make a response.

It sure does.

[Sunnan]

I use Emacs for mail, news, IM (via erc/bitlbee), as frontend to various websites and wikis, and as a planner/calendar.

I've been planning to switch my phonebook over to BBDB as well.

Just because it's from MIT...

[Fly Ricky - The Wine]

doesn't mean it's actually a properly framed idea. The market pressures of usability have pretty much spelled out the answer long ago... different functionality calls for discrete apps in this instance. There's simply not enough synergy between IM and Email being in the same place to make it worthwhile... it's just cluttered. It would have happened long ago and been successful if it were useful because it's not technically very difficult to accomplish. Blah. Some ideas that come out of that place are pretty weak (and others rock.) Oh well.

No, an arbitrary desktop menu

[chriso11]

What I think would be cool would be a multidesktop type of environment. No, I'm not talking about multiple virtual desktops either.

You could have a different desktop for each project. You might have several emails for the given project, a few documents and spec sheets, some pictures, and some code. Keep the hierarchical file system underneath. Everything on the desktop is a link to something in the filesystem. Make it easy to copy, manipulate and navigate between different desktops. Basically, this would be an alternative hierachy, independent of the filesystem hierarchy.

Enfish Onespace

[vivarin]

http://www.enfish.com

Same thing -- hard to make it fast enough.

sounds familiar (blatantly plugging my own work)

[wdebruij]

Reading through the information on the haystack website I got a small deja vu feeling... a few years ago I designed a system that indexes information on your harddrive using (among others) the google directory to create dynamic categorical hierarchal views on your data ( also in RDF ).

Since then the project kind of stopped, but if you're
interested in this sort of research, please check it out at http://atoms.sourceforge.net/

Being the work of a single person, atomsnet is far from a polished application. I still believe the underlying idea has potential, though. I just can't find the time to expand the number of indexing plugins to make it more useful at the moment.

NB A paper about atomsnet was published at CIVR2002

Haystack, Creo's Six Degrees, and Business

[omnipotens]

I work in a small financial services company that contracts through a much larger one. The amount of contacts that each of us makes each day is mind boggling. Haystack was such an exciting idea (I read about it yesterday) that I had to run out and see it for myself. Well, haystack itself is something like what we need-- but it's current iteration is way too buggy.

So, reading other /. posts, I found out about Creo's product, Six Degrees. Unfortunately, I've found this review of that product:

http://www.cnet.com/software/0-3227888-1205-202421 36-1.html?tag=rating
That ain't good!

But, to everyone who's reading, I've got to explain just why this is such an exciting idea to mo. Okay, as I said, we recieve an amazing number of contacts each day. If we could track that, we would be infinitely more efficient. If there were a program that could track that, as well as correlating documents on the hard disk to those people, we'd be golden! The trick, of course, is to have the program do this all transparently, because no one really feels like doing all of the correlation manually. In the end, color me excited, very much so, about Haystack.

Also, if anyone knows of any software for windows that does what I'm looking for, please let me know at

J Gad ik i@L U C.E D U

Thanks!