Thanks alot for your reply, I'm the guy responsible for this post.
Some thoughts:
The beauty of 'Unix Mail' is that it works from large-scale Pop-Toasters down to minimalist clients. However it has some shortcomings. The same is true for Maildirs.
Think about a format which has similar properties of scalability and portability but adding 2 features:
compression
a standard way to add features
I thought about db* (perhaps sleepycat) for a particular reason: It is simple. It can be made portable. And it's already their on a fair amount of systems.
Sendmail and other MTAs could deliver mail to it, MUAs could access it simultainesly, MUAs could store its indexes in it. Imap-Servers could store their UIDs and flags in it. Evolution could share flags and indexes with Imap-Servers.
Think about writing a lib which overrides some libc-calls (open etc.). Insert it via LD_PRELOAD or at build-time and expose a Maildir or mbox-view of your db-file to leagacy software (think qmail). I don't know if it was practical, it would be geekish though:-)
We put mail in another location, but everyone else has done that too, elm:Mail, pine:mail (or is it the other way around?), netscape:ns_mail, etc. At least we now offer the option to read most of this 'in place'.
This last sentence sounds like you also think it stinks that clients do that. There must be a better way. Currently I use: Courier Imap, accessing a Maildir++ (Courier-speak) with subfolders, for remote reading. Pine using a custom built c-client with Maildir-support and overridden subfolder-paths (à la Maildir/.Sent) for console-reading. A.qmail file for filtering/sorting incoming mails. So in a way I already have kind of an unified mailstore. This is however a nightmare to set up. I don't even know wether it is possible to set that up using central configuration files. There must still be a better way.
You are of course right, the problem of the missing grand-unified-mailstore is of course not essential, it does exist however. Is it worth fixing?
Is Camel suited for use in MTAs? Can you link Sendmail, Postfix, Exim, qmail, etc. against it? What about POP/IMAP servers?
Unfortionatly I haven't yet tried out Evolution, I will do that soon though.
Sleepycat, gdbm, ndbm are just libs containing database routines. Do you really think Evolution implements it's own indexing functions, just because it's fun to write them? I bet Evulution also uses eighter standard Berkly db, gdbm or Sleepycat.
Oups, aparently theiy don't. (db for vCards, but something new for indexes?)
2. Evolution's body indexing and summary files are extremely fast and efficient, about the best you'll get. I hear MySQL has text indexing capabilities that are extremely fast, but I'm not sure if they are faster than Evolution's indexer or not. Might be interesting to check this out.
so by default it's still mbox with some database features:-)
However, you have to be aware of the consequences of this. Most importantly, you will not be able to validate any of your PGP/MIME or S/MIME signed messages as according to the RFCs for these types, the signed MIME parts MUST be treated as opaque (meaning that you may not modify them in any way).
That's why I love Slashdot. While most of the issues rised in comments above are things I already thought about, that's a new point. So whatever is implemented, it must be able to reproduce the original message exatly. Even the --cut here 01204737473829 -- parts. That's a very important design criteria.
Please not Sleepycat. If you are so sure that a generic database backend will be better than what Evolution's got, at least have the sense to use MySQL or PostgreSQL.
Sleepycat, gdbm, ndbm are just libs containing database routines. Do you really think Evolution implements it's own indexing functions, just because it's fun to write them? I bet Evulution also uses eighter standard Berkly db, gdbm or Sleepycat.
I concur in that it would be a nightmare having to set up a RDBMS just for using a Mail-client. Although it would be justified for a big-scale POP-Toaster or similar.
I know that my post didn't make it clear enough. I also like the simplicity of mbox and Maildir, because you can use it for huge mailservers down to minimalistic MUAs. Both have shortcomings though, so what I want is something simular, simplistic, but with optional compression and a standard way to add features like indexes, IMAP-flags, etc.
Ironically I thought about something like that: Writing a lib which wraps the libc-open etc. calls and provide leagacy programs with an mbox or Maildir view of the mail-store.
MTAs and MUAs could than access the mail-store via native api or in the traditional way, only needing some LD_PRELOAD flag.
Look at who sponsered the research. Isn't it the same guys within the Bush administration who profite from continious exploitation of natural resources?
Isn't it just another atempt to justify Bush's decision "to do nothing that could endanger the groth of our economy"?
Does any independent source actually think this could work?
I just wonder in how many exciting ways this whole new shit is going to be exploited. Interactions between Javascript, Flash-bytecode, embeded html (?), IFRames, flash in xml in javascript in html in flash (?),....
Internet Explorer is being closed due to an unknown exception in kernel32.dll......
As Linux, the BSDs and other free operating systems must be similarily affected, why can't a few of the major distributions or the FSF join together and risk suing. Perhaps this time Be's Gassée would be willing to speak up in trial.
So what other possible paranoid ranting could one come up with that could make this scenario possible... Hmm... How about if Microsoft bought themselves the US Congress and made it a law? That's it! The government that sued them for antitrust violations is going to turn around and heavy-handedly enforce a complete, 100% monopoly! Yeah!
They don't need the government. Think DVD and regional-codes. I call this a precedent where a couple of companies teamed up to form a cartel, which turned out so powerful that they sucessfully implemented stronger restrictions to the medium DVD than the law would enable them to do. And don't think it cannot happen again.
The guy who wrote Reiser FS has quite some things to say on this very subject: What he says (in short) is:
.) Databases shouldn't exist. They exist only because System-Programmers didn't listen to Application-Programmer's needs.
.) Those Application-Programmers implemented their own solution to the problem: Databases.
That's why he started Reiser FS. It is today the ideal filesystem to store little bits of information ( 100 bytes) which you normally store in Databases, because you don't want millions of small files lying around.
(I know this is a simplified view, read his own arguments for a broader discussion.)
... but for the next 3 years or so you will defnitly be fine with CAT5. Gig-Ethernet works over CAT5, there is even a standard for Fibre-Channel over CAT5. Nics with fibre-connectors still are expensive.
In 10 years however you will probably wish that you had spent the few extra bugs to put in multimode fibres.
This is not about Steganography! Your statement about Steganography is right of course, but this is different.
1) Each of the to partitions ("aspects") in the 1GB space ("extent") is actually 1GB big. It is just that the data in all aspects together cannot exceed 1GB.
2) You could actually have 5 aspects (1 for letters to you mum, 2 for emails, 3 for xxx-pics, 4 for faked ultrasecret data and 5 for real ultrasecret data). You could give the court passwords for 1-4 and there will be no way to prove that there is a 5th aspect. That is the deniability.
The court could suspect it because data in partitions 1-4 only accounts for 850MB of one 1 Gb. But there is no way for them to know for sure.
Neighter can you prove that there is no other aspect, neighter can they that there is.
3) All the aspects _are_ writable. Each one on is own as well as any combination of them. The problem is, when writing to any subset you will of course destroy those not in the particular subset.
4) In other words, when you "forget" a password for any of the aspects, the effects are the same as if this aspect never was there. (If the underlieing cryptosystem is secure.)
In a strict sense its true that the post is "Informative". As informative as if he wrote that Slashdot is usually green, that Rodger Rabbit is a Cartoon and that Bin Laden is evil.
Did I already mention that Bin Laden is evil?
--
Live on earth may be expensive, but it includes an annual free trip around the sun!
Having just finished reading "Applied Cryptography" from Bruce Schneider, I know that anonymous, cryptographic, secure money-exchange protocols exist. But they are difficult to implement, use a lot of bandwith and space and are somewhat slow. They include very advanced crypotgraphic technics like "blinded signatures", "oblivious transfers", etc. IMHO those kind of things are just not ready to be implemented in large scales.
So what do we have in the meantime: Creditcards and not-anonymous e-cash systems. All those systems involve traceability of money transfers. But society simply doesn't want to give up the anonymity of a simple Dollar-Bill or the soon-to-be Euro or whatever.
Think about Illegal Financing of Political Parties, avoiding Taxation (extremly common in Europe because you simply just can't afford to build your 1000 sqare-meters house the official way), Prostitution, Pimps, Drugs. Or think about the husband who doesn't want his sex-shop bill to show up on the family-credit-card.
There are just to many people who have vital interrests in Anonymity.
Slashdot Mirror
To celebrate RMS birthday I set up a GNU/Hurd machine. Check out http://absturz.htu.tuwien.ac.at.
And don't let netcraft fool you. They don't have their OS-detection-logic up to date. Neighter has nmap.
And if you can't reach it, that's a feature (TM) since it's stability is worse than Win 3.0 with DR-Dos 7 underneath it.
EdgarWell, the performance-problem can be overcome, but that still leaves the diskspace-bloat-problem.
Caraclla
Some thoughts:
The beauty of 'Unix Mail' is that it works from large-scale Pop-Toasters down to minimalist clients. However it has some shortcomings. The same is true for Maildirs.
Think about a format which has similar properties of scalability and portability but adding 2 features:
I thought about db* (perhaps sleepycat) for a particular reason: It is simple. It can be made portable. And it's already their on a fair amount of systems.
Sendmail and other MTAs could deliver mail to it, MUAs could access it simultainesly, MUAs could store its indexes in it. Imap-Servers could store their UIDs and flags in it. Evolution could share flags and indexes with Imap-Servers.
Think about writing a lib which overrides some libc-calls (open etc.). Insert it via LD_PRELOAD or at build-time and expose a Maildir or mbox-view of your db-file to leagacy software (think qmail). I don't know if it was practical, it would be geekish though :-)
We put mail in another location, but everyone else has done that too, elm:Mail, pine:mail (or is it the other way around?), netscape:ns_mail, etc. At least we now offer the option to read most of this 'in place'.
This last sentence sounds like you also think it stinks that clients do that. There must be a better way. Currently I use: Courier Imap, accessing a Maildir++ (Courier-speak) with subfolders, for remote reading. Pine using a custom built c-client with Maildir-support and overridden subfolder-paths (à la Maildir/.Sent) for console-reading. A .qmail file for filtering/sorting incoming mails. So in a way I already have kind of an unified mailstore. This is however a nightmare to set up. I don't even know wether it is possible to set that up using central configuration files. There must still be a better way.
You are of course right, the problem of the missing grand-unified-mailstore is of course not essential, it does exist however. Is it worth fixing?
Is Camel suited for use in MTAs? Can you link Sendmail, Postfix, Exim, qmail, etc. against it? What about POP/IMAP servers?
Unfortionatly I haven't yet tried out Evolution, I will do that soon though.
Edgar
Oups, aparently theiy don't. (db for vCards, but something new for indexes?)
2. Evolution's body indexing and summary files are extremely fast and efficient, about the best you'll get. I hear MySQL has text indexing capabilities that are extremely fast, but I'm not sure if they are faster than Evolution's indexer or not. Might be interesting to check this out.
so by default it's still mbox with some database features :-)
However, you have to be aware of the consequences of this. Most importantly, you will not be able to validate any of your PGP/MIME or S/MIME signed messages as according to the RFCs for these types, the signed MIME parts MUST be treated as opaque (meaning that you may not modify them in any way).
That's why I love Slashdot. While most of the issues rised in comments above are things I already thought about, that's a new point. So whatever is implemented, it must be able to reproduce the original message exatly. Even the --cut here 01204737473829 -- parts. That's a very important design criteria.
Please not Sleepycat. If you are so sure that a generic database backend will be better than what Evolution's got, at least have the sense to use MySQL or PostgreSQL.
Sleepycat, gdbm, ndbm are just libs containing database routines. Do you really think Evolution implements it's own indexing functions, just because it's fun to write them? I bet Evulution also uses eighter standard Berkly db, gdbm or Sleepycat.
I concur in that it would be a nightmare having to set up a RDBMS just for using a Mail-client. Although it would be justified for a big-scale POP-Toaster or similar.
I know that my post didn't make it clear enough. I also like the simplicity of mbox and Maildir, because you can use it for huge mailservers down to minimalistic MUAs. Both have shortcomings though, so what I want is something simular, simplistic, but with optional compression and a standard way to add features like indexes, IMAP-flags, etc.
Ironically I thought about something like that: Writing a lib which wraps the libc-open etc. calls and provide leagacy programs with an mbox or Maildir view of the mail-store.
MTAs and MUAs could than access the mail-store via native api or in the traditional way, only needing some LD_PRELOAD flag.
(i submitted the story)
Look at who sponsered the research. Isn't it the same guys within the Bush administration who profite from continious exploitation of natural resources?
Isn't it just another atempt to justify Bush's decision "to do nothing that could endanger the groth of our economy"?
Does any independent source actually think this could work?
I just wonder in how many exciting ways this whole new shit is going to be exploited. Interactions between Javascript, Flash-bytecode, embeded html (?), IFRames, flash in xml in javascript in html in flash (?),....
Internet Explorer is being closed due to an unknown exception in kernel32.dll......
As Linux, the BSDs and other free operating systems must be similarily affected, why can't a few of the major distributions or the FSF join together and risk suing. Perhaps this time Be's Gassée would be willing to speak up in trial.
They don't need the government. Think DVD and regional-codes. I call this a precedent where a couple of companies teamed up to form a cartel, which turned out so powerful that they sucessfully implemented stronger restrictions to the medium DVD than the law would enable them to do. And don't think it cannot happen again.
Caracalla
sorry, I was short on time yesterday when writing the above:
The link to the Reiser FS paper is:
http://www.reiserfs.org/whitepaper.html
Edgar
The guy who wrote Reiser FS has quite some things to say on this very subject: What he says (in short) is:
.) Databases shouldn't exist. They exist only because System-Programmers didn't listen to Application-Programmer's needs.
.) Those Application-Programmers implemented their own solution to the problem: Databases.
That's why he started Reiser FS. It is today the ideal filesystem to store little bits of information ( 100 bytes) which you normally store in Databases, because you don't want millions of small files lying around.
(I know this is a simplified view, read his own arguments for a broader discussion.)
... but for the next 3 years or so you will defnitly be fine with CAT5. Gig-Ethernet works over CAT5, there is even a standard for Fibre-Channel over CAT5. Nics with fibre-connectors still are expensive.
In 10 years however you will probably wish that you had spent the few extra bugs to put in multimode fibres.
Installing the two might be a compromise.
Edgar
Edgar
This is not about Steganography! Your statement about Steganography is right of course, but this is different.
1) Each of the to partitions ("aspects") in the 1GB space ("extent") is actually 1GB big. It is just that the data in all aspects together cannot exceed 1GB.
2) You could actually have 5 aspects (1 for letters to you mum, 2 for emails, 3 for xxx-pics, 4 for faked ultrasecret data and 5 for real ultrasecret data). You could give the court passwords for 1-4 and there will be no way to prove that there is a 5th aspect. That is the deniability.
The court could suspect it because data in partitions 1-4 only accounts for 850MB of one 1 Gb. But there is no way for them to know for sure.
Neighter can you prove that there is no other aspect, neighter can they that there is.
3) All the aspects _are_ writable. Each one on is own as well as any combination of them. The problem is, when writing to any subset you will of course destroy those not in the particular subset.
4) In other words, when you "forget" a password for any of the aspects, the effects are the same as if this aspect never was there. (If the underlieing cryptosystem is secure.)
Edgar
In a strict sense its true that the post is "Informative". As informative as if he wrote that Slashdot is usually green, that Rodger Rabbit is a Cartoon and that Bin Laden is evil.
Did I already mention that Bin Laden is evil?
--
Live on earth may be expensive, but it includes an annual free trip around the sun!
Why don't you send the bill to Microsoft? After all it's their software which sucks.
So what do we have in the meantime: Creditcards and not-anonymous e-cash systems. All those systems involve traceability of money transfers. But society simply doesn't want to give up the anonymity of a simple Dollar-Bill or the soon-to-be Euro or whatever.
Think about Illegal Financing of Political Parties, avoiding Taxation (extremly common in Europe because you simply just can't afford to build your 1000 sqare-meters house the official way), Prostitution, Pimps, Drugs. Or think about the husband who doesn't want his sex-shop bill to show up on the family-credit-card.
There are just to many people who have vital interrests in Anonymity.
Edgar