Simple. You should not store lots of binary data in SQL. I don't think that MS SQL compact edition can handle 10 GB of emails on standard desktop machine.
Your SQL queries won't be able to do full text, body or header search anyway. You need MIME decoders to handle all email formats.
Optimal way of working with emails is to store message envelopes in DB and keep raw messages as files in hashed directory structure.
This is essentially all true. You can store large binary blobs in databases, but all database developers will tell you file systems are better for that.
I've actually written an email archiving system that queried emails from email accounts. You store the relevant bits of the header in the database (to/from/subject/date/etc). Decode all of the MIME attachments, hash them, and store the attachment in the files system using the hash for the file name (saves space/complexity when an attachment is sent multiple times, like signature images). Save all of the MIME metadata in the database. Take the email's plain text and store it in the database for searching (optionally stripping unneeded characters).
This keeps the database relatively small and agile, which is what you want/need for responsive email programs. A desktop with a LAMP stack can provide instant access to tens of thousands of emails that would crush Outlook. With just the text in the database, performing a text search like "DataCorp invoice" is easy. You don't ever search within a binary blob, so having it internal to the database is pointless.
Optional: Hash and store the entire email header (sometimes very useful to look at). Hash and store the entire original email (good/necessary for some environments, but the undecoded MIME attachments take a lot more space than the decoded ones). Attempt to compress every file after hashing it (can save a ton of space with little overhead [gzip] for some file types, and it is only a bit flag in the database to indicate if the file is compressed or not). Scrub attachments like PDF/ODF for text and add to the database for searching. Dynamically reconstruct the emails as a.eml using the various stored bits.
Honestly, it was so easy and fast that I was left scratching my head over how Outlook/PST still performs so poorly.
2GB limit went away in Outlook 2003 though, so now I can finally take a really big pst. When I'm done with that, I'll probably dump my logs.
While the 2GB limit went away, anyone wanting a reasonably good chance of maintaining data integrity isn't going to exceed that limit. PST files were never known for their resilience to error, and that 2GB limit just made things worse.
Also, it's annoying that just having a PST file opened by Outlook causes it to change, which causes it to need to be backed up again by differentials (and consumes space with shadow copies).
I love my HTC Hero but boy is it slow at times. And I'm not just talking about waiting for an app to load, there are times when the entire device just decides to freeze up for 2-3 seconds while queuing input.
Sounds like my iPhone 3G. From the initial 2 firmware to the current 3, there are constant multi-second pauses. Heck, just unlocking it can sometimes take close to 10 seconds. And yes, I've had my phone replaced so I know it's not the hardware. And I reboot it regularly. It's a great device, but a mediocre phone.
Not having pedals or a steering column to deal with in a crash gives the engineers lots of scope to make cars safer.
The reason for the steering column is the force feedback from steering system, and the ability to steer without power. If you're going to make a fly by wire joystick, then it should be pretty easy to make a steering wheel fly by wire too. Max depth would be a couple of inches.
A rather bizarre website, but it's not that interesting to get signatures on a petition. People say all sorts of things to get signatures, so I would guess many of the people that signed did so under a question like, "Do you want to keep the state from taking away marriage? Sign here." The real question is, "how many people have this view" instead of "how many people could they get to sign the petition."
On interesting thing on the front page of the website is
SB 5688 is a 110 page document which includes the phrase "marriage shall apply equally to state registered domestic partnerships" 180 times.
I can't verify if it is true, but it seems as if their issue is the semantics of using the word "marriage". Wouldn't it be easier to drop that verbiage from the bill and pass it? I'm fairly certain that Slashdot is not the source of grand political ideas, so I would think others have thought of it and rejected it for some reason.
If you look at the other links in that list, they all follow the form of an actual statement of stance. The "Same-Gender Attraction" link is completely different, and is clearly (IMO) not a statement of stance, but rather some helpful informed opinions.
In either respect, before you go changing your opinion, you should contact the Public Relations department responsible for that website and verify that page is an official stance of the LDS church. They are open 8:00 AM to 4:30 PM MST at 1-801-240-1000.
Official statements from the LDS church are not given interview style. I read it simply as an interview that was transcribed to help give clarification, but not to state official doctrine.
From the intro, it does not appear that the comments are in any way officially LDS backed comments, rather it is a transcription of a friendly internal interview, and the specific speaker appears to be a few levels from the top. At best, it appears to be the informed opinion of someone familiar with the inner working of the LDS church and gospel. At worst, it is the misled opinion of a man.
I found the transcription by searching Google for '"civil union" site:lds.org' and that was the only relevant result. That would lead me to believe that the LDS church has no official stance on civil unions.
I still believe your comments stand, though as a question. Would allowing a contract called "civil union" that is functionally identical to "marriage" dilute the meaning and importance of "marriage"?
Ah, I did miss that, although it doesn't really change anything. (I'm really curious how the name posting lands as it is a privacy issue I've long wondered about, and I am in government.)
What I really wondered about was how many people is this? Are there 50 people fighting the civil unions, or 50,000? I'm curious just how many people the 'religious right' is that your referring to.
PUBLIC AFFAIRS: Would you extend the same argument against same-gender marriage to civil unions or some kind of benefits short of marriage?
ELDER WICKMAN: One way to think of marriage is as a bundle of rights associated with what it means for two people to be married. What the First Presidency has done is express its support of marriage and for that bundle of rights belonging to a man and a woman. The First Presidency hasn’t expressed itself concerning any specific right. It really doesn’t matter what you call it. If you have some legally sanctioned relationship with the bundle of legal rights traditionally belonging to marriage and governing authority has slapped a label on it, whether it is civil union or domestic partnership or whatever label it’s given, it is nonetheless tantamount to marriage. That is something to which our doctrine simply requires us to speak out and say, “That is not right. That’s not appropriate.
It appears that both you and HeronBladeMaster would accept this solution. However, he believes this would satisfy the majority of 'religious organizations' while you believe they would remain unsatisfied.
I guess the next step would be to see how many people would accept this compromise. I would think the best way to find out would be to first find the largest organizations that support preventing "gay marriage", and second to contact them for an opinion on the solution. Would there be a better way?
Would you be opposed to asking some people at church on Sunday how they would feel about "civil unions" being granted the same rights as "marriages" and see what their reaction is?
Are there examples that would show that a major portion of others working against "marriage" for same sex couples are doing so for semantic reasons? I'm curious because this is the first time I've seen it presented this way, and I'm dubious that adding identical rights to civil unions as to marriages would resolve most of the conflict.
Although, if the process were processing a list, then a faster CPU would finish the list more quickly, which would be contrary to my stated observations. Or are you implying that the process processes a list repeatedly for a specific amount of time?
Do you seriously thing these civil unions as available today are in any way equivalent to a marriage? It is marriage lite. They should all be treated equally under the law. I don't care what word you use to describe it, be that civil union, or marriage, but it should apply to ALL, not just one class of citizens.
HeronBladeMaster said:
Most religious people couldn't care less if gays can get civil unions (I know I don't), it's just the word "marriage" that they don't think should be applied to such unions.... Until you can show me a legal right that can be obtained via marriage but not civil union, then you're merely complaining about semantics.
So it seems that you've provided examples of how a civil union is less functional than marriage under both federal and state laws. You've also stated you don't care what word is used to describe the couple's union contract. However, HeronBladeMaster has said that 'religious people' simply don't want the term "marriage" used for same sex couples, but are otherwise uncaring.
It sounds to me like the solution would be to have federal legislation that grants the same privileges and responsibilities to "civil unions" that "marriage" possesses. At the state level I am guessing that this would either require a federal mandate of the same air, or a state by state change of laws. This would ensure equal rights across various unions, while ensuring the word "marriage" does not have its traditional meaning altered.
No; I'm implying that religious people merely want to preserve the existing definition of the word "marriage". Furthermore, the government already dictates who can and cannot use it (you have to obtain a government marriage license to be called "married").
So, tell me if I have this strait. You're saying that:
The word "marriage" has historically applied only to a legal union between a man and women.
And that if the government also applies the term "marriage" to a legal union between a man and a women, then this will dilute the meaning of the word "marriage".
'Religious people' believe the word "marriage" to be sacred enough to their religion that allowing the government to influence the historical meaning is unacceptable.
If same sex couples were allowed to enter in to the same contract as the "marriage" contract, with all of the same rights and responsibilities, but they called it a "civil union" (or 'Willy Wonka's Wacky Coupling', etc) instead of "marriage" then the 'religious people' would have no issue.
I'll admit that I've never actually heard it put quite that way, so I'm curious if I'm following you correctly. If so, it's very interesting and I'm going to have to think on it.
For reference, here is the first section of the word "marriage" from Merriam-Webster's Unabridged Dictionary:
Etymology: Middle English mariage, from Middle French, from marier to marry + -age -- more at MARRY 1 a : the state of being united to a person of the opposite sex as husband or wife b : the mutual relation of husband and wife : WEDLOCK c : the institution whereby men and women are joined in a special kind of social and legal dependence for the purpose of founding and maintaining a family -- see MONOGAMY, POLYGAMY
And as the Unabridged dictionary simply points the user the the Collegiate version, here is the first section of the term "civil union" from Merriam-Webster's Collegiate Dictionary:
Date: 1992 : the legal status that ensures to same-sex couples specified rights and responsibilities of married couples
If someone has access to the Oxford dictionary, I'm sure that would be much more detailed.
WDS is also painfully slow and an insane resource hog. Unfortunately you need it to search emails in Outlook 2007 cached mode. I've also not seen it reliably query WDS installed on a server. And it will do random things like suddenly attempt to index Outlook Express for all users that have logged into a machine, even though none of the users have used OE, and policies have been set up to expressly not index OE.
In a single user environment, the difference between Windows Desktop Search and Google Desktop Search is night and day. GDS indexes faster, uses less resources, indexes more things (websites you visit in Firefox?), and keeps cached versions of the indexed text. So with GDS you can see the text to a document you deleted 2 years ago, but was indexed then.
I've never used Google's Enterprise Search box, but if it works anywhere as well as their desktop version, I'd recommend that.
I'm a big fan of Search Everything and use it at both home and at work to search through hundreds of thousands of files with no delay. However, it should be noted that it is not necessarily appropriate for larger environments where not everyone should know about all other files. Search Everything reads the NTFS metadata from the drive, but it ignores the file permissions. So if an HR rep has a file (Sally Jones' official reprimand.doc) in her private share, then it will turn up in the results for other users.
CPUs idle when not doing anything. Processes though can do all sorts of crazy things like, polling something constantly, which keeps the CPU loaded. Why someone would do that is a mystery, but that people do it is not.
I would love to hear that you work for Austin Community College.
For those that don't know, Austin is home to the largest university in the US, The University of Texas, which has around 60,000 students. Austin Community College has around 40,000 and is currently on track to surpass the size of UT by 2015.
Of the folks I talk to that take classes at both, most tend to prefer the classes at ACC due to much smaller class sizes and interaction with the professors. The largest class I've seen at ACC is 30 students. At UT it is measured in hundreds. The funny thing is that tuition is an order of magnitude higher at UT.
So how do you get away with charging 10 times as much for classes with 10 times the students?
I think you may have missed my point completely. This is about the Windows Update process, and I was pointing out that in my experience, regardless of the speed of the CPU "they will both be active for the same amount of time". So, on a Pentium II and a Core i7, the update process will keep the CPU pegged for 5 minutes.
Obviously if you are trying to complete specific calculations, then it is better to hurry through them at high power and shut down earlier. In this case though, the process in question is not calculation limited but rather time limited.
I feel as if the conversation has drifted. The point wasn't that it would slow the computer down. The point was that it would use more Watts by keeping a CPU pegged, which would keep it at its highest speed.
I've observed this on machines that I manage, others home machines, and machines that have been freshly installed off of various install media. Trust me, this isn't an incidence of "I forgot to uncheck the 'use all of the CPU all the time for no reason' checkbox.
My observations are pretty consistent across a wide range of XP and 2003 x86/x64 systems. It is one of the SVCHOST.EXE processes that is being used by the update system, not the WAUPDATE process itself. The individual update processes themselves typically use a negligible amount of CPU time.
Slashdot Mirror
Simple. You should not store lots of binary data in SQL. I don't think that MS SQL compact edition can handle 10 GB of emails on standard desktop machine.
Your SQL queries won't be able to do full text, body or header search anyway. You need MIME decoders to handle all email formats.
Optimal way of working with emails is to store message envelopes in DB and keep raw messages as files in hashed directory structure.
This is essentially all true. You can store large binary blobs in databases, but all database developers will tell you file systems are better for that.
I've actually written an email archiving system that queried emails from email accounts. You store the relevant bits of the header in the database (to/from/subject/date/etc). Decode all of the MIME attachments, hash them, and store the attachment in the files system using the hash for the file name (saves space/complexity when an attachment is sent multiple times, like signature images). Save all of the MIME metadata in the database. Take the email's plain text and store it in the database for searching (optionally stripping unneeded characters).
This keeps the database relatively small and agile, which is what you want/need for responsive email programs. A desktop with a LAMP stack can provide instant access to tens of thousands of emails that would crush Outlook. With just the text in the database, performing a text search like "DataCorp invoice" is easy. You don't ever search within a binary blob, so having it internal to the database is pointless.
Optional: Hash and store the entire email header (sometimes very useful to look at). Hash and store the entire original email (good/necessary for some environments, but the undecoded MIME attachments take a lot more space than the decoded ones). Attempt to compress every file after hashing it (can save a ton of space with little overhead [gzip] for some file types, and it is only a bit flag in the database to indicate if the file is compressed or not). Scrub attachments like PDF/ODF for text and add to the database for searching. Dynamically reconstruct the emails as a .eml using the various stored bits.
Honestly, it was so easy and fast that I was left scratching my head over how Outlook/PST still performs so poorly.
2GB limit went away in Outlook 2003 though, so now I can finally take a really big pst. When I'm done with that, I'll probably dump my logs.
While the 2GB limit went away, anyone wanting a reasonably good chance of maintaining data integrity isn't going to exceed that limit. PST files were never known for their resilience to error, and that 2GB limit just made things worse.
Also, it's annoying that just having a PST file opened by Outlook causes it to change, which causes it to need to be backed up again by differentials (and consumes space with shadow copies).
I love my HTC Hero but boy is it slow at times. And I'm not just talking about waiting for an app to load, there are times when the entire device just decides to freeze up for 2-3 seconds while queuing input.
Sounds like my iPhone 3G. From the initial 2 firmware to the current 3, there are constant multi-second pauses. Heck, just unlocking it can sometimes take close to 10 seconds. And yes, I've had my phone replaced so I know it's not the hardware. And I reboot it regularly. It's a great device, but a mediocre phone.
Not having pedals or a steering column to deal with in a crash
gives the engineers lots of scope to make
cars safer.
The reason for the steering column is the force feedback from steering system, and the ability to steer without power. If you're going to make a fly by wire joystick, then it should be pretty easy to make a steering wheel fly by wire too. Max depth would be a couple of inches.
A rather bizarre website, but it's not that interesting to get signatures on a petition. People say all sorts of things to get signatures, so I would guess many of the people that signed did so under a question like, "Do you want to keep the state from taking away marriage? Sign here." The real question is, "how many people have this view" instead of "how many people could they get to sign the petition."
On interesting thing on the front page of the website is
SB 5688 is a 110 page document which includes the phrase "marriage shall apply equally to state registered domestic partnerships" 180 times.
I can't verify if it is true, but it seems as if their issue is the semantics of using the word "marriage". Wouldn't it be easier to drop that verbiage from the bill and pass it? I'm fairly certain that Slashdot is not the source of grand political ideas, so I would think others have thought of it and rejected it for some reason.
If you look at the other links in that list, they all follow the form of an actual statement of stance. The "Same-Gender Attraction" link is completely different, and is clearly (IMO) not a statement of stance, but rather some helpful informed opinions.
In either respect, before you go changing your opinion, you should contact the Public Relations department responsible for that website and verify that page is an official stance of the LDS church. They are open 8:00 AM to 4:30 PM MST at 1-801-240-1000.
Official statements from the LDS church are not given interview style. I read it simply as an interview that was transcribed to help give clarification, but not to state official doctrine.
For example, these statements were from high up the chain, but are now regarded largely as wrong opinion.
http://en.wikipedia.org/wiki/Blacks_and_The_Church_of_Jesus_Christ_of_Latter-day_Saints#.22Curse_of_Cain.22_and_representative_of_Satan
From the intro, it does not appear that the comments are in any way officially LDS backed comments, rather it is a transcription of a friendly internal interview, and the specific speaker appears to be a few levels from the top. At best, it appears to be the informed opinion of someone familiar with the inner working of the LDS church and gospel. At worst, it is the misled opinion of a man.
I found the transcription by searching Google for '"civil union" site:lds.org' and that was the only relevant result. That would lead me to believe that the LDS church has no official stance on civil unions.
I still believe your comments stand, though as a question. Would allowing a contract called "civil union" that is functionally identical to "marriage" dilute the meaning and importance of "marriage"?
Ah, I did miss that, although it doesn't really change anything. (I'm really curious how the name posting lands as it is a privacy issue I've long wondered about, and I am in government.)
What I really wondered about was how many people is this? Are there 50 people fighting the civil unions, or 50,000? I'm curious just how many people the 'religious right' is that your referring to.
As the stated reasons for opposition are specifically moral, I think the question would be appropriate.
This quote, while not by any means a signed and official decleration, is something to be aware of.
http://newsroom.lds.org/ldsnewsroom/eng/public-issues/same-gender-attraction
PUBLIC AFFAIRS: Would you extend the same argument against same-gender marriage to civil unions or some kind of benefits short of marriage?
ELDER WICKMAN: One way to think of marriage is as a bundle of rights associated with what it means for two people to be married. What the First Presidency has done is express its support of marriage and for that bundle of rights belonging to a man and a woman. The First Presidency hasn’t expressed itself concerning any specific right. It really doesn’t matter what you call it. If you have some legally sanctioned relationship with the bundle of legal rights traditionally belonging to marriage and governing authority has slapped a label on it, whether it is civil union or domestic partnership or whatever label it’s given, it is nonetheless tantamount to marriage. That is something to which our doctrine simply requires us to speak out and say, “That is not right. That’s not appropriate.
Is there more information about who is fighting the civil unions other than the creator(s) of the website in question?
http://slashdot.org/comments.pl?sid=1410437&cid=29813867
It appears that both you and HeronBladeMaster would accept this solution. However, he believes this would satisfy the majority of 'religious organizations' while you believe they would remain unsatisfied.
I guess the next step would be to see how many people would accept this compromise. I would think the best way to find out would be to first find the largest organizations that support preventing "gay marriage", and second to contact them for an opinion on the solution. Would there be a better way?
Would you be opposed to asking some people at church on Sunday how they would feel about "civil unions" being granted the same rights as "marriages" and see what their reaction is?
Are there examples that would show that a major portion of others working against "marriage" for same sex couples are doing so for semantic reasons? I'm curious because this is the first time I've seen it presented this way, and I'm dubious that adding identical rights to civil unions as to marriages would resolve most of the conflict.
Although, if the process were processing a list, then a faster CPU would finish the list more quickly, which would be contrary to my stated observations. Or are you implying that the process processes a list repeatedly for a specific amount of time?
Do you seriously thing these civil unions as available today are in any way equivalent to a marriage? It is marriage lite. They should all be treated equally under the law. I don't care what word you use to describe it, be that civil union, or marriage, but it should apply to ALL, not just one class of citizens.
HeronBladeMaster said:
Most religious people couldn't care less if gays can get civil unions (I know I don't), it's just the word "marriage" that they don't think should be applied to such unions.... Until you can show me a legal right that can be obtained via marriage but not civil union, then you're merely complaining about semantics.
So it seems that you've provided examples of how a civil union is less functional than marriage under both federal and state laws. You've also stated you don't care what word is used to describe the couple's union contract. However, HeronBladeMaster has said that 'religious people' simply don't want the term "marriage" used for same sex couples, but are otherwise uncaring.
It sounds to me like the solution would be to have federal legislation that grants the same privileges and responsibilities to "civil unions" that "marriage" possesses. At the state level I am guessing that this would either require a federal mandate of the same air, or a state by state change of laws. This would ensure equal rights across various unions, while ensuring the word "marriage" does not have its traditional meaning altered.
Is that correct?
No; I'm implying that religious people merely want to preserve the existing definition of the word "marriage". Furthermore, the government already dictates who can and cannot use it (you have to obtain a government marriage license to be called "married").
So, tell me if I have this strait. You're saying that:
I'll admit that I've never actually heard it put quite that way, so I'm curious if I'm following you correctly. If so, it's very interesting and I'm going to have to think on it.
For reference, here is the first section of the word "marriage" from Merriam-Webster's Unabridged Dictionary:
Etymology: Middle English mariage, from Middle French, from marier to marry + -age -- more at MARRY
1 a : the state of being united to a person of the opposite sex as husband or wife b : the mutual relation of husband and wife : WEDLOCK c : the institution whereby men and women are joined in a special kind of social and legal dependence for the purpose of founding and maintaining a family -- see MONOGAMY, POLYGAMY
And as the Unabridged dictionary simply points the user the the Collegiate version, here is the first section of the term "civil union" from Merriam-Webster's Collegiate Dictionary:
Date: 1992
: the legal status that ensures to same-sex couples specified rights and responsibilities of married couples
If someone has access to the Oxford dictionary, I'm sure that would be much more detailed.
WDS is also painfully slow and an insane resource hog. Unfortunately you need it to search emails in Outlook 2007 cached mode. I've also not seen it reliably query WDS installed on a server. And it will do random things like suddenly attempt to index Outlook Express for all users that have logged into a machine, even though none of the users have used OE, and policies have been set up to expressly not index OE.
In a single user environment, the difference between Windows Desktop Search and Google Desktop Search is night and day. GDS indexes faster, uses less resources, indexes more things (websites you visit in Firefox?), and keeps cached versions of the indexed text. So with GDS you can see the text to a document you deleted 2 years ago, but was indexed then.
I've never used Google's Enterprise Search box, but if it works anywhere as well as their desktop version, I'd recommend that.
I'm a big fan of Search Everything and use it at both home and at work to search through hundreds of thousands of files with no delay. However, it should be noted that it is not necessarily appropriate for larger environments where not everyone should know about all other files. Search Everything reads the NTFS metadata from the drive, but it ignores the file permissions. So if an HR rep has a file (Sally Jones' official reprimand.doc) in her private share, then it will turn up in the results for other users.
CPUs idle when not doing anything. Processes though can do all sorts of crazy things like, polling something constantly, which keeps the CPU loaded. Why someone would do that is a mystery, but that people do it is not.
I would love to hear that you work for Austin Community College.
For those that don't know, Austin is home to the largest university in the US, The University of Texas, which has around 60,000 students. Austin Community College has around 40,000 and is currently on track to surpass the size of UT by 2015.
Of the folks I talk to that take classes at both, most tend to prefer the classes at ACC due to much smaller class sizes and interaction with the professors. The largest class I've seen at ACC is 30 students. At UT it is measured in hundreds. The funny thing is that tuition is an order of magnitude higher at UT.
So how do you get away with charging 10 times as much for classes with 10 times the students?
I also use pfSense, and while I don't have multiple WAN connections, I am satisfied with it's other features (such as traffic shaping).
I think you may have missed my point completely. This is about the Windows Update process, and I was pointing out that in my experience, regardless of the speed of the CPU "they will both be active for the same amount of time". So, on a Pentium II and a Core i7, the update process will keep the CPU pegged for 5 minutes.
Obviously if you are trying to complete specific calculations, then it is better to hurry through them at high power and shut down earlier. In this case though, the process in question is not calculation limited but rather time limited.
I feel as if the conversation has drifted. The point wasn't that it would slow the computer down. The point was that it would use more Watts by keeping a CPU pegged, which would keep it at its highest speed.
I've observed this on machines that I manage, others home machines, and machines that have been freshly installed off of various install media. Trust me, this isn't an incidence of "I forgot to uncheck the 'use all of the CPU all the time for no reason' checkbox.
My observations are pretty consistent across a wide range of XP and 2003 x86/x64 systems. It is one of the SVCHOST.EXE processes that is being used by the update system, not the WAUPDATE process itself. The individual update processes themselves typically use a negligible amount of CPU time.