I have seen these kinds of situations happen a lot (I'm a statistician who works on computationally-intensive physical science applications), and the best solution I have seen was a BerkeleyDB setup. One group I work with had a very, very large number of ASCII data files (order of 10-100 million) in a directory tree. One of their researchers consolidated them to a BerkeleyDB, which greatly improved data management and access [...] I think the general idea of a key-value store that lets you keep your data in the original structure would work well.
A file system *is* a key-value store.
I suspect those 100,000,000 files were in fact tiny pieces of data which didn't make sense to access using normal tools (from ls to MS Word). That the conversion worked out for *you* doesn't mean that it would be useful to convert *every* set of files into a BerkeleyDB.
Especially not sets of (say) 500 files, 10GB each.
I completely agree. If you have a lot of small datasets that break ls and such (as was the case in my situation), BerkeleyDB provided a great solution. If you have a smaller set of very large files, a different solution is needed (perhaps just the file system with some kind of automated indexing).
I have seen these kinds of situations happen a lot (I'm a statistician who works on computationally-intensive physical science applications), and the best solution I have seen was a BerkeleyDB setup. One group I work with had a very, very large number of ASCII data files (order of 10-100 million) in a directory tree. One of their researchers consolidated them to a BerkeleyDB, which greatly improved data management and access. CouchDB or the like could also work, but I think the general idea of a key-value store that lets you keep your data in the original structure would work well.
I recently developed a small package of statistical tools & made it available under lesser GPL. I made the decision to open-source it for several reasons. First, I wanted to make it easily available to other researchers wrestling with the same problem I was. Second, I wanted to see if anyone could take what I had done and extend it into a better set of tools. Third, having it freely available, code and all, helps to get my name out there and build my reputation. There are plenty of reasons to put out applications without making money from it.
I would have to mention Numb3rs. Granted, they have occasional screw-ups, but 'Backscatter' was a pretty good episode for tech realism. I guess it helps having a group of guys from Caltech & Wolfram looking over your scripts:).
I have quite a bit of experience in this regard. I had 3 PDAs in HS (Zire 71, Tungsten C, and HP 4350), a Treo 600 during my first 2 yrs of college, and now I have a Moto Q. Some advice on PDAs and smartphones for college:
QWERTY is your friend. When going from class-to-class, there is no way you can use the handwriting input fast enough to get assignments down. Physical keyboards are best. Windows Mobile has the best on-screen.
Don't worry about platform compatibility with a WM PDA. PocketMac (http://www.pocketmac.net) is a great piece of software (used it form the HP 4350). It's actually more reliable than ActiveSync, in my experience.
Personally, for storing class and assignment info, I prefer WM. Newer Palms with better calendar software may be better, but this is my experience with Treo 600.
Consider a smartphone. It's a lot easier, especially for contacts. For example, I keep all of my professors' office hours in notes attached to their contact entries. Makes it really easy to access the info.
Get a sturdy case. When you're running from one end of campus to the other, you will drop your PDA and/or phone. My failure to do so is one reason I now have a Q (who knew 5 drops onto pavement was the Treo's limit?). I like aluminum ones; then again, I'm a klutz.
If you have questions, feel free to contact me. Always glad to help a fellow student.
The short answer is kind of. I have a Cruzer Titanium, and it can withstand being dropped onto hard surfaces - up to a point. On mine, the 2 pieces of the cases are in perfect condition, but the glue used to bind them came apart. The internals are still ok, but I had to replace it with a Cruzer Mini (little aluminum one). That's held up much better, between my klutziness and going through the laundry around 10 times.
I think that the most interesting piece of the story has been missed. Ohloh is looking to create a paid service that will assess proprietary software for organizations. Combined with their open-source project DB, I see them building a set of tools that would enable IT managers to assess the relative costs & risks of different solutions far more easily. I would be looking for two developments to make this a reality. First, Ohloh would need to create an estimator of risk for each project based on the available data (a 'risk score'). Second, they would need a way to estimate the cost of customizing existing solutions (open-source & commercial). If they get both of these, they could provide a standardized, risk-adjusted measure of the opportunity cost associated with each development/deployment option. Really hope they are heading this way.
Slashdot Mirror
A file system *is* a key-value store.
I suspect those 100,000,000 files were in fact tiny pieces of data which didn't make sense to access using normal tools (from ls to MS Word). That the conversion worked out for *you* doesn't mean that it would be useful to convert *every* set of files into a BerkeleyDB. Especially not sets of (say) 500 files, 10GB each.
I completely agree. If you have a lot of small datasets that break ls and such (as was the case in my situation), BerkeleyDB provided a great solution. If you have a smaller set of very large files, a different solution is needed (perhaps just the file system with some kind of automated indexing).
I have seen these kinds of situations happen a lot (I'm a statistician who works on computationally-intensive physical science applications), and the best solution I have seen was a BerkeleyDB setup. One group I work with had a very, very large number of ASCII data files (order of 10-100 million) in a directory tree. One of their researchers consolidated them to a BerkeleyDB, which greatly improved data management and access. CouchDB or the like could also work, but I think the general idea of a key-value store that lets you keep your data in the original structure would work well.
I recently developed a small package of statistical tools & made it available under lesser GPL. I made the decision to open-source it for several reasons. First, I wanted to make it easily available to other researchers wrestling with the same problem I was. Second, I wanted to see if anyone could take what I had done and extend it into a better set of tools. Third, having it freely available, code and all, helps to get my name out there and build my reputation. There are plenty of reasons to put out applications without making money from it.
I would have to mention Numb3rs. Granted, they have occasional screw-ups, but 'Backscatter' was a pretty good episode for tech realism. I guess it helps having a group of guys from Caltech & Wolfram looking over your scripts :).
On the $600 PS3's: wait a few weeks. Pissed off girlfriends work wonders; it's how I got my 360 cheap :).
- QWERTY is your friend. When going from class-to-class, there is no way you can use the handwriting input fast enough to get assignments down. Physical keyboards are best. Windows Mobile has the best on-screen.
- Don't worry about platform compatibility with a WM PDA. PocketMac (http://www.pocketmac.net) is a great piece of software (used it form the HP 4350). It's actually more reliable than ActiveSync, in my experience.
- Personally, for storing class and assignment info, I prefer WM. Newer Palms with better calendar software may be better, but this is my experience with Treo 600.
- Consider a smartphone. It's a lot easier, especially for contacts. For example, I keep all of my professors' office hours in notes attached to their contact entries. Makes it really easy to access the info.
- Get a sturdy case. When you're running from one end of campus to the other, you will drop your PDA and/or phone. My failure to do so is one reason I now have a Q (who knew 5 drops onto pavement was the Treo's limit?). I like aluminum ones; then again, I'm a klutz.
If you have questions, feel free to contact me. Always glad to help a fellow student.The short answer is kind of. I have a Cruzer Titanium, and it can withstand being dropped onto hard surfaces - up to a point. On mine, the 2 pieces of the cases are in perfect condition, but the glue used to bind them came apart. The internals are still ok, but I had to replace it with a Cruzer Mini (little aluminum one). That's held up much better, between my klutziness and going through the laundry around 10 times.
I think that the most interesting piece of the story has been missed. Ohloh is looking to create a paid service that will assess proprietary software for organizations. Combined with their open-source project DB, I see them building a set of tools that would enable IT managers to assess the relative costs & risks of different solutions far more easily. I would be looking for two developments to make this a reality. First, Ohloh would need to create an estimator of risk for each project based on the available data (a 'risk score'). Second, they would need a way to estimate the cost of customizing existing solutions (open-source & commercial). If they get both of these, they could provide a standardized, risk-adjusted measure of the opportunity cost associated with each development/deployment option. Really hope they are heading this way.