Here's a thought - just use a hard-RAM based database. Either make a big ramdisk and put your database out there (see my Journal from a few months back, ramdisk throughput is pretty damn fast from the local machine, given certain constraints, and random access writing is hella fast), or use a database that runs entirely in memory (think Derby, aka Cloudscape that comes with WebSphere Application Developer.)
When you got your data, save it out to the hard drive.
Granted it helps to have a box with a ton of memory in it, but they are out there now, almost affordable. If you are collecting more than 4G of data in one session, well YMMV - but 4G is a LOT of data, perhaps consider your approach.
While we are on the subject, let me make the following recommendations:
1. When your app creates a new log file, put in a header. In the header put the following information : retention period, logging level, full application path and name, where the source code to this application is located, who (at the time the code was last changed) can accurately interpret the contents of this log file, the original intent / purpose of the log file, the time / date stamp of the creation of the file. If you can look at the header of any log file and understand all that you are half way there.
2. If your log file is for debugging purposes, put in a line with ProgramName::SourceCodeFile.ext::RoutineName() and include any parameters passed in when the routine was called. If you know the parameters most of the time you can recreate the problem and step through the code by hand.
3. You can put whatever you want in if there is an error, but for God's sake don't suppress the Java dump (stack trace) that comes at the end. Nothing worse than having absolutely nothing to work with when debugging a crash.
4. Consider having the app create a new file on a per-day basis. It doesn't take any more room (don't forget the header as it is a new file) and you don't have to kill the application to delete the old log files. Keeps the log file sizes manageable.
Actually I have found many, many times that a camera or a tape recorder (or digital audio recorder) is a wonderful peace-making device.
Set the tape player on your desk out in the open and just leave it there. Next time someone comes in and says something totally nasty (ie, not paying the last paycheck, or bad-mouthing you, or whatever) just point to the recorder and say "that's on." It doesn't have to be on, but if you manage to time it right even better.
Amazing piece of attitude adjustment, someone knowing that whatever they say or do is on tape.
I once walked into the county court records office once and as the worker-bee walked up (I think I interrupted her game of Solitare on her computer, she didn't look happy) I popped a flash camera up and snapped off a picture. When she asked what that was all about, I explained that I was going to send her picture to the Mayor describing how helpful she had been. And she was very helpful, go figure.
Look 14 seconds into the film - is that a marble bust in the middle of the picture back against the wall - and the head is tilted back? If this doesn't have Batman written all over it, I don't know what does. All they need now is to put a pole in the middle so you can slide down.
Just put up two professional servers, fill em with some nudie-pix and post the links to FARK at the same time. That's about the most intense benchmark known to man.
Hey - totally off topic, but that IS a great book. If you find an online version that is unedited (more importantly, unannotated) - please share. Reading the annotated version is tantamount to trying to eat a nice dinner and having the phone ring every 10 minutes.
As for the Linux vs Windows performance debate - sounds like a paid shill that intentionally tweaked the comparison so Linux would lose - but at least they are being honest about it.
I'm giving: 3:1 odds that the thing pulls an Apollo 1 (January 27, 1967) and does the best Roman Candle impression we have seen in a while right there on the deck. 4:1 odds it pulls a Challenger and does the best air-burst impression we have seen in a while, just as it hits upper atmosphere. 5:1 odds it actually gets into orbit and lasts well over a day before crashing into another satellite, destroying both. 6:1 odds it goes into orbit for a few weeks before going SkyLab on us, doing the best 'expensive man-made meteor' impression we have seen in a while 11:1 odds it makes it into stable orbit and dies up there
Had my chemistry lab instructor in college do the demo with potassium. Read the instructions, and it went something like this: Hmmm, it says 'wrap a small piece in aluminum foil that has had two small holes poked in it, then drop in a bucket of water and step back.' 'Screw that.' (Scooped a piece about one cubic centimenter in size out of the jar (potassium metal under some sort of oil) and then with everybody standing close by paying attention plop'ed it into the water. Two seconds of small bubbles before it exploded in the most amazing purple flash I have ever seen, left burn marks on his lab coat.
Never attribute to malice that which is easily explained by ignorance. Hanlon's Razor, I believe. Maybe I'm ignorant about F/OSS, but I admit it and I'm being open minded about it.
I see quite a few posts in this thread about free (as in cost) software, software with a zero dollar cost - that drives my question (whether right or wrong.)
As for free is in freedom (libre) software, hell I support that (did a presentation today on an application I coded in Java in Eclipse, deployed onto a SuSE 9 server.) My choice. Funny thing is that my company paid about $3,000 for the software (WSAD 5.1.2, box of SuSE plus annual maintenance, etc.)
I'm not against the Libre software movement, but I genuinely believe that the 'gratis' software movement is screwing things up for all of us.
That said - if FOSS isn't about 'gratis' software, maybe we need to start bitchslapping the folks pushing it that direction. If it is about 'gratis' software - someone needs to explain it better than 'well it has been explained ad nausium' or I am going to continue to think it is worse than stupid (it's destructive to software development as a profession.)
VINCENT So if you're quitting the life,what'll you do?
JULES That's what I've been sitting here contemplating. First, I'm gonna deliver this case to Marsellus. Then, basically, I'm gonna write free code.
VINCENT What do you mean, write free code?
JULES You know, like Caine in "KUNG FU." Just walk from town to town, meet people, get in adventures, and give away code I write.
VINCENT How long do you intend to write free code?
JULES Until God puts me where he want me to be.
VINCENT What if he never does?
JULES If it takes forever, I'll wait forever.
VINCENT So you decided to be a bum?
JULES I'll just be Jules, Vincent -- no more, no less.
VINCENT No Jules, you're gonna be like those pieces of shit out there who beg for change. They walk around like a bunch of fuckin' zombies, they sleep in garbage bins, they eat what I throw away, and dogs piss on 'em. They got a word for 'em, they're called bums. And without charging for software, meaning no residence, or legal tender, that's what you're gonna be -- a fuckin' bum!
I really hate to be the one stick in the mud that doesn't come in here raving about how great the GPL is... but I honestly think it is the stupidest thing I have ever heard in my LIFE.
As a professional software developer (employed one at that, maybe a dozen or so years plus a four year degree (well 5, I played a lot in school)) that wants to earn enough money as a professional to have at least a comfortable lifestyle... I look at this 'free software' movement and think WTF?
Who's bright idea was it that all software development should go totally uncompensated?
There are coders, there are QA teams, and their are the guys doing implementations and support - and if all a guy does is code (and code well!), how the hell is he supposed to pay rent, bills, educational costs, food, clothes, etc if the software is all supposed to be free?
The second issue I have with the 'free software' movement is that some of the business managers hear that mantra one too many times and interpret it as 'we don't need to pay developers as much because software is supposed to be free' and then they go abroad looking for ~free (ie offshore) developers - all of a sudden developers become unemployed, but they are still carrying on about how 'software should be free!!!' How much do you think a business manager is going to want to spend on infrastructure and support staff for a free package? Already had that conversation : answer = nothing.
I have yet to have someone explain to me in a calm, rational manner how 'free software' does anything besides destroy the livelyhood of every software engineer in America. Until I can comprehend the underlying economics of how 'free software' = American software engineers making a comfortable living, I am going to continue to believe that 'free software' is the stupidest idea I have ever heard. Yes, Linus is a millionaire. Good for him, that's one - but that doesn't scale very well (or at least it hasn't in the real world, yet to date.)
Freedom of choice is one thing. A good thing and one that I support / encourage. Free as in 'cost zero dollars' is about as stupid and evil as... well as stupid and evil as Bill Gates*
If anybody has a lucid overview as to why zero dollar cost for software is a great idea in the big scheme of employed software developers in the US - I'm eager to read it. We all can't live in our parent's basement for the rest of our lives.
* (Had to throw that one just to screw up all the MS basher flamers that are sure to ensue.)
Even quantum physics, although theoretically 'random', is generally predictable and reliably recreatable for a large T distribution over time. If you want truly unpredictable, unrecreatable, random numbers - let my wife balance your checkbook.
Know what would be funny? If McNealy takes Sun private, then waits a year and goes fucking IPO on us. It would be the next coming of Christ - Sun, inventor of Java and profitable (hint : wait until Sun has a profitable quarter to do this) for a tech company... and then they could pull that stunt Google pulled where they did a Dutch auction of stock prices... they would make so much money, and maybe even start a new revolution, the tech boom second generation!
A sort routine that is theoretically O(n log n) - or even O(n) if we are dreaming... is spiffy until you realize that Java is slow as fuck moving around massive objects, creating and destroying temporary variables on the heap and then pausing for a few seconds to do garbage collection.
Code the same routine in a language that lets you inline some assembly and physically (manually) assign the data in and out of CPU registers and you will see a whole new world of fast.
Note - I wasn't comparing good algo in a weak environment to crap algo in a good environment - I was comparing the same mathematically correct algo, applied in a 4thGL vs coded on the bare metal.
Yea... if only there was a way to make a PC running Windows serve up those files over Ethernet to other computers, perhaps by letting those other computers map to a share or something.
'Professional Excel Development'... that's just wrong. Sort of like 'Heterosexual Man-Man Love' - each of the words is familiar, but strung together like that makes it wrong on so many levels.
How did you type that ø character? That's an idea, just do my passwords in high-bit ASCII characters - I could even write them down on a post-it note on my monitor, not like anybody could figure it out...
Slashdot Mirror
Here's a thought - just use a hard-RAM based database.
Either make a big ramdisk and put your database out there (see my Journal from a few months back, ramdisk throughput is pretty damn fast from the local machine, given certain constraints, and random access writing is hella fast), or use a database that runs entirely in memory (think Derby, aka Cloudscape that comes with WebSphere Application Developer.)
When you got your data, save it out to the hard drive.
Granted it helps to have a box with a ton of memory in it, but they are out there now, almost affordable. If you are collecting more than 4G of data in one session, well YMMV - but 4G is a LOT of data, perhaps consider your approach.
While we are on the subject, let me make the following recommendations :
1. When your app creates a new log file, put in a header. In the header put the following information : retention period, logging level, full application path and name, where the source code to this application is located, who (at the time the code was last changed) can accurately interpret the contents of this log file, the original intent / purpose of the log file, the time / date stamp of the creation of the file. If you can look at the header of any log file and understand all that you are half way there.
2. If your log file is for debugging purposes, put in a line with ProgramName::SourceCodeFile.ext::RoutineName() and include any parameters passed in when the routine was called. If you know the parameters most of the time you can recreate the problem and step through the code by hand.
3. You can put whatever you want in if there is an error, but for God's sake don't suppress the Java dump (stack trace) that comes at the end. Nothing worse than having absolutely nothing to work with when debugging a crash.
4. Consider having the app create a new file on a per-day basis. It doesn't take any more room (don't forget the header as it is a new file) and you don't have to kill the application to delete the old log files. Keeps the log file sizes manageable.
Actually I have found many, many times that a camera or a tape recorder (or digital audio recorder) is a wonderful peace-making device.
Set the tape player on your desk out in the open and just leave it there. Next time someone comes in and says something totally nasty (ie, not paying the last paycheck, or bad-mouthing you, or whatever) just point to the recorder and say "that's on." It doesn't have to be on, but if you manage to time it right even better.
Amazing piece of attitude adjustment, someone knowing that whatever they say or do is on tape.
I once walked into the county court records office once and as the worker-bee walked up (I think I interrupted her game of Solitare on her computer, she didn't look happy) I popped a flash camera up and snapped off a picture. When she asked what that was all about, I explained that I was going to send her picture to the Mayor describing how helpful she had been. And she was very helpful, go figure.
Look 14 seconds into the film - is that a marble bust in the middle of the picture back against the wall - and the head is tilted back? If this doesn't have Batman written all over it, I don't know what does. All they need now is to put a pole in the middle so you can slide down.
It could have been Microsoft powered ... because MS sucks pretty hard.
Just put up two professional servers, fill em with some nudie-pix and post the links to FARK at the same time. That's about the most intense benchmark known to man.
Hey - totally off topic, but that IS a great book. If you find an online version that is unedited (more importantly, unannotated) - please share. Reading the annotated version is tantamount to trying to eat a nice dinner and having the phone ring every 10 minutes.
As for the Linux vs Windows performance debate - sounds like a paid shill that intentionally tweaked the comparison so Linux would lose - but at least they are being honest about it.
No, if you are in a meeting futzing with the keypad on your phone, you pretty much look like an asshole.
That's ATARI porn for you.
You think I'm kidding, but the old timers know what I'm talking about.
Heck you guys are optimistic.
I'm giving:
3:1 odds that the thing pulls an Apollo 1 (January 27, 1967) and does the best Roman Candle impression we have seen in a while right there on the deck.
4:1 odds it pulls a Challenger and does the best air-burst impression we have seen in a while, just as it hits upper atmosphere.
5:1 odds it actually gets into orbit and lasts well over a day before crashing into another satellite, destroying both.
6:1 odds it goes into orbit for a few weeks before going SkyLab on us, doing the best 'expensive man-made meteor' impression we have seen in a while
11:1 odds it makes it into stable orbit and dies up there
Had my chemistry lab instructor in college do the demo with potassium. :
Read the instructions, and it went something like this
Hmmm, it says 'wrap a small piece in aluminum foil that has had two small holes poked in it, then drop in a bucket of water and step back.'
'Screw that.'
(Scooped a piece about one cubic centimenter in size out of the jar (potassium metal under some sort of oil) and then with everybody standing close by paying attention plop'ed it into the water.
Two seconds of small bubbles before it exploded in the most amazing purple flash I have ever seen, left burn marks on his lab coat.
It was amazing. God I miss college.
Never attribute to malice that which is easily explained by ignorance. Hanlon's Razor, I believe. Maybe I'm ignorant about F/OSS, but I admit it and I'm being open minded about it.
I see quite a few posts in this thread about free (as in cost) software, software with a zero dollar cost - that drives my question (whether right or wrong.)
As for free is in freedom (libre) software, hell I support that (did a presentation today on an application I coded in Java in Eclipse, deployed onto a SuSE 9 server.) My choice. Funny thing is that my company paid about $3,000 for the software (WSAD 5.1.2, box of SuSE plus annual maintenance, etc.)
I'm not against the Libre software movement, but I genuinely believe that the 'gratis' software movement is screwing things up for all of us.
That said - if FOSS isn't about 'gratis' software, maybe we need to start bitchslapping the folks pushing it that direction. If it is about 'gratis' software - someone needs to explain it better than 'well it has been explained ad nausium' or I am going to continue to think it is worse than stupid (it's destructive to software development as a profession.)
VINCENT
So if you're quitting the life,what'll you do?
JULES
That's what I've been sitting here contemplating. First, I'm gonna deliver this case to Marsellus. Then, basically, I'm gonna write free code.
VINCENT
What do you mean, write free code?
JULES
You know, like Caine in "KUNG FU." Just walk from town to town, meet people, get in adventures, and give away code I write.
VINCENT
How long do you intend to write free code?
JULES
Until God puts me where he want me to be.
VINCENT
What if he never does?
JULES
If it takes forever, I'll wait forever.
VINCENT
So you decided to be a bum?
JULES
I'll just be Jules, Vincent -- no more, no less.
VINCENT
No Jules, you're gonna be like those pieces of shit out there who beg for change. They walk around like a bunch of fuckin' zombies, they sleep in garbage bins, they eat what I throw away, and dogs piss on 'em. They got a word for 'em, they're called bums. And without charging for software, meaning no residence, or legal tender, that's what you're gonna be -- a fuckin' bum!
I've read Sun Tzu's Art of War a few times, and I don't remember reading the words 'being civilly present in court' in that most hallowed tome.
... check. ... check. ... check. ... check. ... no find.
Burn to the ground
Archers
Flanking with superior firepower
Spys
Being civilly present in court
Sorry man, you are going to have to do better than that if you are going to overcome evil.
Shit, that's funny.
Actually the last time I paid for support was the last time I paid for software.
Hint : not for a long time.
I really hate to be the one stick in the mud that doesn't come in here raving about how great the GPL is ... but I honestly think it is the stupidest thing I have ever heard in my LIFE.
... I look at this 'free software' movement and think WTF?
... well as stupid and evil as Bill Gates*
As a professional software developer (employed one at that, maybe a dozen or so years plus a four year degree (well 5, I played a lot in school)) that wants to earn enough money as a professional to have at least a comfortable lifestyle
Who's bright idea was it that all software development should go totally uncompensated?
There are coders, there are QA teams, and their are the guys doing implementations and support - and if all a guy does is code (and code well!), how the hell is he supposed to pay rent, bills, educational costs, food, clothes, etc if the software is all supposed to be free?
The second issue I have with the 'free software' movement is that some of the business managers hear that mantra one too many times and interpret it as 'we don't need to pay developers as much because software is supposed to be free' and then they go abroad looking for ~free (ie offshore) developers - all of a sudden developers become unemployed, but they are still carrying on about how 'software should be free!!!' How much do you think a business manager is going to want to spend on infrastructure and support staff for a free package? Already had that conversation : answer = nothing.
I have yet to have someone explain to me in a calm, rational manner how 'free software' does anything besides destroy the livelyhood of every software engineer in America. Until I can comprehend the underlying economics of how 'free software' = American software engineers making a comfortable living, I am going to continue to believe that 'free software' is the stupidest idea I have ever heard. Yes, Linus is a millionaire. Good for him, that's one - but that doesn't scale very well (or at least it hasn't in the real world, yet to date.)
Freedom of choice is one thing. A good thing and one that I support / encourage.
Free as in 'cost zero dollars' is about as stupid and evil as
If anybody has a lucid overview as to why zero dollar cost for software is a great idea in the big scheme of employed software developers in the US - I'm eager to read it. We all can't live in our parent's basement for the rest of our lives.
* (Had to throw that one just to screw up all the MS basher flamers that are sure to ensue.)
Even quantum physics, although theoretically 'random', is generally predictable and reliably recreatable for a large T distribution over time.
If you want truly unpredictable, unrecreatable, random numbers - let my wife balance your checkbook.
Know what would be funny? ... and then they could pull that stunt Google pulled where they did a Dutch auction of stock prices ... they would make so much money, and maybe even start a new revolution, the tech boom second generation!
If McNealy takes Sun private, then waits a year and goes fucking IPO on us.
It would be the next coming of Christ - Sun, inventor of Java and profitable (hint : wait until Sun has a profitable quarter to do this) for a tech company
Be happy you aren't a gynocologist.
Right up until you are working on the bare metal.
... is spiffy until you realize that Java is slow as fuck moving around massive objects, creating and destroying temporary variables on the heap and then pausing for a few seconds to do garbage collection.
A sort routine that is theoretically O(n log n) - or even O(n) if we are dreaming
Code the same routine in a language that lets you inline some assembly and physically (manually) assign the data in and out of CPU registers and you will see a whole new world of fast.
Note - I wasn't comparing good algo in a weak environment to crap algo in a good environment - I was comparing the same mathematically correct algo, applied in a 4thGL vs coded on the bare metal.
Yea ... if only there was a way to make a PC running Windows serve up those files over Ethernet to other computers, perhaps by letting those other computers map to a share or something.
Naw, that would never work.
you shouldn't be able to write professional level applications using Visual Basic either.
Ding! We have a winner!
'Professional Excel Development' ... that's just wrong.
Sort of like 'Heterosexual Man-Man Love' - each of the words is familiar, but strung together like that makes it wrong on so many levels.
that builds grammatical sentences by taking a valid syntax and plugging in random verbs, nouns and adjectives in the right places.
Or I could just send you the documentation we got back with the last project we outsourced to India.
How did you type that ø character? ...
That's an idea, just do my passwords in high-bit ASCII characters - I could even write them down on a post-it note on my monitor, not like anybody could figure it out