Domain: ddj.com
Stories and comments across the archive that link to ddj.com.
Comments · 361
-
Re:The grey lady should look before leaping
The story you linked at the Guardian is by Scott Rosenberg, an actual Salon employee who was close to the Salon Premium effort. As a corollary, here is an older article I commissioned from Scott for Web Techniques magazine in 2001, when the Salon paywall was just going up. It includes some technical details, but mostly it's just interesting to see the bookends of the initial optimism and later disillusionment with the effort.
P.S. Interesting that Scott's lede was, "The Web's great free-for-all is coming to a sudden, sharp end." Not so sudden, I guess, but perhaps even more sharp than expected.
-
All on one page version
Link to the Print version--article on one page with no advertising since I haven't seen that posted yet.
-
Re:ads and do it yourselfers
Advertisers wouldn't pay for ads if they didn't work, at least not for long.
That's why newspapers are going under. Because advertisers are deciding that online advertising is more effective than print advertising. Cheaper too.
More effective, ie they work at least better than other forms of ads. They're also cheaper because websites are cheaper. In the print industry, newspapers and magazines revenue from subscriptions barely paid distribution expenses before, profit was made in advertising. However subscriptions have been falling and adverting rates are determined in part my the subscription base.
Personally I'd pay a little more to subscribe to publications I like. And a little more for web access as well without ads. One of my favorite print magazines was "Byte magazine". It did have an online edition but when I just went to it I was redirected to "Dr Dobb's Journal". Another one I loved went out of print but still has a web presence, Transitions Abroad. I had renewed my subscription 2 months before the print edition was stopped but they had arranged with a Canadian magazine like it to take over delivery of the print edition, "Verge Magazine".
Falcon
-
Weekend vs Work and Java
It might be that Python and Ruby are less used in business because of different strengths of those languages - and maybe in business companies don't want to adopt yet another programming language.
Apart from that I did intensive evaluation about 1-2 years ago and I found Java being the best overall option for most common development. Sure, such a conclusion may depend on the branch you are working - I am an allround developer and not a driver programmer or pure web developer.
And as I wrote in other places also - I think just looking at the core language features is barely enough when thinking of what language to use for your next project. There are 3rd party libraries, IDE (features), ntegration options, other development tools and so on that should be taken into consideration also.
Furthermore I think companies will not switch to different languages just because it seems to be more fun. The requirements for a home or private (mostly smaller) project are different than for business projects. Nowadays in IT you get so easily a mess of "legacy systems" to maintain that I guess companies don't like the adoption of many different programming languages also. - Talking of Python and Ruby I think there will be a rising of Scala then also soon. This somehow leads me to http://www.ddj.com/architect/184413224 and I recall a lot of times when me, friends or others thought of another programming languages being THE new language that can be used to solve even the hunger in the third world.
It seems that more people give Python and Ruby a try which is interesting to know but does not necessarily means that those languages are the languages of the future.
-
Re:But will they be useful without concepts?
Unfortunately, without concepts, many of the templates that would make features like those REALLY powerful aren't implementable due to silly things like the compiler insisting upon being able to instantiate member functions that don't make sense for a class and won't be used, just because there isn't a means to tell the compiler "and if this member doesn't make sense, just don't instantiate it, and throw an error IF AND ONLY IF somebody tries to use it." (and yes, I know about SFINAE, but that gets REALLY UGLY to do).
That is your opinion.
However, both Bjarne Stroustrup and Herb Sutter disagree.Some selective quoting (from the links above):
Stroustrup: "Concepts" as currently defined are too hard to use and will lead to disuse of "concepts," possibly disuse of templates, and possibly to lack of adoption of C++0x.
Sutter: Concepts would be great, but for most users, the presence or absence of concepts will make no difference to their experience with C++0x except for quality of error messages.
Sutter: Concepts are almost entirely about getting better error messages.
Sutter: We won't have a de-conceptized working draft for the post-meeting mailing, two weeks after the meeting, but should have one soon after that.
I agree that concepts are useful and important but they are not the end-all be-all you make them to be.
-
Re:How about a REAL C++ feature....
How can it be safe when it's not even correct?
-
Re:Not a bombshell
-
Different Paths Same Destination
Bob Dylan is not famous for knowing English grammar and spelling. He is famous for what he does with them. Teach him Chinese, and he can most likely make amazing songs in Chinese as well.
There is a difference between an intimate understanding and just translating the words. Dylan was great because he mastered the English language to not just follow the rules, but bend and break them - "When you ain't got nothing, you got nothing to lose."
Every programming language has it's own little quirks that must be navigated. High level concepts may translate across languages, however, the most elegant implementation is often derived from experience.
Different skill sets can be applied to the same problem and come up with different solutions that work. There is no perfect curriculum for a particular job, success ultimately depends on how well one adapts their toolset to meet the needs of a particular task. -
Re:Good
"1) High performance VM"
.NET is quickly gaining on Sun JVM. So not for long."2) Code that does what it says without hidden conversions, text substitutions, and macros."
So?
"3) Other languages that are actually useful like Scala and Clojure."
And
.NET has F# (officially supported now!), Ruby, IronPython and so on. Microsoft has even made changes to the CLR to better support custom languages, while Scala and Clojure have to use horrible hacks to implement closures and continuations."And LINQ? Why are you doing database and 'data sources' queries in something like C#? Use python or something like that. You need that super-fast JIT "cc -O0" speed to print out customer numbers and such?"
YES! I need a super-fast JIT to print out customer numbers. Have you EVER written a sizable business application? Do you propose to move all data queries to a separate application and then what, use text files for data exchange?
Also, LINQ is typesafe. You can check at compile-time that your queries are correct and IDE can (and does!) use type information for rich autocomplete and inspections.
Besides, there's also Parallel LINQ ( http://www.ddj.com/windows/212700663 ) which allows me to write parallel queries with nice logical syntax.
-
Re:The times they are changing...
Seems that he's still at RAD:
-
Re:Bull
Actually that's not true
http://www.ddj.com/database/184416281
To maintain the integrity of the registry hives, the operating system performs transaction logging. For all hives except the system hive, when a change is made to that registry hive, the change is written to the corresponding log file. When that log file is flushed to the disk, the first sector of the log file is marked to indicate that a registry change is in progress. After the flush, the changes are written to the actual hive also. If the transaction succeeds, the hive and log file are marked to indicate the transaction successfully completed. If the machine should crash during the transaction, on the next boot, the operating system would detect an incomplete transaction (log file still marked) and perform a recovery by restoring the previous values stored in the log file. Transaction logging is also performed on the system file, but the operating system uses a slightly different process. The
.alt files are complete backup copies of the corresponding hive file. The operating system only keeps a complete backup copy of the crucial system hive, so the only .alt file youâ(TM)ll find is the system.alt.I.e. the registry has its own transaction system. It has to be this way because at least as far back as XP it was possible to boot off a FAT32 drive, so it was impossible to only use NTFS transactions.
-
bad ideas
The open source movement is exactly what should be funded.
That's a bad idea. Government shouldn't be funding either open source or proprietary software. They should let people keep the money they work to earn and let them decide who they want to support.
I had an argument with a microsoftie a while ago, who was convinced that open source was destroying the software industry. I countered that all it was doing was creating a rich infrastructure on top of which other industry could be built.
Next tyme maybe you can remind him, or her, that open source software has been around longer than Microsoft. Members of the Tech Model Railroad Club at MIT were writing open source programs back in the 1950s and '60s.
Falcon
-
Re:Require pay and benefits parity
They may be a little cheaper, but they can't be that much cheaper or they couldn't afford to live here.
Actually that's not true. H1B's often accept a dramatically reduced standard of living, with the intent to make as much money as they can for the duration of their visa, then take it home with them. Sometimes this is still much better than the conditions they left at home.
H1B's make on average 23% less than their citizen counterparts.
The citizen counterparts tend to want things like kids, gadgets, a nice car, good clothes, a house, etc. The H1B tends to have 1 week's worth of clothes, takes public transportation or has an inexpensive used car, lives here alone (maybe with family back home), lives in a one-room studio apartment, and maintains a minimum of optional expenses.
-
Re:...because H1Bs are forms, not people
On a side note, I thought the US was build on people coming from bad situations to live the American dream, you guys sure have changed your mindset lately.
If that was the case, there wouldn't be any complaints, because then they wouldn't be getting paid less. It's the fact that they're only temp workers that get paid up to 23 percent less than Americans in the same positions that cheeses people off. Level playing field--fine. Unfair playing field where management lies about not being able to find qualified personnel and then turns around and pays substantially lower salaries--not good.
And, to continue, the "lower pay" part is illegal. I have actually looked into it (from a business standpoint) before, and, as a business owner, I have to basically sign an affidavit that I will pay the same rate to the foreign worker as a similarly qualified US worker, and I have to swear that I can't find anyone in the US to fill the job.
My wife came over on an H1A originally as a nurse, and it was the same story. The nursing home was getting Filipino nurses to come over so they could pay them shit wages that Americans wouldn't even consider. The Filipinos also put up with *anything* because they could be sent back to the Philippines with a signature from the director.
I have a friend who's in the same shitty position now as a computer programmer - the company illegally didn't pay him for 8 months while he was "benched", but he won't sue them because he wants to be in the US so bad. They owe him tens of thousands of dollars. He ended up finding the current contract that he's working by himself, but still is working through the agency that dicked him over. Looking them up on the internet, he's not the only one they've done this to.
Anyway, it's a mess, but if the US simply enforced the law, particularly the "equal pay" part, the problem would go away.
-
Re:...because H1Bs are forms, not people
On a side note, I thought the US was build on people coming from bad situations to live the American dream, you guys sure have changed your mindset lately.
If that was the case, there wouldn't be any complaints, because then they wouldn't be getting paid less. It's the fact that they're only temp workers that get paid up to 23 percent less than Americans in the same positions that cheeses people off. Level playing field--fine. Unfair playing field where management lies about not being able to find qualified personnel and then turns around and pays substantially lower salaries--not good.
-
Re:multicore dev is fun... much like prison rape!
I believe the biggest problem with multi-core development is a lack of maturity in the tools and libraries available. Taking advantage of multiple cores requires a lot of thread management code, which is great for highly optimized applications but deters run-of-the-mill business and user app developers. There was a recent opinion piece in Dr Dobbs discussing the benefits a concurrency platforms I found interesting. The article is clearly promoting the author's company (Clik Arts), but I agree with his argument that the complexities of multi-core development need to be handled in a framework and not applications.
-
Re:It's great that there's money for this stuff...
-
Re:Limited use
The link to the actual Dr. Dobb's article is way more readable.
-
Re:PHP?
-
Logo, LISP, Scala, F#, Erlang, and Haskell
Functional programming is making a comeback- it's going to be to the 2010s what OOPs was to the 1990s. I'd suggest these, and make recursive loops a major sticking point. Dr Dobbs has a nice article on why these functional languages make excellent methods for taking advantage of multi-core processors.
-
Mono 2.0 -- Interview with Miguel de Icaza [audio]
Dr. Dobb's http://www.ddj.com/ talks with Miguel de Icaza about Mono 2.0 in this audio interview.
-
Youngster!I watched Microsoft lie and cheat to cripple DR-DOS in the marketplace in the early '90s, as documented in Dr. Dobb's Journal . I hated them for such behavior, and for distorting the field in which I earned a living.
Where I come from, there's a saying: "If you associate with brigands and you are silent, you are a brigand yourself."
-
Re:Are processes really that much heavier?
yeah thats old school linux threads. New school is different. When in doubt, seek dobbs
-
Re:SoftRAM
SoftRAM/SoftRAM95 was non-diluted snakeoil.
RAM Doubler for the Mac was a real and non-snakeoil product. That was mostly due to the "classic" Mac OS doing a horrible job of managing memory, so the potential for improvement was huge.
There were similar products available for win3.x and Win9x, some of which at least tried to do what they advertised. The performance benefit of using them (at least the RAM-compression) was pretty much non-existent though.
Linux / OS X / WinNT already has quite decent virtual memory management so the potential performance benefit is limited.
-
beats AARD by 2 years
That beats AARD by two years.
-
Re:Is it just me?
The preprocessor algorithm I described in the Dr. Dobb's article is the one I used for parsing the code of this study. A strange preprocessor construct in the Linux kernel caused the macro-expansion algorithm I used previously to fail.
-
Re:Some people can handle threads...
Sure you can wrap up the c handlers in c++ objects which then have automagic destructors, but that is a lot of extra layers and code just to get around using a goto.
Let me preface this by saying I don't think gotos are always "bad", that said, I don't think they're the best solution for this example when writing in C++. Extra layers, yes (but mostly negligible), but extra code? This is why we have libraries. To wit:
def foo
library_thing1
ScopeGuard u1 = MakeGuard(boost::bind(undo_library_thing1)); ...
stuff that can go horribly bad ...
end
I apologize for the lack of indentation, I just couldn't get it right.
You don't even need to create a new class if you don't want to, and the setup and teardown are right next to each other, so you don't have to scan down to the bottom of the function to be sure you cleaned up after yourself (goto or no).
- scope guard: http://www.ddj.com/cpp/184403758
- boost::bind: http://www.boost.org/doc/libs/1_35_0/libs/bind/bind.html
-
H.26x: what is it?
Nice article on what H.26x is at ddj: http://www.ddj.com/201203492
I had no idea how tangled the standards were... ugh. -
Supercomputer definition
"A supercomputer is a device for converting a CPU-bound problem into an I/O bound problem."
- Ken Batcher
http://www.ddj.com/184404102 -
Re:IBM Open-sourcing Experience
There's been some question as to Eclipse being derived from IBM's VisualAge products. FWIW, Wikipedia does a decent job at spelling it out but isn't too linear in its description. Object Technology International( OTI ) made a product in the early 90s which was based on Smalltalk and had a click-n-drag GUI application builder which included building apps with both GUI elements and non-visual/GUI elements. VisualAge was based on this product and IBM bought this company. After IBM built VisualAge for C++ ontop of the Smalltalk product, they then built VisualAge for Java and added Java capabilities to the Smalltalk runtime of VisualAge. Then, trimmed down and pure Java version of VisualAge was created and called VisualAge for Embedded and targeted Java on embedded devices. This, Java-based product was what eventually became Eclipse.
Here are some links to follow up on the synopsis above:
http://en.wikipedia.org/wiki/Object_Technology_International
BTW, I saw this product demo'ed on OS/2 in the early 90s and was pleased to then see the same kind of visual builder in the VisualAge product. Unfortunately, if you build it, they don't always come when they only see through Windows.
http://en.wikipedia.org/wiki/VisualAge
Getting back to SOM -
This is a review of VisualAge for C++(Windows) v3.5 with SOM v2.x and COM linkage:
http://www.ddj.com/windows/184415551
LoB -
Re:It's a race
See, the funny thing is that the changes going into Python 3 are fixes for much of what people have complained about in Python 2.x and prior.
Moreover, every step of the way they've built translators to move code from 2.x compatibility to 3.0 compatible -- and it'll catch when it can't translate the code and tell you as much. It seems pretty slick from everything I've seen. In many cases the fixes are ones you could easily do yourself in seconds with a good text editor. This will be a minor speed-bump for most users
For more info, check out the recent Doctor Dobb's Journal interview (audio) with David Goodger -- it's about PyCon 2008, but it also covers Python 3 as well.
-
software innovation
it looks like we're just going to roll over and leave whatever innovation is left in the software field to the Chinese and the Indians.
And Brazil and other countries of South America.
Falcon -
Goodbye to VB
They've shifted scripting paradigms before. Word used to have its own dialect of Basic, and Excel originally did all its scripting with those @ functions.
What's really painful is not the death of VBA as such. What's painful is Microsoft's decision to do away with the whole Visual Basic paradigm without providing anything to replace it.
What do I mean by by "Visual Basic paradigm"? I don't mean the (very sucky) language. I mean the integration of the language to all those COM interfaces that permeate Microsoftland, including Office. These COM interfaces are all part of object frameworks, but because they're interfaces rather than objects, you don't have to master the object framework in order to use them
When MS got bored with COM and decided to move on to .NET, they neglected to replicate this functionality. They did provide a .NET version of VB, but it's just another OO language. So VB.NET programmers have to master the .NET object framework. Might as well learn C# and be done with it.
I'm a user of OneNote, which was the first MS Office application to be released without a builtin Visual Basic engine. You can automate OneNote, but the learning curve's much steeper than it would be with VBA. I've never found time to assault it.
Even though I've always despised the pre-.NET dialect of Visual Basic, I find I'm missing it terribly. -
Re:Would we know the difference?
What confuses me is that we still haven't adopted languages and practices that naturally scale to multiple cores
Sadly, I do still subscribe to what was once a great magazine, but now is a large advertisement with an article or two, DDJ. The had a fairly interesting article regarding what would be necessary for your comment to be true in smart compilers.
http://ddj.com/dept/architect/202401072 -
Development Methodology
Adopt a development methodology that defines what features will be developed and what features WILL NOT be developed. Now you have a place to put all those "great ideas" people have while you're still making the base product. You can also keep them in mind while making other modules, so you know you can make them later. This eliminates scope creep. It may not seem like a piece of paper will work, but as a society we all know the power of forms. Act like it's not your choice, that someone in management is making you use the methodology. No one will want to contradict anyone else so they will just accept it. And when it works, and you come in on-time and on-budget it will become part of the corporate culture.
Your deliverables (above) will be a part of your project charter. You will also include stuff like: a list of stakeholders, RISKS AND ASSUMPTIONS (such as a deadline not being met, etc.), testing, and of course "Success", which will be a list of metrics that define a successful product (ie: it can generate payroll checks, it can print report A, etc). Then, take your project charter (look it up on google) and put a bunch of lines on the bottom for you, the team and the management and the key users to sign off on. Do not start work until it gets signed off. Then make a copy for yourself and file the original with the project documentation. Work and complete all the features to be developed. At the end, take the project charter and make sure everything is fulfilled, then give it to the "customer" and have them sign off again for completion (after you demo the software).
Usually scope creep means poor project management, and as a developer you can't expect anyone else to do it. Just do it, you will be very thankful you did. Also, if it's a short project, try something called Quick-Kill project management. Large projects need a better methodology. I use one I made up that's based on the quick kill and some microsoft stuff, with some unix version control stuff, and oracle business process analysis stuff... Over time you will develop your own methodology and become a star senior programmer making $300,000 a year.
Have fun -
Re:Firefox Memory Leaks, C++ Memory Leaks
I'm the lead for a group of over 20 engineers and programmers that produces public safety communications equipment and have direct experience with memory leaks large scale C++ projects AND Firefox. The system we sell includes embedded devices, a Linux/Oracle/Apache server for management, and management terminals. Management and monitoring is done via web browser, either on Windows or Linux. We officially support both Firefox and IE, and both leak memory. We strictly control what is installed on the management PCs - no browsers extensions, no ActiveX components, no Firefox Ad-ons. Typical usage is to open the browser and monitor one page continuously for weeks until an alarm is shown.
Let me be clear, both Firefox and IE leak memory so badly that even on management PC with 2GB of RAM we have to require the end user to restart the browser every week. We are monitoring FF3 and looking forward to reduced memory leakage. In the referenced article it discusses reducing memory fragmentation. OK, that's a worthy goal but first fix the memory leaks. Memory fragmentation and memory leaks are related, but different beasties. A memory leak almost always results in fragmentation, but fragmentation can happen simply from an unfortunate memory allocation/deallocation pattern.
Regarding C++ and memory leaks: over 2 1/2 years we've worked on the embedded code, which is pure C++, we have hunted exactly one memory leak. And that leak turned out to be from the OS. We use Boost smart pointers, RAII, exceptions, and exception safe code. We have no trouble with leaks or fragmentation, despite a fairly high turnover rate and a customer base that would quickly notice memory leaks requiring reboot of the embedded devices. -
Re:Supercomputer is a term that changes
Sony's estimate of 1 teraflop per unit is more than a little bit optimistic, and also counts all the operations performed by the graphics system, which the average researcher won't have access to. For a slightly more realistic estimate of what 8 ps3's running linux can do, I'd point you towards http://www.terrasoftsolutions.com/store/index.php?submit=software&submitimg%5Bhardware%5D%5Bsolutions%5D=1, who sell turnkey ps3 clusters. They are claiming that 8 units together break the theoretical teraflop mark, which seems more realistic to me. While that isn't among the fastest computers in the world by any stretch, it's still solidly in the realm of what I'd call a supercomputer.
The cell is a fantastic piece of equipment - Dr. Dobb's has what I think is an excellent analysis of the kinds of performance benefits that it offers at http://www.ddj.com/hpc-high-performance-computing/197801624. I'm currently running one at home in a PS3 (for neural networks that drive an AIBO - I love Sony's tendency to dump hugely expensive hardware at mass-market prices), and I have every intention of picking up more used ones over the coming months to cluster together as the networks continue to grow. Even all by it's lonesome with code that's far from optimized, the one I have is running about 10 times faster than my main desktop for roughly equivalent computations.
(Note that your mileage may vary - I just happen to like playing with systems that parallelize really well) -
Technology implicationReading his post I saw a few quotes that raised a red flag for me such as:
"But at every step, it seemed our needs clashed with Rails' preferences. (Like trying to turn a train into a boat. It's do-able with a lot of glue. But it's damn hard. And certainly makes you ask why you're really doing this.)"
and
#2 - OUR ENTIRE COMPANY'S STUFF WAS IN PHP: DON'T UNDERESTIMATE INTEGRATION
and
By the old plan (ditching all PHP and doing it all in Rails), there was going to be this One Big Day, where our entire Intranet, Storefront, Members' Login Area, and dozens of cron shell scripts were ALL going to have to change..
Speaking of tastes: tiny but important thing : I love SQL. I dream in queries. I think in tables. I was always fighting against Rails and its migrations hiding my beloved SQL from me.
What these quotes say really is that this guy doesn't know about "Technology Mapping 101". Here is what I wrote about technology mapping* about 2 years ago (incidentally that's about the same time this guy started his Ruby adventure :) )
"Technology mapping can have a significant impact on the ability to actually implement the architecture. The wrong mapping can make adhering the architectural guidelines very cumbersome and sometimes nearly impossible."
Every technology or framework has its own architecture. This architecture poses constrains and makes certain things easy (like using the ActiveRecord pattern in Rails) and certain things harder (like not using O/R Mapping ) so, for instance, on the case mentioned above a better technology mapping might have been RBatis (iBatis for Ruby/Rails) which lets you map SQL statements to objects. It is important to note (in Rails case) that one of the core tenets for Rails is preferring convention of configuration when you don't do that you have to work hard(er) - as you are working against the framework
Another problem with the technology mapping here was his point #2. It is a pity he only saw it in after the fact. It can be justifiable to switch everything but you've got to allow this change to occur iteratively. While I generally dislike the software architecture = building architecture metaphor, using it here does make sense: building software for an existing business (vs. greenfield development) is like building a new intersection. You have got to think about building all the detours that would allow the traffic (business) to continue to operate, sure it might run slower in the intermediate phase but it can't stop altogether.
So again, Once you have an architecture that fits your business, take a look at the technology options you have. try to choose the best fit. Whatever you choose - take a look at the implications of that technology and think about the tradeoffs you may find that you either have to adjust your architecture or change the technology altogether - if you don't you might find you waited 2 years of development effort or even more..
* Technology Mapping is one of the steps of a set of activities I identified as needed to make sure your have a solid architecture. The activities goes by the acronym SPAMMED and you can read about them more in this article and/or this presentation
copied from my blog
-
Re:Sure
Pfft.. Real men code websites in Java and ASP. Scalability and performance are for pussies. My server to chugs at 10 hits/minute and it likes it.
Funny, but many banks and stock markets run their trading on the Java platform. At JavaOne 2007 for instance, Anna Ewing, Nasdaq CIO talked about handling hundreds of thousands transactions per second with millisecond deadline requirements. -
Re:So they couldn'tI can't speak directly from experience optimizing for each, but I'd say you've got your facts a bit screwy on bus bandwidth there from what I've both read and observed. Have a read of the Dr. Dobb's article It may be tricky, but the performance gains are worth the effort.
They comment on both the difficulty of fully optimizing for the cell as well as the immense capabilities of it:In this article, we present strategies we've used to make a Breadth-First Search on graphs as fast as possible on the Cell, reaching a performance that's 22 times higher than Intel's Woodcrest, comparable to a 256-processor BlueGene/L supercomputer--and all this with just with a single Cell processor! Some techniques (loop unrolling, function inlining, SIMDization) are familiar; others (bulk synchronous parallelization, DMA traffic scheduling, overlapping of computation and transfers) are less so.
They accomplish some impressive optimizations and comment on how many more should be available if the effort were put in. -
Re:Are relations obsolete?It depends on the application. I have done quite a bit of work with Jean-Claude Wippler metakit database. It is essentially a column based store with some clever underpinnings to make it relational as well. We mainly used it to track molecules and assay results coming off of our plate-readers. The nice thing is that this streaming data can be saved to disk and used in our LIMS system. What it does really well is scan and sort columns, the throughput is really quite amazing. It gets pretty slow for relational queries, but has the benefit that results of queries are persistant in the sense you can chain them together in a relational algebra fashion.
When I was getting interested in this, I also came across kdb a relational database based on the K programming language. kdb is also a column based store with one huge benefit, the interpreter that analyzes the columns,K fits inside a level 1 cache. The throughput of kdb is immense which is why it is mainly used to track and analyze financial data and other streaming and real-time data. It used to be you could try it out for free, I'm not sure what the state is currently, but I suggest at least giving it a try, it's not cheap but it certainly changed my view of databases.
Just my two cents. -
It's a couple links deep...
Read this interview with Dr Dobbs:
A floating-point matrix multiply using the new SSE5 extensions is 30 percent faster than a similar algorithm
I believe this helps gaming and other simulations.
Discrete Cosine Transformations (DCT), which are a basic building block for encoders, get a 20 percent performance improvement
And then we have the "holy shit" moment:
For example, the Advanced Encryption Standard (AES) algorithm gets a factor of 5 performance improvement by using the new SSE5 extension
If I get one of these CPUs, I'll almost certainly be encrypting my hard drives. It was already fast enough, but now...
As for existing OS support, it looks promising:
We're also working closely with the tool community to enable developer adoption -- PGI is on board, updates to the GCC compiler will be available this week, and AMD Code Analyst Performance Analyzer, AMD Performance Library, AMD Core Math Library and AMD SimNow (system emulator) are all updated with SSE5 support.
So, if you're really curious, you can download SimNow and emulate an SSE5 CPU, try to boot your favorite OS... even though they say they're not planning to ship the silicon for another two years. Given that they say the GCC patches will be out in a week, I imagine two years is plenty of time to get everything rock solid on the software end.
-
Re:Map and reduce?
Ironically, C++ has just this in the form of one standard algorithm at least. Hey, I know, it's pathetic, but technically there is at least *one* that I know of. From the article here:
http://www.ddj.com/cpp/184403769
"-for_each applies the operation in a definite order, namely starting at the beginning and proceeding to the end of the input range; no such guarantee is given for transform.
-The operation supplied to transform must not have any side effects; no such restriction is imposed on the operation supplied to for_each."
Of course, theory and practice are different things, so I guess it doesn't. Heh. And don't get me started on valarray... -
OpenBSD
Most (if not all) OpenBSD code I have dealt with gives me a warm tingly feeling inside. It isn't C++, but they have their reasons.
-
Paper
-
Drupal on Dr. Dobbs
It should be noted that the first chapter of the book is available on Dr. Dobb's and gives a good overview of what Drupal is and how it works.
-
Terracotta - a Clustered JVM
Terracotta (an open-source solution to clustering at the JVM level) approaches this problem by recognizing that multi-threaded programs can become multi-JVM programs. Here's a great article just posted on Dr. Dobbs -> http://www.ddj.com/dept/java/199703478 with more details and links about Terracotta.
Disclaimer: I work for Terracotta. -
Some details of the computation size.
From http://www.ddj.com/blog/portal/archives/2007/05/w
o rld_record_fo.html Using the sieve program developed at the University of Bonn, NTT, EPFL, and the University of Bonn respectively provided 84.1 percent, 8.3 percent, and 7.6 percent of the calculation resources, and the calculation amount equivalent to 95 years of operation on a 3-Ghz Pentium D. PC clusters at NTT and EPFL, consisting of 110 and 36 PCs, respectively, were run in parallel for more than two months for the calculations. The results were 47 non-trivial solutions of the simultaneous equations defined by an approximate 70,000,000 x 70,000,000 large sparse linear matrix. -
Re:Sad or Telling?Actually, no, this was a deliberate test set up to tag people running a competitors product and error screen. What made it look really really bad, I think, was that the AARD Detection Code was intensionally hidden: "Microsoft's encrypted code to disguise what it had done was unravelled by Geof Chappell in England, who commented at the time that "the only error' is that the user is running Windows with someone else's version of DOS."
Dr Dobbs has pretty pictures of setup.exe being detected. They describe the workings of the code on page 4:The first step in discovering why the error message appeared under DR DOS but not MS-DOS was to examine the relevant WIN.COM code. However, the WIN.COM code that produced this message turned out to be XOR encrypted, self-modifying, and deliberately obfuscated--all in an apparent attempt to thwart disassembly.
The code also tries to defeat attempts for a debugger to step through it. For example, Figure 2 shows a code fragment in which the INT 1 single-step interrupt is pointed at invalid code (the two bytes FFh FFh), which disables DEBUG. The same is done with INT 2 (nonmaskable interrupt) and INT 3 (debug breakpoint). However, since modern debuggers (I used Nu-Mega's Soft-ICE) run the debugger and debuggee in separate address spaces, the AARD code's revectoring of INTs 1-3 has no affect on the Soft-ICE debugger. In any case, these attempts to throw examination off-track are in themselves revealing.
But what does "country information" like the DOS default upper case-map have to do with a network redirector? Why does a piece of Windows care whether this mapper is located in the DOS data segment? And why should it care whether the first FCB-SFT is located on a paragraph boundary? What kind of "errors" are these, anyway?
These are all reasonable questions. In fact, the address of the default upper case-map has nothing to do with the network redirector, and no other part of Windows cares about what particular form is taken by DOS's default case-map or first FCB-SFT pointers. The AARD code has no relation to the actual purpose of the five otherwise-unrelated programs into which it has been dropped. It appears to be a wholly arbitrary test, a gratuitous gatekeeper seemingly with no purpose other than to smoke out non-Microsoft versions of DOS, tagging them with an appropriately vague "error" message. -
Re:Sad or Telling?Actually, no, this was a deliberate test set up to tag people running a competitors product and error screen. What made it look really really bad, I think, was that the AARD Detection Code was intensionally hidden: "Microsoft's encrypted code to disguise what it had done was unravelled by Geof Chappell in England, who commented at the time that "the only error' is that the user is running Windows with someone else's version of DOS."
Dr Dobbs has pretty pictures of setup.exe being detected. They describe the workings of the code on page 4:The first step in discovering why the error message appeared under DR DOS but not MS-DOS was to examine the relevant WIN.COM code. However, the WIN.COM code that produced this message turned out to be XOR encrypted, self-modifying, and deliberately obfuscated--all in an apparent attempt to thwart disassembly.
The code also tries to defeat attempts for a debugger to step through it. For example, Figure 2 shows a code fragment in which the INT 1 single-step interrupt is pointed at invalid code (the two bytes FFh FFh), which disables DEBUG. The same is done with INT 2 (nonmaskable interrupt) and INT 3 (debug breakpoint). However, since modern debuggers (I used Nu-Mega's Soft-ICE) run the debugger and debuggee in separate address spaces, the AARD code's revectoring of INTs 1-3 has no affect on the Soft-ICE debugger. In any case, these attempts to throw examination off-track are in themselves revealing.
But what does "country information" like the DOS default upper case-map have to do with a network redirector? Why does a piece of Windows care whether this mapper is located in the DOS data segment? And why should it care whether the first FCB-SFT is located on a paragraph boundary? What kind of "errors" are these, anyway?
These are all reasonable questions. In fact, the address of the default upper case-map has nothing to do with the network redirector, and no other part of Windows cares about what particular form is taken by DOS's default case-map or first FCB-SFT pointers. The AARD code has no relation to the actual purpose of the five otherwise-unrelated programs into which it has been dropped. It appears to be a wholly arbitrary test, a gratuitous gatekeeper seemingly with no purpose other than to smoke out non-Microsoft versions of DOS, tagging them with an appropriately vague "error" message.