Load List Values for Improved Efficiency

An anonymous reader writes "Reduce the number of database hits and improve your Web application's efficiency when you load common shared list values only once. In this code-filled article, learn to load the values for drop-down lists when your Web application starts and then to share these loaded list values among all the users of your application."

Comments

You know what's great

[Anonymous Coward]

When telling us there's code... TELLING US WHAT LANGUAGE. It's Java.

Re:You know what's great

[Anonymous Coward]

How is this 'news for nerds'? I mean, were all pretty intelligent people on this site... Surely the tech-savvy readership of slashdot don't need to have basic, obvious computer-science 101 techniques like 'caching' explained to us - do we?

I'd like to point out that one of the problems with this approach is that the cache could become inconsistant if someone updates the database after you have populated the menu. Worth thinking about for all you 31337 h4x0rs out there...

Re:You know what's great

[eyeye]

Reams of java too, though I guess much of that is not directly involved in the caching.

In persistent perl:

{ #note extra scope
my @dropdowndata;
sub whatever{
#call sub/method unless you already have the cached version.
@dropdowndata ||= load_dropdown_data();
}
}

Re:You know what's great

[Anonymous Coward]

I don't care what language was used. It could have been VB. What matter to me is the concepts.

Should I agree with the concept, it's then up to me to implement the solution in my favorite language.

Re:You know what's great

[Hament]

yes please is it java? hament verma http://linkedin.blogspot.com/>

Re:You know what's great

[Anonymous Coward]

Thank you so much for telling me this. I would have been completely stuck if you hadn't, I had missed the java headers right at the top of the first bit of code and thought it was ZX80 assembler.

Re:You know what's great

[tomhudson]

The application loads all drop-down list items at the start of the user session and uses them for the entire session. The drop-down list items are stored in the session. In a user session, the number of database hits is constant and the database hit count increases proportionally to the number of sessions.

I'd like to point out that one of the problems with this approach is that the cache could become inconsistant if someone updates the database after you have populated the menu. Worth thinking about for all you 31337 h4x0rs out there...

Storing it in the session to fake "persistence" is lame. If its something that's not going to change over the time the person is using it, store it in a file, and include the file in the equiv. of their home dir (you DO give your users their own home dir, to reduce database accesses, don't you?). The added benefit is, if they change the values, you can store the new values in the same file, and you also have persistence between sessions. If its something that all users of that app will be using, then storing it in a common file is also more intelligent.

Right tool for the right job - not every piece of info you serve needs to be stored in a db.

Re:You know what's great

[Anonymous Coward]

Re your sig.. do you have any reliable evidence for that 100,000 figure?

Re:You know what's great

[Anonymous Coward]

While we're all savaging this poor bastard, how about a mention of his superb error-handling and logging mechanism?

try { ListValuesLoader listValues = new ListValuesLoader(); listValues.createValidValuesList(); } catch(Exception e){ System.out.println("There is a problem in getting the list values"); }

Ah, yes, good ol' standard out ...

Re:You know what's great

[ahmusch]

I respectfully disagree with your statement, although likely not the intent. Every piece of info you serve should be stored in a database, but not every piece of info you serve needs to be served from a database every single time.

Re:You know what's great

[Anonymous Coward]

Thanks, you just gave me another reason never to consider perl for anything serious. WTF does that code do? Can you explain it to me? The Java code was 100 times clearer. I'd hate to have to maintain code like the sample you posted.

Re:You know what's great

[tomhudson]

Shit - it's not even Troll Tuesday :-)

Every piece of info you serve should be stored in a database

... absolutely not. A database is not the end-all and be-all of computing, not when static or semi-static information, or information that is easily maintained as text/html.

I recently had the "opportunity" to test out a system that stored live video in a database - majorly dumb idea. We ate the money we spent on it - hardware and software - but if I had known ahead of time that's what they were doing, it would have saved us the money.

Here's one example - you have a list of accounts that 50 people can pick from, but only 2 people can update, and this on an infrequent basis.

Much better to store the list as a file and $include it.

Re:You know what's great

[Anonymous Coward]

The figure of 100,000 has been thoroughly debunked and, as a result, you can pretty much ignore anything said by someone who chooses to use it (would you pay attention to an author whose sig says "The Earth is flat!"?).

Anyway, you can google for the debunkings (http://www.google.com/search?hl=en&q=bush+killed+ 100%2C000+debunked) yourself but one page that talks about it is at http://www.papillonsartpalace.com/dveathby.htm.

And, before anyone gets all upset and accuses me of being pro-Bush, I'll state here that I am an Objectivist and a Libertarian. The deluge of lies and gross manipulations from both sides of the GOPDem monster make me sick.

Re:You know what's great

[ahmusch]

A database is not the be-all or end-all of computing; however, it is the be-all and end-all of data storage and access when things such as consistency, concurrency, and recoverability are at issue.

If you take your example of codes in a file, how will you distribute it? How will you maintain it? How can multiple users access it? What happens if someone accidentally deletes it?

Databases are designed to solve exactly these types of problems.

You need to have the One True Copy of the data, in all cases. If you wish to distribute a flat-file or marked up copy of that data provided it's completely static aside from a software revision, then that's fine. But keeping key data unprotected -- or worse, opening up yourself to multiple masters -- demonstrates to me that you're thinking about things at the hobby/toy project level, certainly not at a distributed or COTS level.

Your data is your application. It doesn't matter if the data is static or not. And for goodness sakes, if you're going to be using a database anyway for the dynamic data, eat the storage and keep the master copy in the freaking database!

(And if you're not using a database at all, well, that's further evidence to me that you're thinking at the toy/hobby level of project scope.)

If you were going to store large, binary data sets (such as video) in a database, I'd tell you to use Oracle and either store the data out-of-line with BFILEs, using the database as an indexing mechanism, or store it in the data as BLOBS, depending on whether access time (use BFILES!) or recoverability (use BLOBS!) was more important. I don't know what that project used, but writing either of the apps with those data storage concepts is pretty trivial.

Re:You know what's great

[Anonymous Coward]

Wow. One variable declaration and one conditional assignment is too fucking complicated for you, huh? I don't want you anywhere near any code I ever have to use.

Re:You know what's great

[Surt]

There's an important difference to be understood between storing your data in a database, and serving your data from a database. There's a further important bit of understanding that can be gleaned with experience which is that most persistent file systems are databases. Now whether you use an SQL database, or one in another format is just a matter of feature choice. Which db features are most important to you? Queries? Persistence? Reliability? Performance? Etc.

Re:You know what's great

[tomhudson]

No.

A database is not the be-all or end-all of computing; however, it is the be-all and end-all of data storage and access when things such as consistency, concurrency, and recoverability are at issue.

For the types of lists mentioned in the article, simple files are easier to maintain in a consistent state than as data in a database.

Ditto for concurrency.

Why? Because of the lowered overhead, and because you can restrict write access to the whole file at the filesystem level, rather than an individual record. "chmod go-w". Problem solved.

Also, backing up and restoring is also simplified. No locking of records needed, etc.

Simple files are also MUCH easier to recover than a corrupt database. I've had to do both. Guess which one I prefer ...

If you take your example of codes in a file, how will you distribute it? How will you maintain it? How can multiple users access it? What happens if someone accidentally deletes it?

TFA was talking about serving lists through a web server. There is no "distribution". Its included dynamically in the generated page ... the only person who has direct access to it is the process running as the webserver.

Databases are designed to solve exactly these types of problems.

You need to have the One True Copy of the data, in all cases. If you wish to distribute a flat-file or marked up copy of that data provided it's completely static aside from a software revision, then that's fine. But keeping key data unprotected -- or worse, opening up yourself to multiple masters -- demonstrates to me that you're thinking about things at the hobby/toy project level, certainly not at a distributed or COTS level.

Again, you missed the point of the article - serving up a list persistently via a web server. Stuffing everything in a database in such a case is as inappropriate as making everything an object in C++ programming. It leads to a mess. the right tool for the right job ...

Your data is your application. It doesn't matter if the data is static or not. And for goodness sakes, if you're going to be using a database anyway for the dynamic data, eat the storage and keep the master copy in the freaking database!

First, your data is NOT your application. Your data is your data. Your application is your way of accessing that data. The same data can interact with multiple applications.

Also, its better to keep your master copy elsewhere than in the same place you're sticking your data. Otherwise, you have a single point of failure that will kill you.

(And if you're not using a database at all, well, that's further evidence to me that you're thinking at the toy/hobby level of project scope.)

Again, you seem to have totally missed the point of the article ... and my reply in context. But you also seem to have missed a bit of history - the first databases were not much more than hierarchial file systems. The concept still holds value today ...

Again (for the nth time), the right tool for the right job.

Re:You know what's great

[spauldo]

Perl often looks like that (although it doesn't have to). Spend a month working in perl and it's clear as glass. Perl's syntax takes some getting used to, but you could say the same of languages like forth or tcl.

Re:You know what's great

[eyeye]

Nice link AC, how does a right wingers opinion does not equate to "thoroughly debunked" . Falluja was a massacre although that guy probably didnt see anything about that on Fox news.

You should have linked to their warning page instead:
http://www.papillonsartpalace.com/warning.htm

Freedom Page Warning!

Please Read This Before Proceeding

THE FREEDOM PAGE IS HOME TO
POWERFUL CONSERVATIVE THINKING!

MY MARVELOUS WIFE HAS EXPRESSED CONCERN THAT THE FREEDOM PAGE MAY PREJUDICE YOUR FEELINGS ABOUT HER ART.

I WORRY ABOUT THAT AS WELL BECAUSE SHE IS SO PRECIOUS TO ME. HOWEVER THERE HAVE BEEN MANY HUNDREDS OF READERS WHO HAVE COME ACROSS THE FREEDOM PAGES THAT HAVE LET ME KNOW IT BENEFITED THEM.

IT IS NOT INTENDED TO OFFEND THE CASUAL READER.

BE ADVISED HOWEVER THAT IT WILL INFORM AND EDUCATE THE READER ON SUBJECTS THAT I FEEL AFFECT YOUR FUTURE.

PLEASE USE YOUR BACK BUTTON NOW IF YOU ARE EASILY OFFENDED BY CONSERVATIVE IDEAS.

HOWEVER, IF YOU LIKE ME FEEL
THAT GOVERNMENT AS IT CURRENTLY IS MANAGED IS ENSLAVING YOU THEN PLEASE GO TO THE Introduction to the FREEDOM PAGE.

IT IS A MARVELOUS RESOURCE FOR SOME OF THE MOST BRILLIANT POLITICAL THINKING GOING ON AT PRESENT.

THE INFORMATION MAY DISTURB AND PERHAPS FRIGHTEN SOME READERS.

BE THAT AS IT MAY.
I SEEK THE TRUTH.

Powerful conservative thinking... lol..

Re:You know what's great

[Anonymous Coward]

Please feel free to provide any evidence you may have.

Re:You know what's great

[Anonymous Coward]

I think I actually posted a link to many, many pages that disprove the inane claim about 100,000 dead that people cling to. Use this thing called "Google" at http://www.google.com/ and search for "Bush killed 100,000 debunked". In addition to the page you couldn't handle you will find about 11,600 other pages.

I actually fail to find anything in the warning page to show that what he wrote about the "100,000" isn't true. I suppose if he said "George Washington was the first president of the United States" you'd come up with some "conservative conspiracy" which proves otherwise. I suggest you try some "powerful thinking" of any sort.

Re:You know what's great

[eyeye]

If you are getting your factual information by searching for terms that support your argument and ending up with right wing nutjobs then this only shows how weak your argument is.

In addition to the page you couldn't handle you will find about 11,600 other pages.

LOL!
http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF -8&q=jesus%20alien

947,000 pages support the jesus being an alien fact... I take it you do too since it seems to meet your standard of research.

Re:You know what's great

[Anonymous Coward]

OK, you're clearly some sort of idiot but I'll try one last thing. Please prove your assertion is true. You see, that's generally the way science and logic works. The fact that you've made a groundless assertion does not make it true and it is not my job to disprove it to your satisfaction. It is your job to provide evidence you're not full of shit.

Gee, that's a lot tougher isn't it? You would have thought finding proof of 100,000 murdered Iraqis would have been pretty easy, wouldn't you? Go ahead, provide some proof... I'm waiting... You're a brainless dumbass and you'll prove it in your next posting.

Changes to the lists?

[saundo]

Interesting article, but preloading those values will invariably lead to out of sync conditions when the backend changes. Nothing mentioned in the text as to how to cater for that eventuality.

Re:Changes to the lists?

[Anonymous Coward]

When the mostly static list values in the back-end database get updated you restart the app server.

Re:Changes to the lists?

[AndrewStephens]

An excellent point, the article assumes that the data will not change very often, if at all. However, if the list data doesn't change very often then there is little point storing it in the database in the first place.

Not to say that the article was actually bad or anything, its just a little light on when you would want to use this, and what some of the problems with this approach are.

Re:Changes to the lists?

[Anonymous Coward]

"However, if the list data doesn't change very often then there is little point storing it in the database in the first place."

Really? What about articles on the web or posts here in Slashdot? They almost never change and are usually stored in the database.

Re:Changes to the lists?

[Flibz]

I find that something that helps there, particularly if different users have diferent data for the list is to use javascript.

Basically, create a session variable for the user (we'll call it cacheTimeStamp) with the current date/time to the hour (i.e. yymmddhh).

When the page with the drop-down list is called, check current timestamp. If it's expired, create a new javascript .js file & delete the old one.

Then, when the <script> tag is called, use thecacheTimeStamp value to ensure a current .js file is called and let the client browser handle the caching...

Re:Changes to the lists?

[AndrewStephens]

"However, if the list data doesn't change very often then there is little point storing it in the database in the first place."
Really? What about articles on the web or posts here in Slashdot? They almost never change and are usually stored in the database.

It doesn't matter that the content of the articles doesn't change, the whole "list of articles" keeps growing as new articles get posted, hence the need for a database.

What this article is about is data that doesn't change very ofter (or at all). In slashdot terms, this would be the list of moderator categories (insightful, funny, troll, etc). This list changes very infrequently, and could easily be cached (as in the article), or even hard-coded save a database query (I have no idea how it is actually implemented).

Re:Changes to the lists?

[Digital11]

If you were using .NET 2.0 you could use a database dependency cache model which checks to see if data has been updated every so often, and if it has it flushes its cache and reloads the data.

Granted, you could also write it in Java, just saying that its readily available in .Net.

Re:Changes to the lists?

[zorander]

class Category 'parent IS NULL')
end

before_save :invalidate_root_cache
def invalidate_root_cache
@@root_cache = nil
end
end

Being that that was a lot of java code, this is an example in rails. before_save marks a method that should be framework-invoked before commiting a change to the database on the categories table (which is introspected from the 'Category' class name, but can of course be overridden). This caches the category list and expires it whenever it may have changed.

This can of course be made much more elaborate. I'm sure that java code is doing more, being that there's so much of it. Is there a reason why a simpler solution to the problem is unfit? This sort of caching is safe and easy to implement across a system on common queries.

Of course, there's also memcached which provides the same effect without writing a line of code. Am I missing something?

Re:Changes to the lists?

[XMyth]

That and even with .NET 1.0 the solution describe in the article is built-in as well. Application, Session, and Cache collections.

Re:Changes to the lists?

[MemoryDragon]

not necessarly, you just have to set an interceptor which is triggered once the values are edited and updated... very easy to to, either with direct db related interceptors, application events or aspects...

Re:Changes to the lists?

[Anonymous Coward]

Why not be a tad smarter about it? Instead of some fixed timeout for cache data, have a quick query to find out the last time a dataset changed and use that to determine if your cached info is out of date.

Unless that's what you already meant . . . my reading skills are often lacking.

Re:Changes to the lists?

[Cobralisk]

Maintaining separation of code, presentation, and data would require a place to store the list choices. Logically, this would be in the database, where you keep the rest of your data, which is dependent on the list choices. You cache the data since it is basically static. This approach can be more efficient than repetitive redundant database lookups, and more elegant than hard coding the choices into the script, or the .

Re:Changes to the lists?

[Cobralisk]

Of course, there's also memcached which provides the same effect without writing a line of code. Am I missing something?

Most likely IBM programmers in India are paid by the line of code. In which case, you're missing everything.

Re:Changes to the lists?

[Anonymous Luddite]

>> posts here in Slashdot?

Apples and oranges.

You're talking about adding items (posts) to a recordset (slashdot thread). The items are static but the recordset is not. It changes frequently.

The caching they're talking about involves a recordset that seldom changes, and would therefore be suited to storage outside a database and rebuilt as it changes - IE one trip to the database per change rather than one trip per view. This wouldn't make sense with something like a slashdot thread where records are added non-stop...

Re:Changes to the lists?

[Anonymous Coward]

Before making assumptions about your database, check with the DBA or research the architecture of the database. Many DBMS's will cache frequently accessed static query results.

Re:Changes to the lists?

[ahmusch]

Of course, the article leaves one key point unsaid but implied...

Understand the nature of your data.

In any system, there's basically three kinds of data:

-- Static: This is the stuff that changes at a glacial pace, such as state codes, currency codes, and so forth. (for bonus points, put all the static code/description values in a single table with a type identifier for an even larger performance increase at the cost of slightly more complicated code.)

-- Configuration: This is the data that drives the logic of the application. (Go ahead, put it all in code. Good luck with the maintenance.)

-- Data: The actual records that you process or interrogate at some level to do something.

If the JDBC monkeys don't understand the nature of the data at this fundamental level, then telling them to cache data is a recipe for concurrency and consistency nightmares. Not because they mean to, but because they don't know any better.

Caching the static data is probably safe, but the configuration data is only *somewhat* safe to cache. It's too expensive to continually round-trip for everything, so what my project has done is implement a warm-boot process. For every transactional record it attempts to process, it checks a warm-boot status table. If the time has changed, the app flushes and repopulates its caches of both static and configuration data. Sure, it's a little kludgy, but it gives us at least parts of the best of both worlds.

Re:Changes to the lists?

[Anonymous Coward]

that's why you only do this when the lists are static. Or you build it like a cache and add some sort of dirty bit. This sort of thing shouldn't be considered anything new. sigh...

Re:Changes to the lists?

[Anonymous Coward]

ooo u could maybe even store these list values in a cookie.

Re:Changes to the lists?

[spells]

Shhh. M$ is evil. And simple connection caching and result caching makes this a small concern.

In an ASP.NET / IIS / SQLServer, if you do ZERO manual optimization, your connection to SQLServer will be cached and SQLServer will cache the result set, so you will be saving the round trip from your app server to your data server.

Congrats if you have the time in your job to analyze, measure, and optimize this type of thing.

Re:Changes to the lists?

[orasio]

Well, I develop apps in Java, and of course, apply a caching technique for lists.

My data does change, but I store it in a tree.
When someone changes a dropdown, I just erase the cache. When someone wants the list, it gets refilled. All of it is safely synchronized.

Of course, I don't believe it's worth an article, and I don't believe it belongs in /. frontpage.
IBM developerworks is a nice source of information when you want to program the mainstream way. It's good for teams, because it makes easily understandable code. Of course it's boooring and not news, though.

Re:Changes to the lists?

[rainman_bc]

Furthermore, in most web apps, you're already making a trip to the database on most pages to load data. If your connection is open, the trip to the database to select a small recordset from a table has low cost.

One feature I love about ColdFusion is that you can cache queries with a time limit. So if you have a busy app, you can cache the query for even 60 seconds and it'll make a difference.

And yes, I know you CAN do that in PHP and the same with the ADODB library too. (One lib I don't think any php dev should be without IMO)

Re:Changes to the lists?

[willCode4Beer.com]

I'm going to have to argue that storing data in static member variables is generally a bad idea. (Of course, so is using scriptlets in the JSP) Excessive use of static variables can lead to unpredictable behavior, and can be hard to debug. Also, syncing the data becomes more complex because of multi-threaded access to the variable.

While working on high performance web apps where we want to cache the data and prevent having it become stale we genereally to store it in the application context ( ServletContext.setAttribute(name,object) ) with date information.
So, you create a class to hold a date object representing the moment the data was cahed, and the data you wish to cache. Have your servlet or action class (you do have some kind of MVC right?) check that the date is not beyond your pre-dermined max life, if it is, re-fetch the data, otherwise, continue on. Some apps would continue on if another process was busy updating. Depends on requirements and load.

This would also mean not fetching the data at application startup. Which, depending on your app, can be a good thing. If your app is deployed in a cluster, and different boxes may get different types of requests you may not actually need all of the data cached on every box.

We also have a servlet (security controlled) that can be called to flush the cache. So, when an editor is using a content management tool and hits publish, the last step and updating the DB is to flush the cache.

Of course, when I see a *java* programmer using the old *Hashtable* and Vector classes, I'm instantly prejudiced againt his code.

Re:Changes to the lists?

[RetroGeek]

Set up a listener.

So when the data changes (through some UC servlet), have the servlet contact the listeners.

Re:Changes to the lists?

[joost]

I have found that using OSCache works better than anything I could build myself. It caches jsp snippets, whole pages and has support for distributed systems. Not affiliated, just a happy user. www.opensymphony.com/oscache

Re:Changes to the lists?

[curunir]

FWIW, caching is also built into Hibernate and other ORM solutions for Java.

I would imagine the benefits of caching would be outweighed by the drawbacks of maintaining a custom caching solution in most cases unless profiling the application has shown that it is absolutely necessary.

Re:Changes to the lists?

[Anonymous Coward]

Sorta, you still have the connection overhead to the DB, which can be unwanted depending on the app. It's usually better to cache this kind of stuff closer to the front end rather than relying on SQL Server.

Huh?

[Renegade Lisp]

Sorry I don't get it. Of course, when you load your data from a cache in main memory, even from within the same address space, you are several orders of magnitude faster than if you make the trip to the database each time. And by several orders of magnitude I mean six to seven orders: you'll easily be a million times faster for a given operation. (A database roundtrip is on the order of tens of milliseconds, while a lookup in a Java hashtable takes mere nanoseconds on typical hardware.)

What's the point? Since when is Slashdot a forum for random tech tips (and not very thrilling ones at that)? Did IBM pay to get this posted? Is Slashdot trying to make fun of IBM by actually posting it?

Re:Huh?

[0x461FAB0BD7D2]

It's because of all the Adblock users out there that we get ads as stories now. The summary even reads like one. Or it's just a slow news day.

It all depends on how tight your tin foil hat is.

Re:Huh?

[Anonymous Coward]

Slashdot is just trying to get the word out on simple techniques to help withstand the crushing impact of a slashdotting. (Something about it being part of their hours of community service, not sure what that's all about.)

Re:Huh?

[computational super]

I think you're being FAR too polite here, sir. Feel free to drop in an occasional, "Are you f-ing kidding me with this drivel?" in your critique of this type of ridiculously simplistic and obvious article.

On the other hand, there's a good take-away here. If this "revolutionary technique" was so mind-bending to IBM consulting services, I know where I won't be spending my consulting dollars...

Re:Huh?

[fedor]

simple techniques!? IBMs examples are pieces of over-engineered bloatware....

why not just:

getValues()
{
if (list==null)
list=loadList();
return list;
}
??

Isn't that easier than writing XML-configuration files?

Re:Huh?

[fedor]

It's a joke... The article was written by Ali G... look at the picture at the end of the article.

Re:Huh?

[cablepokerface]

Hahaha. You're right!

If only I had mod points.

Re:Huh?

[Renegade Lisp]

On the other hand, there's a good take-away here. If this "revolutionary technique" was so mind-bending to IBM consulting services, I know where I won't be spending my consulting dollars...

It's interesting though, that this kind of "revolutionary technique" can be indeed a step forward in the I-don't-know-how-many projects where developers aren't aware of these simple issues. I'm in the consulting business, I've seen them. For them, this kind of consultant might actually be worth the money, and there's nothing ridiculous about it. Some shops simply have no need to have the brightest and smartest developers in-house.

Re:Huh?

[Xugumad]

Except, this isn't at the "Brightest and smartest" level. This is basic caching. If this was a discussion of the relative merits of linked lists vs doubly linked lists vs hash maps for various different data types, and read/write patterns, we can start thinking best and brightest.

If it went into a discussion on how to store your data in memory so that the data the application is looking for is probably in the processor cache, we can definitely talk best and brightest.

Simple caching, however, is basic stuff.

Re:Huh?

[Renegade Lisp]

Except, this isn't at the "Brightest and smartest" level. This is basic caching. If this was a discussion of the relative merits of linked lists vs doubly linked lists vs hash maps for various different data types, and read/write patterns, we can start thinking best and brightest.

Absolutely agreed. And the things that you mention are those that you'd wish your expensive IBM consultant was capable of. I stand by my statement however, that

1. there are shops where none of the in-house developers is aware of the option to cache frequently-used data in the app's memory, and of the vast performance benefits this can bring
2. yet still, such a shop might be a successful enterprise that manages to maintain its position in the market. Sometimes they hit upon a consultant that helps them to improve their stuff vastly, and it's all the better (both for the consultant and the company :-)

This is not "best and brightest", absolutely. But it is a realistic description of what you find out there, quite frequently.

Re:Huh?

[cloudmaster]

So it's perfectly OK to do things sub-optimally, even if all it would take is a general understanding of the most basic programming practices to make a product orders of magnitude better? "Well, it mostly works, even though it's dog slow." Do you run Windows? :)

The observation that doing things wrong doesn't always run one's employer into the ground does not magically justify doing things wrong. It's this acceptence of adequecy that's behind the poor state of many things today, but that's a rant for later. My post's good enough for now. ;)

Re:Huh?

[Jeff Hornby]

while a lookup in a Java hashtable takes mere nanoseconds on typical hardware Really? What about a C hashtable? or a Pascal hashtable? or even a VB hashtable? (some people get very tired of Java programmers who think that they invented everything)

Re:Huh?

[Renegade Lisp]

Really? What about a C hashtable? or a Pascal hashtable? or even a VB hashtable? (some people get very tired of Java programmers who think that they invented everything)

Not me, sir. I learned my trade on totally different languages; I was just referring to Java hashtables because the article was about Java to begin with. For languages such as C, I'll happily grant you another order of magnitude if you wish :-)

Re:Huh?

[miu]

So it's perfectly OK to do things sub-optimally

In the real world of low pay and low prestige IT work it is fairly common for things to be done in a far less than optimal manner. People who know how to do better may just not bother, why make it complicated if it meets the spec - it's not like anyone is gonna give you a cookie if you build an order entry system that is twice the speed they expected.

At the other end programmers just out of school or training programs may have learned some theory and much of the nuts and bolts, but not really have an understanding of how to connect theory to practice. In an effort to save money I can easily imagine a shop in which such inexperienced coders are the vast majority of the workers.

Either way in reality you are gonna have people who cannot or won't bother applying basic knowledge to optimization. It is short sighted greed that is behind the poor state of things today, acceptance of adequacy is nothing more than a result of that greed.

Re:Huh?

[cloudmaster]

I dunno, I think that acceptance of adequacy may have been fueled by greed, but that the low standards are behind the problem. Greed would exist either way, but it took acceptance of sub-par products to make realization of the greed possible.

Larry Wall was right - good programmers have a sense of Hubris, and want to make their programs "right" whether anyone else will notice or not.

Oblig. Simpsons Quote

[bigtallmofo]

"And people, I can't stress this enough. Put your garbage in the garbage can."

"Garbage in the garbage can. Hmm. Makes sense."

Re:Oblig. Simpsons Quote

[Darth_Burrito]

Heh, I once wrote a generic query caching system in .net and garbage collection was indeed a serious problem.

Re:Oblig. Simpsons Quote

[FuzzyBad-Mofo]

Let me guess, it did a format on the c drive? Darn smart-assed garbage collection routines..

Slow news day? Christ.

[Electroly]

This just in! Caching frequently-used data yields performance improvements! Film at 11!

April Fools...

[Kr3m3Puff]

Is this April First?

Wow, next an article about using "for" loops? The benefits of "bubble sort"? "Binary trees"?

Re:April Fools...

[Anonymous Coward]

for loops = good
binary trees = good
bubble sort = bad

One of these things is not like the others... :)

Re:April Fools...

[Vombatus]

Careful there...

Someone might hold the patents for those techniques.

Well duh!

[AndrewStephens]

Keeping frequently used data in static singletons, who would have thought it!
Seriously, this is probably good advice for someone just starting out programming, but I would expect anyone with any experience at all to know about this. Its hardly a revolutionary new technique.

Re:Well duh!

[computational super]

Actually, I was pretty disappointed at the author's choice of singletons for the static data in this (painfully obvious) article... this is actually the sort of thing that ServletContext was designed for. Singletons always bother me because they're basically global variables, with all the problems that global variables bring.

Re:Well duh!

[DaHat]

Be careful with mentioning design patterns like Singletons, you may lose most of the spaghetti code programmers.

Done this for years

[Ckwop]

Well thank you captain obvious.. I've been doing this for years with ASP. Just load the contents of the listboxes into the Application object.

In ASP.NET you can even do cache invalidation when the database changes. Simply create an extended stored procedure that's fired when any of you update/insert producers run that write to the changed record ids to a Queue (using Microsoft's Messaging and Queuing service) then have a thread in the ASP.NET process that periodically check the queue for new messages and clear the values that have changed out of the cache.

Because the Queuing service works across networks it's a really neat way to provide scalabity in web applications - if you can't wait for SQL 2005 which will provide cache invalidation on database updates as standard.

Simon.

Re:Done this for years

[cablepokerface]

In ASP.NET you can even do cache invalidation when the database changes

That's right. Have you also tried ASP.NET 2.0 yet? (still in beta though). You don't have to do any extra works whatsoever. You can just say: Hey I want this data in the webapplication to be valid until that stored proc has a different resultset. No programming involved. It's really awesome.

Re:Done this for years

[Anonymous Coward]

Figures that what everyone else things is old news, you ASP guys think is still really cool. Make sure to hail your chief for misleading you yet again into thinking you are on the cutting edge.

Re:Done this for years

[cablepokerface]

Really? What programming platform does that already for a long time? Ow, wait, you don't know, and you checked the AC box to make absolutely positive that you can troll without being harmed. You must be a smart fellow.

Re:Done this for years

[Anonymous Coward]

Don't be silly. ASP developers have been caching data in ASP since it was first released.

ASP may not be the greatest platform, but don't underrate it's developers. They were the first Microsoft users to abandoned Microsoft's for-sale products; they went to ASP partially because using it incurred no incremental cost. As a result of their willingness to step away from Microsoft's pay-as-you-go approach, today there are at least 6 times more ASP pages on the Internet than ASPX (.NET), and that will remain so for years much to Microsoft's embarrassment.

Now that .NET has been released and ASP abandoned by Microsoft, most of those developers are moving away from ASP to other non-Microsoft web tools. But there's no need to denigrate what they did or why they did it. Microsoft tried to use IIS + ASP to keep their developers; instead it provided a path that allowed those developers to escape Microsoft's embrace.

Re:Done this for years

[willCode4Beer.com]

Whats even more interesting is that you are doing it the way it *should* be done in J2EE. Your application object can be equated to the application context or more correctly put the ServletContext object.

public void doGet(HttpServletRequest req, HttpServletResponse resp) {
ServletContext ctx = getServletContext();
// store the data
ctx.setAttribute("dataKey", dataObject);
// retrieve the data
Object dataObj = ctx.getAttribute("dataKey");
}


Funny, I never thought I would compliment an ASP programmer. :)

Re:Done this for years

[Whyzzi]

Agreed. I figured that all out on my own when I made I links database application a few years back, and believe me I didn't even know how to program ASP. As the catagory list grew longer, the administrative interface running with the catagory select list began to get unbearably time comsuming.

The first thing I did was to stick the selection list into the session object - then the time it took to log into the admin interface grew to 4 seconds. Looking further on the M$ website I eventually discovered the application object. When I wrote that in, poof no more obscene delays. Of course I wrote the select list building feature as a function so I could recreate the list as soon as a new category was created.

Need to now more about classic ASP and the Application Object? Start here.

Re:Done this for years

[Anonymous Coward]

Oh, you think YOUR'E cool, I did the same thing in ASP 3.0!

And it made me want to kill myself.

PHP-ADODB Caches these queries

[displague]

With ADODB for PHP (and perl http://adodb.sf.net/, you can call CacheGetAll, CacheExecute, etc... The query resultset is saved to a temporary file. This avoids having to create the cache within the same function you would normally call without having to write extra code.

[first useful post?]

Re:PHP-ADODB Caches these queries

[displague]

Correction: should read "and python" not "(and perl."

You forgot something...

[Anonymous Coward]

Were's the patent?

Mod parent up!

[Renegade Lisp]

No need to post anything else in this thread! ROTFL

You answered my question b4 I asked, partially ..

[RedLaggedTeut]

At least partially.

So, you can save result in a file, but can yo usave them in memory too, using PHP?

using smarty template

[chrisranjana.com]

and of course smarty.php.net is there too !

Java Servlet init question ..

[RedLaggedTeut]

In init(), there is
ListValuesLoader listValues = new ListValuesLoader();
but no global.

However, I can't find the object where the data is actually cached. Is it a Singleton somewhere? Pretty well hidden it must be ..

Re:Java Servlet init question ..

[Vengeance]

The individual list types are all implemented as singletons.

That is, they have static 'getInstance' methods, maintain private static references to their own solitary instance, and have private constructors.

This is a very standard OO pattern, and seems to be one thing the article got essentially right.

SQL

[smittyoneeach]

For additional style points, load all lookup tables in one query, and concatenate the HTML into the values.
But I may be the Kurtz of SQL...

Re:SQL

[Anonymous Coward]

hardcoding HTML is certainly evil...

Is This Consultant on YOUR Payroll?

[rimu guy]

If you have yet to read the 25-page FA, may I present the precis:

Database hits are expensive. Reduce them where possible. For example, cache static lookup data.

The simplicity of the point however is lost in the complexity of the article. It covers web.xml settings, servlet classes, list value loaders, persistence backends for said loaders, data source 'helpers' for said loaders, custom object classes for the loaders, several subclasses for said object classes, and a jsp page (to boot). Phew.

The author refers to this design as a quick and easy approach. It is not. If you are not familiar with Java and read this article, please do not be put off. He could have demonstrated the point with a far simpler example. E.g. static variable, sql statement, jsp code, done.

[The author] has worked with IBM Global Services for one year, and has five years of experience in J2EE-related technologies. And it shows. I dread to think how much a fully realized IBM Global Services project would cost should all its consultants apply this sized sledgehammer to each small task. Hopefully the article was not written up on the client's dime as well.

--
Java Hosting

Re:Is This Consultant on YOUR Payroll?

[Vengeance]

I have come to expect this kind of thing from IBM consultants. This guy is a 'Lead Developer', and he's writing a Software 101 article, padded with extraneous crap in a likely effort to meet some imposed minimum length.

Re:Is This Consultant on YOUR Payroll?

[CastrTroy]

Is that "Lead" the element Pb, or "Lead" as in, in charge. Seems like his code is as heavy as the former.

Re:Is This Consultant on YOUR Payroll?

[gbjbaanb]

What you wrote there is so underrated, I wish all the people who read this article would hit on your comment and realise that writing good code is more about making it simple and working, than it is to use all the latest, coolest, make-me-seem-clever crap that we have to put up with nowadays.

And, to realise that those consultancies charge a large amount of cash because their developers are crap and produce this over-engineered rubbish. (and is another reason why all those large-scale government projects they produce don't work properly if at all)

Ah. happy now, time for a lie-down :)

Re:Is This Consultant on YOUR Payroll?

[willCode4Beer.com]

I question his experience. He would have a hard time interviewing in my shop using that code for a few reasons.

• Using static variables to store global data instead of the ServletContext
• Using Vector instead of ArrayList
• Using a Hashtable instead of a HashMap
• Magic strings

P.S. Before you mention synchonization remeber as of Java 1.2, to get a synchronized Map (in place of Hashtable) or List call Collections.synchonizedMap(map) and Collections.synchronizedList(list).

Umm...

[Anonymous Coward]

How is this news? This is a random tutorial article from the net. Not newsworthy at all.

GoF Decorator pattern is better for this task

[MarkEst1973]

Assuming one writes a well-factored application, I believe Decorator makes a much better caching mechanism that what the article presents.

A Data Access Object should have 1) an interface (defining add, remove, retrieve, etc.) and 2) a standard implementation of the interface that reads/writes to the database on every method invocation.

A Decorator can implement the data access interface, delegating all method invocations to a wrapped instance of the standard implementation. Decorate the behavior of the standard impl. by providing a cache, checking the cache before retrieving a model and updating the cache before saving a model.

Because the standard impl. and the decorator share the same interface, you can have a factory create instances for you. Your code doesn't know or care which instance it is using. Mix and match Decorators to your heart's delight. A logging Decorator (track what data is being access, etc.) can be thrown into the mix, and again your calling code wouldn't be the wiser.

This pattern is easily unit tested and load tested. It doesn't require a running web container to test or run. It Just Works(tm).

Re:GoF Decorator pattern is better for this task

[Anonymous Coward]

For a drop-down list?

Re:GoF Decorator pattern is better for this task

[MartinG]

why not?

If you think it's overkill then I suggest you haven't understood. It's still just some cacheing code, but its put in a decorator class so it is useful to all callers. It also has the benefit of being easy to prevent dirty reads because it knows about writes and updates.

Pretty damn cool idea if you ask me. (and one which I will be playing with myself when I get back from work!)

thanks to the grandparent poster for mentioning this.

Well, duh

[Tim C]

It's called caching, and it's been done since people had to load in commonly-used external references.

I've not RTFA, so perhaps it's truly excellent, but why the hell has this been posted? Anyone who's writing any sort of application and not making intelligent use of caching is either really junior, or should probably be looking for a new job.

Re:Well, duh

[ErikTheRed]

Absolutely. I've been meaning to write an article on this (in my near-infinitely long queue, after the stuff that pays real money). The problem is that many web designers (esp. those without other programming backgrounds) learn the beauty of storing data in databases and they become overly enamoured with it. Databases are appropriate for many web applications but if you're hitting the database to, for instance, draw your home page, you're going to beat your server to death and your viewers are going to see a lot of "Cannot connect to MySQL / MSSQL / whatever".

There are two ways (maybe more, but these work for me) around this: 1) fairly simple sets of information that are infrequently updated should be stored in a text file (.INI format, XML, whatever you just pulled out of your ass). This would probably include page menus, tables of links to external sites, etc. 2) Cache the results of frequent, simple queries. The logic goes something likes this:
A) Collect your variables, turn them into a filename-compatible string, with two extensions - one for counting page hits on an hourly (use discretion here) basis, the other for caching generated pages. B) Check to see if the page counting file exists; if it does, then increment the count. No elaborate locking mechanisms are necessary, because we're just trying to find out if a query is getting repeated at a high frequency. If we miss one or two that's ok. Check to see if the count is beyond a certain threshhold. If it is, then we start the page caching process; if not we send the page to the browser. C) Check to see if such a cached generated-page file exists. If it does, send said file to the user's browser. Be sure to 'touch' or otherwise increment the file's mod date in the process (for the next step). D) Run a reaping process to make sure your server doesn't fill up with such files. Delete anything that hasn't been updated in an hour (or day or whatever) E) If you haven't already sent a cached page to the browser, generate the page as normal. The only difference is that rather than 'echo'ing or 'print'ing or whatever the results to the user's browser, you should accumulate them in a rather long string (hopefully you're using a language that won't let this overflow). F) If your query count is over the threshhold , then write this string to the generated-page cache file. G) Write the string out to the user's browser. H) You can still have 'slightly dynamic content' like rotating banner ads, etc. implemented with a search-and-replace function within the generated page cache file / string.
Even though this looks like a lot of work, it's usually less than 50 lines of code per page, much of which can be copy/pasted, worked into a library, etc.
Wow, just outlined the article... maybe I'll get to it sooner than I thought...

Re:Well, duh

[curunir]

No offense, but a lot of this sounds like poor advice.

Firstly, a lot of your caching solution should depend on what tools or language features are available for the language you're using to implement your site. For example, advocating caching things to disk isn't really necessary when you're using Java since it's more effective to use in-memory caches. If you use an ORM solution like Hibernate, caching happens behind the scenes and you don't even have to think about it. However, if you're using a scripting language like Perl or PHP, you can't use in-memory caches and have to turn to some sort of on-disk storage.

Secondly, even if you're going to cache to disk, there's usually better ways to do it than having special files (case 1) or saving the entire generated page to disk (case 2). For example, the en vogue way of doing disk caching in PHP is to serialize your data structure to disk and then unserialize it on every request. From there, the page can be generated normally and you've still avoided the trip to the database while retaining far more flexibility when it comes to making your page dynamic.

Thirdly, caching isn't rocket science. In every language I can think of, third-party tools exist to help you handle that aspect of your app for you to keep you from having to write your own custom implementation. Unless you've profiled your application and know that optimizing a certain part will have substantial benefit, using an off-the-shelf solution will make your application far more manageable and far more friendly to developers not already familiar with the code.

The main thing to remember is that if you're having this issue, chances are it's not new and someone has effectively solved your problem in the past...just go look for their solution. The world needs another roll-your-own caching solution like it needs another proprietary video or PHP/MySQL book...

Re:Well, duh

[ErikTheRed]

With all due respect, I think your reaction is little bit knee-jerk on this. While, as it is often stated, there is no silver bullet and there will definitely be places where this is inappropriate, there are quite a few areas where it will work nicely. I'll be the first to admit I'm fairly naive when it comes to Java Server's caching capabilities (I'm more of a php / perl / shell scripting guy).

Serializing data to the disk is a good suggestion in many cases. Perl is fairly rich in this area, however, PHP (far more popular for web development, especially with less experienced programmers) is not. It has a serialize to string function, however, this is not terribly useful if you have several variables (unless you want to deal with delimiting the variables, making sure the proper character(s) are escaped, and then unrolling that later).

I'd absolutely agree that caching isn't rocket science for experienced programmers, however, many web developers cut their teeth on PHP (or - gasp / choke - visual basic) and have very little background in theory outside of making their pages work. They probably will not seek to become programming experts. The beauty of this approach (ok, maybe not beauty, but perhaps 'utility') is that it's extremely simple to understand. Basic file I/O and string concatination have a very gentle learning curve for the target audience. They don't have to worry about installing third-party libraries on their web server (which may or may not even be possible). They also don't have to worry about the various cases and caveats inherent in third-party caching solutions. This is simple enough to understand in its entirety - there are no 'black-box' aspects to it.

My goal here is to not turn web developers into professional programmers - not that it's not a noble goal, it's just not a realistic one. It's for people to be able to browse sites without seeing "could not connect..." errors (oh, yeah, and to help people have web sites that don't go down like a two-dollar hooker). You have to keep in mind that sometimes quick & simple is better than complex but elegant.

Re:Well, duh

[curunir]

...however, PHP (far more popular for web development, especially with less experienced programmers) is not. It has a serialize to string function, however, this is not terribly useful if you have several variables (unless you want to deal with delimiting the variables, making sure the proper character(s) are escaped, and then unrolling that later).

Why would you bother with delimiters when PHP is perfectly capable of serializing arrays/hashes. Just dump all your variables into an array and serialize. The code necessary to implement this kind of caching is literally 3-4 lines, with a couple more to check the timestamp on your cache file for expiration purposes. As I said before, there's nothing new here...I've seen a ton of PHP apps/libraries that implement caching in this manner.

I agree that there are a ton of inexperienced PHP coders out there writing bad code who will never fully understand the theory of good v. bad caching solutions. But if you're going to try to teach something to those programmers, you should understand these issues a bit better. It's not hard to do a little research before advising others to use some new method you've cooked up. Otherwise, it's just a case of the blind leading the blinder.

re:

[Anonymous Coward]

What? and give us all the details so we don't have to rtfa.
Like we need that stuff anyway to make a post!! neh eh!!! *pokes out tongue*

Just use JDO and you're done

[Anonymous Coward]

Sheesh. Sometimes I think IBM consultants still live in the 90s.
This is standard procedure and any good JDO implementation will do this for you.

http://en.wikipedia.org/wiki/Java_Data_Objects

Laugh it up fuzzballs

[Anonymous Coward]

Laugh while you can, people. Because Srinivasa Rao Karanam is training your replacement as we speak.

cool, ibm advertising

[Anonymous Coward]

wow, ibm advert. Nice /. - how much are they paying you to advertise in your stories?

Comment removed

[account_deleted]

Comment removed based on user account deletion

Memcached

[captainclever]

Or you could use something like Memcached. Works with pretty much any language, and tonnes less hassle. (Thanks LJ ;))

Re:Memcached

[captainclever]

Oh, and you can write postgresql triggers that invalidate or repopulate memcache entries.

News?

[Baavgai]

Seriously, I've you're into servlets and you haven't been caching non dynamic data, shame on you.

By same token, if you're on of those twits who caches five years of data warehouse information in the application layer, there's a special place in programmer hell for you...

Re:You answered my question b4 I asked, partially

[RussGarrett]

Memcached. I think you'll find that is actually the way it should be.

Well duh...

[Anonymous Coward]

As the subject :>

Not that bad a hint.

[Ancient_Hacker]

Yes, it's obvious to anybody with half a brain. But from the looks of several apps out there, there's a LOT of coding being done by the lower half of the bell curve:

• One very old 200,000 line app, written in assembler, that could only ever run on one particular CPU, had a #define for the number of bits in a packed character field (6 bits BTW).
• One really losing Java app does about a bazillion (200+) separate SQL queries to ask for things that have not changed in 50 years. Funny, the app runs slowly, even on a rather hefty server cluster. It runs much slower than the old CICS mainframe app it replaced, which ran in one 30 MHz CPU, 4MB of RAM.
• Many apps do SQL queries to get the names of the days of the week. And the names of the months. And the abbreviations for same.
• It's a

Re:Not that bad a hint.

[ergo98]

• It's a

Looks like you had an SQL Statement timeout there.

Re:Not that bad a hint.

[Anonymous Coward]

Many apps do SQL queries to get the names of the days of the week. And the names of the months. And the abbreviations for same. Well if you were living in Kazakhstan (or one of the other -stans) where the dictator went around renaming days and months in honour of friends and family members, this might be handy if they change on a regular basis.

Re:Not that bad a hint.

[PickyH3D]

Many apps do SQL queries to get the names of the days of the week. And the names of the months. And the abbreviations for same.

The general intent for something like this is for internationalization.

Obviously, they just never got around to doing it for anyone outside of the English language. I do think this stuff could be all cached in a hash table though, and marked by their language name to load the data, which could then be defaulted to English if nothing was found.

Re:Not that bad a hint.

[ergo98]

To actually really reply to your post. Firstly I will entirely agree with you that there are a lot of people on the bottom half of the curve - this is a tremendous contingent for whom caching will be a remarkable revalation.

# Many apps do SQL queries to get the names of the days of the week. And the names of the months. And the abbreviations for same.

I don't see a big problem with this. Of course it should be cached, as this article covers, however this is a fairly typical technique to support multilingualism.

And for the follow-up article...

[azuroff]

...this "Lead Developer" at IBM will discover the wonderful performance benefits of pooling database connections, rather than open a new connection every single fricking time he hits the database. No wonder he saw a massive performance increase when he learned how to cache the lookup values.

And what's up with the "obj_" prefixes in some of the code listings? Newsflash: Java is an object-oriented programming language, and most competent Java programmers can figure out that a variable called resultSet probably isn't a primitive. You don't need to waste another 4 characters pointing out that fact.

Genius - Quick PATENT IT

[Anonymous Coward]

Wows thats such genius, it will revolutionise programming. Quick patent it!

recursion, indirection, and...

[10am-bedtime]

...caching (the subject of the article) are the three fundamental algorithms, so it was told to me by a little bird.

of course, i immediately made a base case, trapped the fowl and filled my belly (to put into practice such sound advice, you see ;-). mmmm, cs stew...

Re:You answered my question b4 I asked, partially

[Rakishi]

Make a virtual hd? (memdrive or whatever its called).

It seems that php docs state:
" Optionally you can use shared memory allocation (mm), developed by Ralf S. Engelschall, for session storage. You have to download mm and install it. This option is not available for Windows platforms. Note that the session storage module for mm does not guarantee that concurrent accesses to the same session are properly locked. It might be more appropriate to use a shared memory based filesystem (such as tmpfs on Solaris/Linux, or /dev/md on BSD) to store sessions in files, because they are properly locked."

Somewhere else it says:
"To use shared memory allocation (mm) for session storage configure PHP --with-mm[=DIR] ."

You can probably google to get more info, but it seems that mm is a viable alternative.

What is this post for?

[christoofar]

Any web developer who'se picked up a book for PHP/ or NET Web Development would know how to do this in Apache or IIS.

From the 'duh' department...

[Anonymous Coward]

Woah, you can cache objects in memory? You don't need to go to the remote database to get a list of states names? Holy crap!

SQL is the plague of informatics

[Anonymous Coward]

A *terrible* language designed not by a computer mechanic but by ignorant math professors. Horrible. Worst part is that the new generation of programmers thinks they should do everything in SQL.

See LK discussion and linus' reply:
>
> Why not to use sql as backend instead of the tree of directories?

Because it sucks?

I can come up with millions of ways to slow things down on my own. Please
come up with ways to speed things up instead.

Linus

Yes.

[Anonymous Coward]

But, have you actually looked at the crap most open source projects turn out? (Certain ones are turning out incredibly high quality works of art; most, like slashcode, are crap.) I think this is actually a real hint to the majority of slashdot readers who are completely clueless.

Performance?

[PIPBoy3000]

What's the point of all these fancy new RAID arrays and such if we developers can't get increasingly sloppy with our coding practices?

Though I kid, I find that populating drop-down boxes for applications are pretty minor in the grand scheme of things. My performance issues tend to stem around doing cross-server database queries involving tables with millions of rows. That's when things get slow.

I'm sure there are some people who have to do this level of optimization - folks like Google and Amazon. The only place I've had to deal with this sort of thing is with things like our Intranet home page. In that case, all the drop-down boxes became JavaScript includes that were dynamically recreated every time the database changed. It went from an ASP-based page to completely static, taking considerable load off the server.

Too complex for VBS

[AwaxSlashdot]

Well, the complexity of the problem made me think it would be VBS. This is the kind of example you can found in 'Advanced VBS' (or even Expert level).
Oh wait, you were talking about coding language ...

What's your Vector, Victor?

[Vengeance]

After bracing myself with some more coffee, I read a bit more of this article.

Bad, bad, bad.

What's with the Vectors, anyway? I haven't used those in years.

Re:What's your Vector, Victor?

[BubbaPuryear]

They're for the riviting follow-on article... ya know, how to "performance tune" by not "over synchronizing" your code...

Breakthrough

[Nyhm]

Wow! What amazing breakthroughs Web developers come up with! Why, soon there might be a good way to maintain client-side state for Web applications. What I will coin as "client-side applications" could provide endless opportunity for rich, dynamic user interaction. These "client-side applications" could even do their own processing and have storage from the local host. If only there were some type of, what I'll call, "socket" to maintain a persistent data connection for protocols designed to fit the needs of the "client" and "server." Some clever web developer will figure how to accomplish this over HTTP...

... It'll take a lot of work, layer upon layer of protocols (all riding on HTTP), and countless new "web technologies," but some day Web-based applications might just become as usable as my hypothetical vision of "client-side applications."

Re:How did this make it at Slashdot

[tarquin_fim_bim]

Slashdot editing has been outsourced to India.

Just In

[Anonymous Coward]

That was not for the run-of-the-mill programmers. That was for quiro in californian school.

Lookup Loader library

[Anonymous Coward]

The proposed solution is not rocket science, but as the examples suggest, there can be a fair amount of code to get there, which can sometimes be tedious to implement (especially when having many such lists).

One may want to look at automating this process using a product like the (open source) Lookup Loader library (http://www.esslibraries.com/ess/libraries/lookup/ home.do or http://sourceforge.net/projects/esslibraries). Its purpose is to automate the (re)loading of such lists as much as possible. Configuration can be done through XML files. The library is extensible and not limited to databases and fully supports I18N data (something the article does not mention).

Is this guy for real?

[955301]

Wow, if you can publish an article about not loading the same data over and over again, then I've got some work to do!

How about an article on how to make an application run the same operation over and over instead of repeatedly executing it from a cron job?

Or an article about keeping your strings in a string table instead of dispersing them throughout your code?

Or perhaps I could write about how to use cut-and-paste to aid in producing Javadoc instead of typing @parameter over and over again?

Is this jackass for real? Publishing this garbage is almost as bad as filing for patents on trivial extensions of ideas. I hope the author at least realizes the only good to come from it is to pad his resume.

Re:Hungarian & long var names

[Anonymous Coward]

Vector obj_VectorOptions = new Vector();

What's with the mixing of Hungarian and long variable names?

I've been programming Java for 8 months and even I can tell this is a waste of space.

Slashdot-Talking about the obvious for years!

[siberian]

Lame, this is basic stuff. Give me a break.

Hmmmm

[mgbaron]

The solutions provided in this article have very limited usefulness. I support the inclusion of a good web efficiency article on slashdot, but this article is a waste of time and its techniques are a waste of energy unless, for some reason, your site uses a ton of dropdowns.

I read another article on IBM once about web efficiency that was much more thourough and much more slashdot worthy, but I can't find it unfortunately. When I do, I will post it.

If this is news for slashdot....

[Anonymous Coward]

...no wonder the site is so slow.

- Anonymous

Where did you get that from?

[hrvatska]

Where did anyone present this as a "Revolutionary technique"? I don't see that as being said or implied anywhere. It's a how to article for people who are new to Java, with a (nearly) complete end to end example. Should it have warranted an article in /.? Perhaps not, but your crticisms of IBM consulting services, based on a how to article for new Java developers, seems to be quite a leap.

Externalize Picklists...

[zettabyte]

He should be externalizing the picklists to a .properties file. He'll get the caching through Properties and the strings will be ready for localization.

If the picklists are at all updateable while the application is running, he can cache as he does, but he'll a mechanism to invalidate the cache and re-read from the database.

Forgetting that for a moment:

1. Hungarian notation is NOT necessary in Java. Period. End of story.
2. No one uses Vector anymore.
3. There are some nice tag libraries, so STOP PUTTING JAVA CODE IN JSPS!

If you're a code monkey, memorize these two quotes:

Increasingly, people seem to misinterpret complexity as sophistication, which is baffling - the incomprehensible should cause suspicion rather than admiration. - Niklaus Wirth

and

...it is simplicity that is difficult to make. - Bertholdt Brecht

(quotes from http://www.vanderburg.org/Misc/Quotes/soft-quotes. html)

Re:Externalize Picklists...

[FriedTurkey]

No one uses Vector anymore.

Why the hell not? Is it uncool now? Do I need to stop using it for l33t status?

There are some nice tag libraries, so STOP PUTTING JAVA CODE IN JSPS!

One might argue it is easier to read especially for something simple.

Are you one of those types that spend all thier time critizing other people's code just because it is not how they would code it? I hope I never work with you.

Re:Externalize Picklists...

[Anonymous Coward]

One might argue it is easier to read especially for something simple.

The trouble with putting java code into JSPs is that people start to mix the presentation layer with business logic. That and web designers should really be designing the HTML and they shouldn't be required to know any java. Plus code in jsp's isn't easily resuable.

Re:Externalize Picklists...

[philipborlin]

There are very good reasons not to use Vector. The main one is that it internally synchronizes all calls. Any algorithms 101 class will show you why that is a false sense of security.

The classic example is:
vector.size();
vector.get(0);

Each one of those calls is synchronized internally but the JVM can still switch threads inbetween the two calls causing a race condition. To make that code thead safe you need to synchronize externally:

synchronized(vector) {
vector.size();
vector.get(0);
}

Now the code is thread safe but there were three synchronization calls (our explicit call and one for size() and one for get()). Very inefficient.

Re:Externalize Picklists...

[zettabyte]

There are other responses above mine which better address your questions/disagreements. My responses below are meant to be humorous and/or obnoxious, depending on your perspective.

Why the hell not? Is it uncool now? Do I need to stop using it for l33t status?

Yes. It's uncool. If you wish to be 1337, you need to stop using Vector. Only \/\/4|\|k3rz use Vector. At least that's what they told me in my Java class at DeVry(1). ** braces for DeVry graduate flames **

One might argue it is easier to read especially for something simple.

Those of use with inferior programming skills prefer to separate our View code (JSPs) from our Controller code (Servlets). We do so because we are not capable of remembering where some page setup code was originally written (Where is that list box set up? In the servlet? the display.jsp? the header.jsp include? the vars.jsp include? one of the includes in one of the includes? The base servlet? The action servlet? the form? the application initialization plugin? etc.). Following this practice, we always know where to go to find the setup for a view. However, someone with your superior coding skills, who can clearly remember where all page setup code exists within an application (whether written by you or not), need not adhere to this rule.

No, it is I who hopes to never work with you. I would clearly be out of my league and unable to keep up, what with having to use design patterns like MVC to help me remember where to find code that I have (and even have not) written.

Are you one of those types that spend all thier time critizing other people's code just because it is not how they would code it? I hope I never work with you.

Yes. I am. I'll give you an example.

"thier" != "their"

and

"critizing" != "criticizing"

(1) I have nothing against DeVry graduates or their skills. I don't know any DeVry alumni, nor have never worked with anyone having any affiliation with DeVry. I'm merely making a joke.

Re:Externalize Picklists...

[metamatic]

Of course, if only Sun had gotten the language right the first time, we wouldn't have five different array implementations to choose from.

Almost ...can be viewed as an exercise in caching

[Terje Mathisen]

"Almost all programming can be viewed as an exercise in caching."

For the last ten years or so, this has been my .sig, and this article is just a pretty bad example:

Yes, you should probably cache some or most of this stuff, but you should also consider which of these lists might contain dynamic data:

Is it enough to reload the list at application startup, every N days/hours/minutes, or would it be possible to send a signal from the updating process to trigger a reload only when needed?

Terje

"Returns the boolean value"

[Anonymous Coward]

I really hope this is a joke...

/**
* Returns the boolean value.
* @return boolean
* @param option java.lang.String
*/
public boolean isValidOption(String option) {
option = (option != null) ? option.trim() : option;
return optionsVector.contains(option);
}


The namings are horrible; If I'm using a variable I don't need to be reminded of its type. I certainly don't need to be reminded that my options vector is an object. I do need to know more about a method than its return type. And just to whine, I don't need to set the value of option to option if it is null and certainly don't need to check if my Vector contains it.

Re:"Returns the boolean value"

[Nasarius]

He doesn't even use the Java naming convention of putting an underscore in front of your instance variables. I spent a few seconds wondering where optionsVector came from, because I thought pretty much all Java coders followed the naming conventions.

And the rest of the comments are just as bad. If you're just going to restate the name of the method, please don't bother. What the fuck is "Return the list of values" supposed to mean?

what a steaming pile of shite

[MrBlender]

How much wanking about could you possibly do to make a cache.

What, The Invention of Cache?

[cowgoesmoo2004]

Thanks for the hot tip... I'm really anxious to see how the world changes now that the concept of the cache has finally been brought to light. Behold, sarcasm, I've invented sarcasm! Wow, maybe we should create a /. article for me?

Obligatory paraphrased Dilbert quote

[3770]

The son of the PHB: My dad always told me to eat the last pickle before drinking the pickle juice.

Dilbert: Something something something!

The son of the PHB: Do you have any idea how much it hurts if you get a pickle in your eye?

Hahahahah! Well, I guess you had to be there.

Point is, the article should have been in developers, not the main page.

Re:You answered my question b4 I asked, partially

[prionic6]

Although saving data in memory is useful, note that files get cached by the OS. Session handling in PHP is using files, too, and is "just fast enought" (TM pending).

An article about caching on a geek board?

[tjstork]

Come on. How can you call this concept even novel. Too much data to load from a slow process, so load it up front and cache it. I don't think you could call yourself a geek if you haven't bumbled into that one.

Come on guys, you can do better than this!

um, right...

[Pfhreakaz0id]

I agree with other comments. This is pretty basic, but then again you can't beleive some of the stuff people do SQL queries for. I started at one job, took over a web app that had a lot of forms with a "STATE" drop down. Every time they were going to a ref table to build the drop down. I gave them unbelievable crap.. "when's the last time we added a state to the U.S.?"

The point of stuff like this is, you can keep a dynamic page to generate the html and just make the damn thing an include. You can always regen the include if a reference table changes. Or build something fancier to check if data has changed and regen.

Re:um, right...

[Anonymous Coward]

Literacy is very basic stuff, too: BELIEVE me.

This time, shoot the messenger

[SuperKendall]

The guy who wrote this just wanted to show a nice concise summary of different things you can do to cache static data on startup but still keep it in a DB.

So please direct all flames (A) to whoever posted this as "News", and (B) to whoever accepted this "story".

Or perhaps Hemos said to himself "Damn, a lot of sites sure are slow latley. Perhaps they know not of caching." and then posted the story.

Database hits

[StormReaver]

The number of database hits isn't the whole story, and is really not all that important. My web+database apps are LAPP driven, which provides some help with caching frequently used data (such as drop down lists).

Linux uses all otherwise unused memory for file caching. My database machines are loaded with 3-4GB of RAM each, so there is lots of memory used for caching.

PostgreSQL uses shared memory caches to increase performance (though I don't know the exact nature of its shared memory usage), and it works very well. Well constructed joins on multiple tables with millions of rows each, with reasonable filter criteria, return instantly.

My PHP applications populate (on average) 8 drop down lists per page, after determine access rights, user preferences, etc via other queries.

When accessed from our LAN, page construction is instantaneous. Rather, page construction is limited mainly by the speed with which Firefox (and Konqueror) can render HTML. For all intents and purposes, it's instantaneous.

Many lists frequently change, so caching at the application level is probably not going to provide any benefits. Multiple database hits aren't necessarily problematic if the database and operating system's internal caching already does a good job.

Re:Database hits

[Samus]

It's good that your application performs well under your current load. However many times applications that perform well under minimal load perform terribly under higher loads. In your case you are grabbing a database connection and holding on to it far longer than you would need if you were able to cache even 50% of your list boxes 50% of the time. In server based applications you grab the resource when you need it and release it as soon as you can. You also try to prevent your app from doing as much work as possible. Caching can play a big role in this. It's a simple idea but can be tricky to implement, but well worth the time and effor for a high usage app. Slashdot does lots of caching. In fact much of their pages are pre-rendered. That's why you get the message that it may take a few minutes for your comment to appear after posting.
One other thing that I have seen performance wise related to usage. Sometimes an app works great even under high load when it is first rolled out. Then as time goes by it gets slower and slower because of the volume of data that gets accumulated. Think how long a table scan takes against a 100 row table vs a 1 million row table. Tuning your data access code and database can really have high payouts. I don't think I would rely solely on my OS to provide a cache for me. What happens when you need to scale out and your database is no longer on the same box as your web server? The data may be cached but on the wrong box. You'll still have to take a trip over the wire to get it. The difference between the memory bus and the network is like the driving the audobon and a driving thru a school zone.

Re:Database hits

[jbplou]

All modern RDMS's cache their data pages, be it SQL Server, Oracle, Sybase DB2 or one of the lesser database engines. Unless you have enough memory to keep all your data pages and index pages in memory you are going to be cycling pages out of the cahce. If you application server can hold the data in memory for a time period, even if for a relatively short time such as 1 or 2 minutes(big difference on a busy server) you will see better database and application performance, because your RDMS won't be wasting its memory on data that you know hasn't changed. It is always going to be faster to go to memory in your webserver to retrieve values than to go out to your database even if the database resides on the same machine as your webserver and it as the data pages you are looking for in cache. Your arguement about FIrefox or Konqueror render time is irrelvant, unless your users are using an antique machine or you are pushing lots of multimedia, render speed is meaningless because is basically instantaneous on IE, Mozilla, Netscape, Opera or whatever else browser you choose to use. My guess is that your servers are no where near saturation so caching doesn't matter to you.

that was quick and easy ???

[MrBlender]

If you implement the quick and easy approach I've described ...

Hate to see what happens when he is feeling creative

He needs a bigger moustache to hide behind.

Not as unnecessary as you think

[the_rev_matt]

You'd be surprised what a novel concept this is to a lot of web developers. Particularly ones that came from other languages and learned web dev on the job. A directive like this coming from IBM may actually penetrate the thick COBOL* muddled minds of a lot of less than stellar developers I've worked with.

* feel free to substitute VB here, as I've worked with young programmers who only learned VB in college for whom this approach is beyond their comprehension. Their normal solution is "More RAM".

Re:Not as unnecessary as you think

[Anonymous Coward]

Also feel free to substitute PHP. The last five years of web application design patterns is mostly lost on them.

Re:Not as unnecessary as you think

[jbplou]

VB.NET has a cache object for this specific use, I think a VB.NET programmer would know about this concept.

Vector has setSize, ArrayList doesn't

[irritating environme]

that's the only reason I've used Vector, aside from the minor synchronization distinction. Why doesn't ArrayList have setSize()? Anyone know this?

Re:Vector has setSize, ArrayList doesn't

[zettabyte]

My guess would be b/c the semantics of a List (aka a Sequence) would mean it is 'unbounded'. With that said, you can construct an ArrayList with an initial capacity, to keep the arraycopy's to a minimum:

// we have < 100 things to put in here
ArrayList foo = new ArrayList(100);

But I'm not sure what you're after so this might not help you.

php

[psbrogna]

You can do the same thing (caching frequently accessed data for dynamically generated pages) in PHP using the shared memory library. (SHMOP?) In regards to the sync issue one /.'r mentions below, you can poll for a TS that indicates the underlying SQL data has changed.

Data integrity

[coyote-san]

The reason you would maintain "static" data in the database is for data integrity.

For instance, the list of US states and Canadian provinces does not change very frequently. I think the last Canadian change was about a decade ago, the last US change nearly 50 years ago.

The "full name, abbreviation" name used in most pulldown lists (full name as label, abbreviation as value) can obviously be considered static.

So why keep it in a relational database? Simple - you can use it to provide referential integrity for all "state" fields in the rest of the database.

This isn't a huge deal with states, but it can be very important with domain level enumerations. Your form actions may be well-behaved, but a robust system must account for clowns who feed their own data directly into your action URL.

(As an aside, this isn't a theoretical problem. I've heard stories of people getting an order form for, oh, a laser printer. They capture it, change the price of the printer from $499.99 to $49.99 and submit the order. The action accepted it, and when the company attempted to refute it they lost because it was considered a bona fide negotiation since the web site could/should have been programmed to reject forms with altered prices. They made an offer, the client made a counteroffer, and the company accepted it. This depends on your state, etc. Given the current political climate I wouldn't be surprised to learn that this is now considered computer fraud with a 10 year prison sentence.)

Re:Data integrity

[CrackHappy]

I like the problem you give at the bottom.

Having created an online store application, I have struggled with security issues. If it weren't for those, it would have easily taken 1/2 the time that it did to create the application.

One of the main things I concentrated on for the back-end db piece was to verify the data that comes in against the DB itself, and it never accepts data from the end user on pricing (whether or not it's in the form or hidden in the form).

I built the application with two assumptions:
1) Never trust the client.
2) Be wary of the 3rd tier - you never know when it will become hacked.

Thankfully I've never had to deal with #2, but expect to someday! Access information for the DB isn't even stored on the 3rd tier, it's on the 2nd tier, and the 3rd tier has to ask the 2nd tier for access to query the 2nd tier. It's a relatively simple setup but seems to be effective.

That and those damned buffer overflows! ARGH.

It's a lot worse than just the Vectors

[attonitus]

There's the use of Hashtable; the non-thread-safe singletons; the traversal of the entire lists in ValidValuesTable.getOption; the unnecessary storing of the options and values string-arrays; the failure to pre-size the ArrayList; the fact that DropDownItems aren't immutable and, my favourite (repeated several times throughout the code):

private void loadStatusList() {
Hashtable obj_Hashtable = new Hashtable();
ListValuesDAO listValuesDAO = new ListValuesDAO();
obj_Hashtable = listValuesDAO.getListValues("STATUS");
...

I hope that IBM Global Services' QA for products is better than their QA for developerworks articles.

Am I the only one...

[frakir]

... who reads Reduce the number of database hits

as 'increase the number of database misses'
rather then 'reduce the number of database accesses'?

Caching found in early humans

[Anonymous Coward]

Yes folks.. when early humans realized that they needed information quickly they would write it down. Thus removing the need for sometimes long response times searching the brain.

This is news?!?

Re:You answered my question b4 I asked, partially

[the quick brown fox]

Note that the session storage module for mm does not guarantee that concurrent accesses to the same session are properly locked.

This should be read: "Do not use." Unsafe concurrent access is a great way to introduce critical bugs that are only found after the app is deployed to production servers. (Just ask any ColdFusion developer who has used sessions in CF 5.0 or earlier.)

Slashcache?

[rainmayun]

I wonder if Slashdot does caching of frequently posted stories.

Re:Slashcache?

[Anonymous Coward]

No, because that would required looking at stored data, which is obviously never done considering the number of dupes.

seems pretty self-evident

[whoisshe]

what *i'd* like to know is:

when presenting an HTML table, with a big drop-down in each row, is there some way to send that list to the client only once instead of numOfRows times?

Re:seems pretty self-evident

[rleibman]

Sure, send it down once inside javascript, have javascript load all of your dropdowns for you.

Next in the Series...

[brockbr]

"How to Use Semicolons To Avoid Compiler Errors"

Its just optimization

[crovira]

Rule 1: Never do anything you don't need to. (run time optimization)
Rule 2: Never do anything twice. (cacheing)
Rule 3: Empty cache that won't be required. (space optimization)

If you design properly, your software will run fast, stay fast and small.

Remember "Dr. Dobb's Journal of Computer Calisthenics and Orthodontia (Running Light Without Overbite"?

Its just as true now as it was back when you were writing for machines with K (not M) of RAM.

if you haven't done this for the last 8 years

[Anonymous Coward]

then you're a really crappy web app developer

Re:Almost ...can be viewed as an exercise in cachi

[TheLink]

Well much programming can be viewed as an exercise in data compression. You take lots of requirements and constraints with lots of redundancies and overlaps and compress it to code.

A smart programmer can do very good compression. A wise programmer will do it so that you can add likely stuff later and recompress without taking too much time (perhaps at the expense of lower compression).

Of course if it's bad programming it's an exercise in data loss, corruption or explosion ;).

Crazy like a fox

[VGR]

I have to wonder if the real purpose of this article's presence on Slashdot is to demonstrate how pitifully primitive the average programmer in the commercial world is.

If you're still in school, you may not be aware of this, but a huge percentage of people who program for money are no more competent than the article's author. Many are just discovering these basic (no pun intended) programming techniques for the first time.

This tells us two things:
1. You can get work even if you barely know a compiler from WINMINE.EXE.
2. If you choose to become a truly competent programmer, you will kick ass in the commercial world. The need for such people is great.

Re:Crazy like a fox

[t_allardyce]

If you choose to become a truly competent programmer, you will kick ass in the commercial world. The need for such people is great.

Not as great as you think; an over competent programmer, just like an over competent doctor or pilot is a bad thing. A company aims to hire someone who is competent enough for the job at hand but not so good that they're too expensive. Most products don't need to be the best and very often its in fact cheaper to throw extra memory or cpu power at some software rather than pay to write it better. But yes, its amazing what some people don't know or re-invent.

They've discovered caching

[Urusai]

Wow. Hoodathunk you could actually store data in an application's memory space?

Next article: How to persist cached data by writing it back to the database.

Underscores for instance variables

[metamatic]

Underscores for instance variables? Never heard of that one, and I've just checked The Elements of Java Style and can't find any reference to it.

Re:Underscores for instance variables

[curunir]

It's not part of the elements of style, but some people use it. I like it because it eliminates the need for the 'this' keyword in setters.

private int _something;
public void setSomething(int something) {
_something = something;
}

vs.

private int something;
public void setSomething(int something) {
this.something = something;
}

Also, it keeps you from having to track down an annoying type bug...

private int something;
public void setSomething(int soomething) {
this.something = something;
}

In the above case, the extra 'o' won't cause a compilation error, but the setter will mysteriously do nothing. If you prefix your instance variable with an underscore, you'll get a compile time error.

Re:Underscores for instance variables

[metamatic]

Well, FWIW Elements of Java Style says to use 'this.'.

Personally, I'm not a big fan of glyphs on variable names; though this brings to mind Ruby's use of @ and @@, and isn't all that offensive.

In my own code, I never reuse the same name for local and instance variables.

lamedot.org

[Anonymous Coward]

I hope that stories like this and the one about the "Juicer" that isn't, don't become common... lame. What's going on? Is there just not anything interesting coming in today, so they decided to fill some space with some boring stories?

Large corporate sweatshops

[Anonymous Coward]

This is just typical of your large corporate sweatshops. Think "Accenture", think "Avenade" (partly owned by Accenture) think "IBM GSA". Managers get pitched the premium sizzle for the IT steak and get lumped with a shed-load of graduate programmers that think the old farts, that take too long to do something, really don't know how "the new way" of doing things.

Yuck

[rkuris]

Writing hundreds of lines of code to improve performance leaves you with a maintenance nightmare in a few years.

I think this problem is a DATABASE problem. If the query is common enough, try caching it in memory on the database server. Something like the mysql query cache is your friend, not hundreds of lines of code!

Terribly sad! This is a horrible article.

[Anonymous Coward]

The article title sounds as if the author has figured out some amazingly cool trickery to avoid database hits, in reality the author is making a very specific monolithic mediocre serverside cache (not even a mediocre distributed cache or a clever highly performant local cache). The title of this article probably should have been "Populating HashTables using bad JDBC".

How did this get posted on Slashdot? How did this get posted on IBM? How did this get published at all? Why, oh why for the love of all that right and good, does this guy still have a job?

Ugh!

with Smarty...

[dickens]

With Smarty, and PHP, you could put the list in its own cached template. Check to see if the template is cached before displaying it and if not go to the database.

You can even use Smarty's html_options function to build your lists for you.

And when something changes the data for the list, just clear the cache. The next time it's needed the cache will be updated.

hash the SQL queries, redirect to a file if same

[Thundersnatch]

We run an off-the-shelf application that handles this problem nicely. Their database layer has a query object that takes a "cache timeout" value for each query issued.

This layer replaces all whitespace in the query with single spaces, then hashes the SQL query, and stores the results in the filesystem (as XML or plain text, depending on the nature of the result set(s)). The hash becomes the filename (e.g. "704A94757A794ABB8DEFCB48E86B97B1-cache.tmp"). Every time the same query is issued again, the cached file is used instead of a database connection, until the timeout age of the result set has passed. After the timeout has elapsed, the query is re-issued to the database, and the results are re-cached.

Now, this application has everything stored in the DB - the position of controls on forms, application logic, permissions, etc. This new caching system they recently added drastically reduced the load on our DB servers. By 50% or more. But all the flexibility of having DB-defined forms and workflow is still present.

This app is built in C# on .NET, and I'm not sure if these kewl caching features are built into ADO.NET or not. This is the only .NET app I've ever seen built with this sort of "universal" DB caching. But I think it's a very cool way to make an efficient but still entirely DB-driven app.

Seems like a bit of trouble to something...

[ClippyHater]

...that can be automated.

For instance, on some sites that I've developed, whenever an admin page is used to add to menus, it *does* update the DB, but it also recreates a static .php file that has the menus as .php code, and that file is included whenever the menu is used. No database access for pages that need the menus, no restarts necessary whenever the semi-static data is modified. Pretty much a win-win for all involved.

Or did I miss the point of the article?

Is this a real problem?

[Animats]

For this to be a useful optimization, you have to have a web page where 1) the web page really has to be generated from a database, 2) the options for the SELECT vary enough that they need to be retrieved from a database, 3) the specific page is loaded enough that cacheing is helpful, 4) the number of different pages with this issue is moderate, and 5) traffic is high enough that any of this matters. That combination isn't all that common.

The more likely situation is that the Java fanatics got carried away and generated every page dynamically, whether it needed to be or not, and as a result, the server is choking.

Some People Need Their Hands Held

[Anonymous Coward]

While this is obvious to anyone who bothers to actually care about design... it isn't so obvious to everyone. Perhaps I should explain.

For the past year or so I have had to come into a team and do hardcore J2EE development (not a problem I enjoy it). But the older developers are hopelessly clueless and out of date. (Fortunatly there are younger, saner heads... but age is a red herring.) They are under the impression that because they are using an Object Oriented (really object aspect, I believe) language, they are doing OOP.

This attitude basically results in things like this: JSPs that contain thousands of lines of Java, connect directly to the database (no pooling and right in with the HTML), cut and paste SQL code (the same code copied everywhere), and they all post and repost to themselves to do processing. They don't create any actual useful objects (say an Abstract Data Type) to encapsulate data. They think design patterns were just cooked up last year.

They are so far from understanding OO that when I mentioned something on Booch's books (Object Oriented Design Using Examples) where he states that:

"a concept qualifies as an abstraction only if it can be described, understood, and analyzed independently of the mechanism that will be used to realize it"

(I was trying to explain why a Servlet was an object, but that just using it did not make something OO even thought they tell management that they are all OOP experts.) They look at me like I am crazy and one of them even said that he thought that Booch didn't know anything about OO (ok alot of them are gonzo over the cancer that is PowerBuilder and those types usually have no clue about design or anything).

They are under the impression that everything I know about Software Engineering I learned in school (it is very hard to explain to them that I had a more theoritcal education, including grad school). So when I got to work away from the old guys doing an app the right way (Entity beans, SessionFacade, tiering, MVC, etc) they got mad and refused to use Java anymore (and I work at a very big company).

What is the point of this rant (and believe me there is much more, these guys are in the stone age)? Well, articles like this one may seem blatantly obvious ideas to /.ers, but to people wh never bothered to learn anything in their twenty years of development want you to hold their hands through everything. Age really is a red herring (I know alot of developers and many of them are much older than me and introduced me to some of this stuff... age has nothing to do with it).

This is a stupid article to put on /. but it exists for those old guys who never bothered to pick up a trade rag in the last twenty years (these guys spend most of their time oogling press releases from vendors).

Don't Load List Values for Improved Efficiency

[Sinner]

I recently wrote a cool search caching thing for a web application, to avoid re-allocating memory for the search results every time. When I was done, I looked to see how much memory I was saving: a cool 10k per request. Wow.

Given the potential for bugs in the code, I should probably trash the whole thing and replace it with an simpler implementation, but having spent 2 hours writing it I haven't the heart.

The moral of this story is what we all already know: avoid premature optimisation. The bottlenecks in your application aren't going to be where you're expecting anyway, so just choose the simplest implementation that could possibly work. If it turns out to be too slow or inflexible later, then fix it later.