Users often have objectives for the solution that can't be easily translated to requirements. These involve concepts like:
- adaptability
- manageability
- scalability
- usability
Since these objectives are nearly impossible to test - they are traditionally left out of the requirements. And the resulting solution might map well to the requirements documentation - but map poorly to the objectives.
The development team will then often walk away from that engagement declaring victory or "Mission Accomplished". And then when they discover that the user hates their solution, won't give them a reference, and won't re-engage for more work - then they have to come up with excuses like:
- we had poor requirements
- the users needs changed
- the users don't know what they want
What they're failing to recognize is that:
1. the users knew their objectives - but the developers process was incapable of using objectives
2. requirements gathering is the single most error-prone/defect-ridden, expensive, and inaccurate phase of the software life cycle.
3. users are are seldom industry experts/SMEs/Gurus/etc. They often only have experience with one instance of the business functionality that they are now providing objectives for. A process that relies upon them to be experts is bound to fail.
4. the business changes quickly, and some requirements can't be known until users get a chance to play with the result.
So, bottom-line is that requirements-oriented software development just doesn't work. About the only thing it might help you to do is to protect yourself in a civil liability case when your disgruntled customer sues. On the other hand, focusing on those requiremnts (the letter of the agreement) rather than the objectives (the spirit of the agreement) is probably why they are disgruntled in the first place.
I'll trade working prototypes for specs any day of the week. All too often large organizations focus on paperwork that nobody reads, nobody maintains, and nobody can really grasp. We'll document the hell out of a process, and while there's a ton of talk about it - it's usually clear that many people just don't 'get it'.
These projects often have an unusually high failure rate. You know the type:
- takes 9 months to get documents written, approved, corrected, approved again, updated for changed business, etc, etc.
- then management/staff changes and documents are turned over to a new guy that has to learn domain before he can complete them
- then two more months of playing with documents occurs
- then pallet-load of documentation is shipped overseas for another group to read
- this group asks NO questions, identifies standards compliance issues, but fails to understand project objectives
- then the project is canned
- meanwhile, the documents probably never really did a good job of expressing the user's objectives like adaptability, managability, scalability - but only hard requirements. And since the documents themselves aren't testable - they contain the most grievous errors that would have been encountered in the entire project. Had it not been canned.
Contrast this with an agile project (not necessarily XP) - with a focus on monthly deliverables. Each month the dev team can demonstrate that it is making forward progress, the user can actually play with the system and understand it far better, and since everyone knows that change will be frequent - they embrace it - rather than hide from it.
This is how my current project is going. Its predessor was paper-bound, cost millions, and died a pathetic death. This project began before the paper-project died, and began delivering value almost immediately. At about 5% of the cost of the paper project it has been embraced by the entire department, is adding a ton of value, and has a bright future.
Now, I'm not saying that there aren't potential pitfalls to be wary of in the agile world - but these are generally easily-managed compared to the pitfalls of the procedures-to-fix-procedures world.
Oh, and BTW. The study is grossly flawed. They show how Indian development projects have a higher rate of waterfall artifact delivery (arch specs, functional specs, detailed designs, etc) - and then also a much higher rate of delivery of some iterative practices (pair programming, etc). I suspect the real reason is that they are often assigning multiple extremely junior developers to projects - and hoping that in teams they can rescue one another. I've worked on four out-sourced Indian projects in which this occured: almost all developers were straight out of school, and could barely write any code at all. This wasn't true of all of them, and I'm sure that given sufficient time they will develop the skills. But - their pairing of developers and testers isn't due to good methodology - it's due to a desperate attempt to use under-skilled staff.
>>Or would you blame the workman who cuts off his arm with the buzz saw's totally unprotected blade?
> Yes, I would: he was obviously doing something with the saw that was inappropriate
Yeah, that's pretty much the line of the discredited 19th century factory-owners. They all insisted that worker injureries were due to carelessness on the part of the worker. Then people began to noticed that eventually almost all workers became completely disabled. About that time folks began to realize that a tool that requires you to be perfect 100% of the time is a flawed tool. Or a self-mutilation device, you pick.
And the same arguement keeps resurfacing, btw. Not just among factory owners trying to preserve maximum profitability. But also amoung techies trying to defend crappy products:
- RTFM
- can't get a printer to work with cups? must be a newbie blah, blah, blah
Then in the late 90s Usability and Information Architecture really took off. These guys saw a a few patterns:
- the jack-assed argument that difficulty with a tool was the user's fault - resulted in lost users, lost sales, lost revenue. Those who insisted on blaming the workman rather than the tool - thankfully went out of business.
- usability challenges also caused security vulnerabilities - when users couldn't figure out how to secure a device it became a liability to everyone. So, in this case the tool harmed the entire community not just one workman.
The only interesting thing in the above comment is that you actually got modded-up for repeating a completely discredited notion. Sigh, probably just a clever troll and I fell for it...
> Security does not belong in the database because it removes context of action.
aha
Well, rather than debate whether or not the data should be persisted in the context of its rules and actions (which doesn't work in reporting, btw), lets get down to what simply works.
You need to plan on including security awareness within each layer and component of your architecture. Since we've moved beyond client-server we seldom now authenticate & authorize individual users in the database. That's fine, but locking down the data should still be done in any database, even if the granularity is now at the application level rather than the individual.
The difference between today and ten years ago is that we're now mostly implementing database security at the application level of granularity.
Here's a trivial example: - 20 users are in ldap group finance
- all are also in ldap group report
- 10 are also in ldap group invoice - 4 userids are used by the database:
- userid invoice has ownership privs on invoice schema
- userid gl has ownership privs on general ledger schema
- userid fa has ownership privs on fixed-asset schema
- userid report has read privs on above schemas - The invoice application will authenticate users via ldap service - The invoice application will use ldap service to authorize users using either report or invoice group depending on what the user attempts to do - The application(s) will connect to the database using either appropriate database userid. This will be used by database to ensure that invoice application does not attempt to write to general-ledger schema, etc.
This isn't the end of your security concerns in the database. You've also got to lock down all your file systems, ensure that no users can create new database objects, etc, etc. That's also fairly straight-forward.
Anyhow, don't get all hung up about where security must go - or in the interests of some kind of misguided purity you'll end up leaving giant gaping holes in your defenses. Implementing security at the application level of granularity is both simple and effective. Take advantage of it.
> As for what MySQL does better? Thats easy speed.
Really? Without partitioning? Without automatically maintained materialized views?
Ah, what you probably meant to say is that mysql does a read-only index-oriented workload better on small hardware than its larger competitors.
I'll buy that.
But, it's nonsensical to think that mysql can scan 300 million rows faster than Oracle/DB2/Informix when you add partitioning into the picture. In that case they can often simply scan the 5% of the data needed.
Same situation with maintained materialized views. Doing both reporting and transactions on same database? Bet those reporting queries are beating up the database pretty badly! Materialized views can be just the ticket here.
Or, how about a mixed workload? Have you compared mysql to oracle on a server supporting 3000 inserts a second along with 20 users simultaneously reading from the database? No?
Or how about on large hardware? Can you spread a database across a cluster of servers and have them all cooperate on each query in parallel? Informix & DB2 could do that since the mid-90s. You can spread DB2 across a hundred 8-way servers in a fully parallel configuration.
Well then, I suppose you can say that for a subset of what databases are used for MySQL is very fast. But that's about it. But there's an entire set of other situations out there in which it comes in way behind the more sophisticated alternatives.
> A tuned query set is pretty much unportable no matter what DBMS you are using.
As long as you avoid premature optimization, this usually doesn't happen. Sure, at the extreme edges it does, but I'm usually only there for data warehouse aggregation or recursion.
> 3. A very clean and easy to use API, in a language that is universally understood and almost universally portable (C) for incorporation into other products > such as PHP, Perl, Python, Apache itself, and an endless amount of other applications.
Portable - only in the context of its ability to compile & link. Not very portable in the sense that its users require it.
MySQL has a nearly complete disregard for ANSI SQL standards - which results in the least portable syntax of any relational database I've used. This list includes:
* mysql
* postgresql
* oracle
* informix
* db2
* sql server
* sybase
Additionally, since it's the only entry on the above list that is missing or has limited support for transactions (yes I know - you can get it with its slower innodb file system now), views, etc - much of the sql written is brain-dead compared to other databases. So, one query in oracle/postgresql/etc can easily turn in three in mysql. This mapping queries between mysql and other databases causes considerable performance problems - since most other databases provide the best performance (in most cases) with given a single query to perform a unit of work.
So, if the non-ansi syntax isn't a big enough pain-in-the-butt, having to rewrite the queries and application to get reasonable performance often is. Amazingly, I've found that it's easier to convert an application from SQL Server to Oracle than from MySQL to Oracle.
Ah well. I suppose much of this will improve over time as they rewrite their engine to include more and more of that functionality that nobody wants.
But in all seriousness - there's not much to analyze about its popularity guys. It pretty much boils down to:
right place at the right time It owned the light, easy and free niche four years ago. It's a different ballgame with postgresql now, but mysql has built up quite the following in the meanwhile.
My original post sounded sufficiently contrarian that I figured I should post a follow-up:
Link to a site dedicated to the ultimate sturdy vehicle: http://www.binderbulletin.org/ There you'll find stories from hundreds (maybe thousands) of current owners of 25-40 year old vehicles.
If you don't believe that a 30 year old vehicle can provide reliable and affordable daily transportation - just ask these guys.
sometimes old technology kicks butt. I've got a pair of 70s IH Scouts that I bought for a few thousand dollars years ago.
They're now over 25 years old, are driven every day, and never break down (well almost).
Advantages
- initial cost was very low
- labor is cheap & easy
- parts are very cheap and readily available
- most components are extra-heavy-duty, and so last hundreds of thousands of miles
- seven passenger convertible
- can use it to pull stumps on the weekend then commute topless during the week!
- gets better mileage than a new truck
- more fun to drive than most new trucks
Disadvantages
- no cup-holders
- no airbags
- no cup-holders
- loud on the highway
- even with extra emissions equipment, it isn't as clean or efficient as a new economy-oriented vehicle.
And the best part? After a day of listening to vendors describe how their shiney new product has made everything we're using from 2003 so obsolete...getting into a vehicle designed in the early sixties that still outperforms many new vehicles on the road. Screw disposable, build something amazing and folks will use it for decades.
One of the biggest problems with off-shoring is that it is only cheaper for large projects using outdated waterfall methodologies.
Sure, I suppose that moving these offshore could save costs (aside from extra risk & time incurred in communication).
But the REAL issue is that these projects shouldn't be moved offshore - they should be killed. Then restarted with small agile teams. I suppose there might be a project somewhere that doesn't fit the agile model - though in 22 years of development, I haven't come across it yet.
So the answer to offshoring is actually to change the question - not whether to offshore, but whether offshoring can work at all in an agile project. The common wisdom is *not at all*.
> Please go an get a decent, non-language specific book on testing before reading and listening to this > article.
Sure usability, user acceptance, system, string, and integration testing are all valuable. But why can't developers without any knowledge of these start with built-in unit testing?
And keep in mind that few books on testing cover the fairly recent phenomenon of test-driven-development (or test-centered-development).
The use of built-in test harnesses and focus upon developing tests as documentation of requirements is probably the biggest single improvement in testing in twenty years.
In one fell swoop the debugger is rendered obsolete...
1. I've never had a problem with memory management in python. Can't say it doesn't exist, just never impacted my production applications.
2. Implementing great built-in test frameworks doesn't need to wait for memory management improvements. I'm seeing almost immediate pay-offs from this kind of built-in testing.
3. I'm implementing python as an alternative to java in large applications - with complete success. Easy to learn, easy to maintain, fast enough to handle millions of rows of data a day - what's not to like?
> The "problem" with mainframes is not so much that they are old, but that most of the applications didn't use relational databases.
DB2 was the second commercial database out there (following oracle by 1 year around 1983). It's been on mainframes since the very begining. I first starting working on relational databases in 1986 - and it was about 10 years before I finally began to meet a reasonable number of unix or windows developers with database experience. And oh yeah - I've gone through *hundreds* of resumes while hiring back in the 90s, so i'm not hallucinating here.
> If the applications used relational databases, then one could much more easily slowly replace > COBOL applications with a more pleasant language of implementation in piecemeal.
Hmmm, on an IBM mainframe you've got rexx (like a simple functional version of python), c, java, pl1, etc. All reasonable languages. And they've been there for a while - I wrote C in MVS back around 1992, and Rexx back around 1990. They all talk to DB2. And btw, you can develop good systems with JCL & COBOL it is a little challenging, but not impossible. And I've seen COBOL systems that were far easier to manage and use than their modern counterparts. Of course, much of this has to do with the skill of the developers.
> A mainframe could make an ideal web server because of its security and multi-processing > capabilities. If this is the case, then why is it not done often?
Reason #1: many mainframe shops are still running the software legacy from the early 90s. They don't have a linux lpar and old-timey protocols aren't ideal for this (often require expensive middleware). But for those organizations that set things up right, there's no real technical limitations.
Reason #2: simple economics. Web serves are the simplest applications out there - and it's awfully easy to deal with scalability & reliability through redundancy. They're probably the worst candidate for rehosting on a mainframe. Database servers, on the other hand, are good candidates.
However, in my experience the best candidates for hosting on a mainframe - are database standby servers. You can run a nearly infinite number of the things for nothing - since almost all are idle, and db2/oracle/sybase/postgresql/mysql/etc - all run just fine on suse/whatever linux on vm.
I'm sure it's a nice solution - but the reason for it probably has more to do with reliability than performance.
On big unix hardware these days (i'm talking large IBM/Sun/HP clusters) databases typically sling *billions* of rows around. The cost / transaction is far lower than a mainframe, and in my experience they are far faster and more scalable.
However, I don't think that any had the programmability or adaptability of the system 360.
The role of the 'system programmer' (the mainframe equivilent to today's 'system administrator') was created by the system 360. Prior to that every computer I remember lacked so much configurability - that it was a single-purpose machine.
you know, in some ways isp was way ahead of its time.
I remember back around '90 - 14 years ago that in diaglog manager I could create a complete database table update UI with all crud functionality (create/rename/update/delete) in about 150 lines of simple rexx code. That's with zero reuse.
Then I encapsulated that, it only required about 40 lines of actual code.
I've since worked in a lot of java shops and am accustomed to see a thousand lines of code for the same purpose - and everything is written from scratch.
Well, MVS & ISPF were pretty far behind in some ways, but Rexx & Dialog Manager are still ahead of java & j2ee in some ways. And that's from 15 years ago.
the only problem with users changing their minds - is the developers who insist that they don't.
These developers obviously are unaware that the single most unreliable & error-prone part of system development - is requirements gathering.
So, if you follow a waterfall methodology and deliver everything at once - you're almost guaranteed to have disgruntled users with major change requirements. If instead you provide them with a series of prototypes these problems are easily handled.
So, no - asking or expecting the users to understand the best way for their jobs to work is rediculous - since their jobs should be changing over time, and few have probably seen how it's done everywhere else anyway. The best thing is to develop adaptable systems in iterations - and go to work with the attitude that change is good.
Paper records will only help catch the most blatant of adjustments.
By distributing the shaving of hours - almost nobody will notice. And even if they do - you can chaulk it up to an accident or 'glitch' in the system - and almost everyone will believe you.
This is the same problem with paper voting receipts - they're almost completely useless. The only thing they help with is if the fraud is extremely blatant. But if that fraud is at all sophisticated, and just occasionally/randomly adjusts reality - it won't be noticed, and it won't be dealt with seriously.
Meanwhile, you've reduced your [payroll costs|voter results] by 10% with probably just 0.0001% chance of getting caught.
built a huge reporting system for a baby bell a few years back. Potential savings turned out to be up to be in the range of *hundreds* of millions / year - due to massive automation that system could perform.
Everything was going great - our little team got the entire system prototyped and working fine in six months. Going straight into production was mostly just a point of paperwork at that point. Total budget and project duration were both *far* under typical telco projects.
But then...
1.) New management team took over. Since IT stinks so badly at the telcos, they never deliver, and were terrified of high user expectations that we had developed. So, they tried to do everything to kill the project.
2.) Some functionality we planned to deliver was shifted to Bellcore (now Telcordia) - one of the most incompetant software shops in the world. We estimated a cost of $500k, Bellcore said $6 million.
3.) New alcoholic manager screamed at team members (just for the hell of it apparently) and forced us to switch from standard server platform to HP. Then spent much of year getting flown around to HP events at HP's cost.
4.) I left - tired of being screamed at by alcoholic. All other project members were also looking around.
5.) Telco discovered bookkeeping errors that required it to shelve all projects in process. Since purchasing & porting to HP slowed production down by 3 months - project wasn't in production yet. Was mothballed.
6.) Entire team let go.
*2 years go by*
7.) I find a posting in a job site for developers to work on system that sounds just like this one. I speak with consulting firm - yep, it's the *exact* same system, and they're trying to take it out of mothball. But they're paying an indian subcontractor almost nothing to do it. I tell them that I designed a vast amount of this complex system and knew it like the back of my hand. As the most valuable resource that they could possibly find - they offer me about $20/hour to join the team for 3 months.
I quickly decided that unpaid unemployment was better than the shoe-sales salary they were offering to fix the problem that their incompetence created.
JohnFluxx wrote: > Could you tell me why you would use stored procedures? > It just seems better to have another layer that handles that logic, seperate from the database. That way you can change databases easily.
Sure - that's a good approach - especially if you've got a product that you want to support on N databases. However, it's pretty difficult to implement a complex application with completely generic sql anyway. So, when working with complex apps - the use of simple stored procs can actually simplify porting.
But in a more typical situation -when you're developing an application for a single database, and your primary concern might be that the client may want to switch databases in the future (but it will still only run on one database product), *then* the case for stored procedures is much stronger.
Typically I'll implement stored procedures for four reasons:
#1 Parallel Development: stored procedures allow the database & application teams to develop in parallel. For example: as soon as the object model is created, the database team can approve it and commit to delivering stored procs that map to that spec. The same day the developers start writing code. While the developers are writing this code the dbas figure out the physical model and then map that to the procs. I've used this technique to often cut 3-5 weeks off project time-lines.
#2 Physical Model Adaptability: often in complex applications performance dynamics can change over time, requiring that the data model be tuned to handle new situations. With an abstraction layer of stored procedures the dbas are freed to easily make these modeling changes without significant interaction with the developers. This works better for some organizational structures, and even when the dba & programmer are on the same team, it still allows the one person with the greatest sql skills to perform the entire change - and does not require a team to perform impact analysis on both java and the database.
#3 Database Performance: again, in complex or performance-intensive applications, some queries can be extremely complex. However, the folks who are typically the best at tuning the queries are the dbas, not the developers. For example, I sometimes have to split a query into separate steps, with creation of a cartesian-product table as a first step, then joining against a few other tables as next steps, then pulling everything together in a 4-6th step. I can encapsulate that query behind the scenes and offer a relatively simple-looking table to the developers to work with. The programmer's best attempt at tuning their query took 2 minutes, and I got it down to 2 seconds - but there's no way that they can maintain the query I created. Other performance-benefits occur when the dba partitions data across a cluster, and includes logic to determine which node to run part of the query upon. This shouldn't be in the application layer - since it's very database system dependent.
#4 Miscellaneous - there are other potential benefits as well - such as keeping all the sql code where the dba can automate access plan creation, impact analysis, etc. Another misc benefit is in the creation of a security layer.
Of course, note that in none of the above scenarios am I recommending large or complex stored procedures. The kind I'm recommending are 99% sql - and easily port from one database to another.
But the problem with using java in the database is that the population of skilled developers and adminstrators of that solution drops to near-zero.
I know quite a few developers are experts with java, and quite a few with sql. But I only know one who's an expert at both.
Stored procedures are invaluable, though I mostly keep it simple. If I had a choice I'd go with a native impementation of python - something both easy to learn and powerful.
> For those Oracle lovers in the crowd, take a look at the benchmarks - MS SQL rules the lower and middle > ground. It would rule the high end except lack of platform has held it back.
um, which benchmarks would those be? www.tpc.org doesn't have many benchmarks for desktop-sized servers (which is where sql server really does beat oracle/db2/etc). And as far as it being held back by its platform - without any of the parallel features of oracle/db2, and without any of the partitioning features - it has zero chance at the high-end.
It's basically *years* behind either of those two databases. This has nothing to do with windows, it has everything to do with lack of high-end database features in sql server. Microsoft has done a good job of improving the database client UI and adding usability features to low-end database functionality. But it hasn't added the high-end functionality, nor has it really delivered a great UI (for example: the SQL Server GUIs all sort date columns alphabetically rather than cronologically).
> Yukon is going to kill Oracle in the middle space because of development features. Got news for ya, people pick databases for reasons other than development features.
The business press doesn't talk much about postgresql - but companies, vendors, and skilled database developers & administrators are.
It has plenty of momentum, and is clearly ready and getting used in a wide variety of applications now.
MySQL does have an incredible amount of momentum, more than most products out there. However, it'll have to be completely rewritten from the ground up before it really becomes a threat to commercial products. That will probably take years to get right. Postgresql on the other hand is just filling in feature gaps - and has a much more evolutionary upgrade path ahead.
Is it the best database for a linux or unix shop? Is it the best database for large reporting or search applications? Is it the best database for projects or companies with a small budget?
Ah, the answer to all of the above is 'no': - zero portability - parallelism and partitioning is primitive - licensing costs for a 4-way server can easily hit $100k, and in many configurations are more expensive than other top commercial products (db2 for example).
When it comes to prototyping, sql server is at the top of my list. However, when it comes to delivering powerful capabilities, automating operations, and scripting changes - then it's at the bottom of my list.
But I will agree with you on the.net stuff - integrating that into the database is a bad idea.
Sure Sun's in trouble - they optimized their business for the dot-coms, took advantage of the hype, and now everyone's far more interested in cheap than sexy.
But even their sexy servers aren't all that attractive anymore as the intel line gains more ground on them. And really, the the only reason for buying 24+ way SMPs was due to limitations in oracle clustering. And now they're moving away from that.
Java's fine - if you like heavy, ponderous languages. A few years ago I worked at a system integrator and performed a study of our productivity - it actually took us longer to create an application in J2EE/Oracle in 2002 than it did in CICS/DB2 in 1987. The only good thing I can say about java is that I suppose it's better for large application development than c or c++, and it runs on more platforms than.net. And I suppose it's just about the only language being taken seriously for large application development on open systems today.
So, now what's Sun left with? Overpriced hardware and cumbersome software. Should we be deferential with them because of all that they've done for us? Please - they spin so much hype it's disgusting, and their sales team is almost as sleazy as that of sybase or oracle.
Slashdot Mirror
Ok, but what about objectives?
Users often have objectives for the solution that can't be easily translated to requirements. These involve concepts like:
- adaptability
- manageability
- scalability
- usability
Since these objectives are nearly impossible to test - they are traditionally left out of the requirements. And the resulting solution might map well to the requirements documentation - but map poorly to the objectives.
The development team will then often walk away from that engagement declaring victory or "Mission Accomplished". And then when they discover that the user hates their solution, won't give them a reference, and won't re-engage for more work - then they have to come up with excuses like:
- we had poor requirements
- the users needs changed
- the users don't know what they want
What they're failing to recognize is that:
1. the users knew their objectives - but the developers process was incapable of using objectives
2. requirements gathering is the single most error-prone/defect-ridden, expensive, and inaccurate phase of the software life cycle.
3. users are are seldom industry experts/SMEs/Gurus/etc. They often only have experience with one instance of the business functionality that they are now providing objectives for. A process that relies upon them to be experts is bound to fail.
4. the business changes quickly, and some requirements can't be known until users get a chance to play with the result.
So, bottom-line is that requirements-oriented software development just doesn't work. About the only thing it might help you to do is to protect yourself in a civil liability case when your disgruntled customer sues. On the other hand, focusing on those requiremnts (the letter of the agreement) rather than the objectives (the spirit of the agreement) is probably why they are disgruntled in the first place.
I'll trade working prototypes for specs any day of the week. All too often large organizations focus on paperwork that nobody reads, nobody maintains, and nobody can really grasp. We'll document the hell out of a process, and while there's a ton of talk about it - it's usually clear that many people just don't 'get it'.
These projects often have an unusually high failure rate. You know the type:
- takes 9 months to get documents written, approved, corrected, approved again, updated for changed business, etc, etc.
- then management/staff changes and documents are turned over to a new guy that has to learn domain before he can complete them
- then two more months of playing with documents occurs
- then pallet-load of documentation is shipped overseas for another group to read
- this group asks NO questions, identifies standards compliance issues, but fails to understand project objectives
- then the project is canned
- meanwhile, the documents probably never really did a good job of expressing the user's objectives like adaptability, managability, scalability - but only hard requirements. And since the documents themselves aren't testable - they contain the most grievous errors that would have been encountered in the entire project. Had it not been canned.
Contrast this with an agile project (not necessarily XP) - with a focus on monthly deliverables. Each month the dev team can demonstrate that it is making forward progress, the user can actually play with the system and understand it far better, and since everyone knows that change will be frequent - they embrace it - rather than hide from it.
This is how my current project is going. Its predessor was paper-bound, cost millions, and died a pathetic death. This project began before the paper-project died, and began delivering value almost immediately. At about 5% of the cost of the paper project it has been embraced by the entire department, is adding a ton of value, and has a bright future.
Now, I'm not saying that there aren't potential pitfalls to be wary of in the agile world - but these are generally easily-managed compared to the pitfalls of the procedures-to-fix-procedures world.
Oh, and BTW. The study is grossly flawed. They show how Indian development projects have a higher rate of waterfall artifact delivery (arch specs, functional specs, detailed designs, etc) - and then also a much higher rate of delivery of some iterative practices (pair programming, etc). I suspect the real reason is that they are often assigning multiple extremely junior developers to projects - and hoping that in teams they can rescue one another. I've worked on four out-sourced Indian projects in which this occured: almost all developers were straight out of school, and could barely write any code at all. This wasn't true of all of them, and I'm sure that given sufficient time they will develop the skills. But - their pairing of developers and testers isn't due to good methodology - it's due to a desperate attempt to use under-skilled staff.
>>Or would you blame the workman who cuts off his arm with the buzz saw's totally unprotected blade?
> Yes, I would: he was obviously doing something with the saw that was inappropriate
Yeah, that's pretty much the line of the discredited 19th century factory-owners. They all insisted that worker injureries were due to carelessness on the part of the worker. Then people began to noticed that eventually almost all workers became completely disabled. About that time folks began to realize that a tool that requires you to be perfect 100% of the time is a flawed tool. Or a self-mutilation device, you pick.
And the same arguement keeps resurfacing, btw. Not just among factory owners trying to preserve maximum profitability. But also amoung techies trying to defend crappy products:
- RTFM
- can't get a printer to work with cups? must be a newbie
blah, blah, blah
Then in the late 90s Usability and Information Architecture really took off. These guys saw a a few patterns:
- the jack-assed argument that difficulty with a tool was the user's fault - resulted in lost users, lost sales, lost revenue. Those who insisted on blaming the workman rather than the tool - thankfully went out of business.
- usability challenges also caused security vulnerabilities - when users couldn't figure out how to secure a device it became a liability to everyone. So, in this case the tool harmed the entire community not just one workman.
The only interesting thing in the above comment is that you actually got modded-up for repeating a completely discredited notion. Sigh, probably just a clever troll and I fell for it...
> Security does not belong in the database because it removes context of action.
aha
Well, rather than debate whether or not the data should be persisted in the context of its rules and actions (which doesn't work in reporting, btw), lets get down to what simply works.
You need to plan on including security awareness within each layer and component of your architecture. Since we've moved beyond client-server we seldom now authenticate & authorize individual users in the database. That's fine, but locking down the data should still be done in any database, even if the granularity is now at the application level rather than the individual.
The difference between today and ten years ago is that we're now mostly implementing database security at the application level of granularity.
Here's a trivial example:
- 20 users are in ldap group finance
- all are also in ldap group report
- 10 are also in ldap group invoice
- 4 userids are used by the database:
- userid invoice has ownership privs on invoice schema
- userid gl has ownership privs on general ledger schema
- userid fa has ownership privs on fixed-asset schema
- userid report has read privs on above schemas
- The invoice application will authenticate users via ldap service
- The invoice application will use ldap service to authorize users using either report or invoice group depending on what the user attempts to do
- The application(s) will connect to the database using either appropriate database userid. This will be used by database to ensure that invoice application does not attempt to write to general-ledger schema, etc.
This isn't the end of your security concerns in the database. You've also got to lock down all your file systems, ensure that no users can create new database objects, etc, etc. That's also fairly straight-forward.
Anyhow, don't get all hung up about where security must go - or in the interests of some kind of misguided purity you'll end up leaving giant gaping holes in your defenses. Implementing security at the application level of granularity is both simple and effective. Take advantage of it.
> As for what MySQL does better? Thats easy speed.
Really? Without partitioning? Without automatically maintained materialized views?
Ah, what you probably meant to say is that mysql does a read-only index-oriented workload better on small hardware than its larger competitors.
I'll buy that.
But, it's nonsensical to think that mysql can scan 300 million rows faster than Oracle/DB2/Informix when you add partitioning into the picture. In that case they can often simply scan the 5% of the data needed.
Same situation with maintained materialized views. Doing both reporting and transactions on same database? Bet those reporting queries are beating up the database pretty badly! Materialized views can be just the ticket here.
Or, how about a mixed workload? Have you compared mysql to oracle on a server supporting 3000 inserts a second along with 20 users simultaneously reading from the database? No?
Or how about on large hardware? Can you spread a database across a cluster of servers and have them all cooperate on each query in parallel? Informix & DB2 could do that since the mid-90s. You can spread DB2 across a hundred 8-way servers in a fully parallel configuration.
Well then, I suppose you can say that for a subset of what databases are used for MySQL is very fast. But that's about it. But there's an entire set of other situations out there in which it comes in way behind the more sophisticated alternatives.
> A tuned query set is pretty much unportable no matter what DBMS you are using.
As long as you avoid premature optimization, this usually doesn't happen. Sure, at the extreme edges it does, but I'm usually only there for data warehouse aggregation or recursion.
> 3. A very clean and easy to use API, in a language that is universally understood and almost universally portable (C) for incorporation into other products
> such as PHP, Perl, Python, Apache itself, and an endless amount of other applications.
Portable - only in the context of its ability to compile & link. Not very portable in the sense that its users require it.
MySQL has a nearly complete disregard for ANSI SQL standards - which results in the least portable syntax of any relational database I've used. This list includes:
* mysql
* postgresql
* oracle
* informix
* db2
* sql server
* sybase
Additionally, since it's the only entry on the above list that is missing or has limited support for transactions (yes I know - you can get it with its slower innodb file system now), views, etc - much of the sql written is brain-dead compared to other databases. So, one query in oracle/postgresql/etc can easily turn in three in mysql. This mapping queries between mysql and other databases causes considerable performance problems - since most other databases provide the best performance (in most cases) with given a single query to perform a unit of work.
So, if the non-ansi syntax isn't a big enough pain-in-the-butt, having to rewrite the queries and application to get reasonable performance often is. Amazingly, I've found that it's easier to convert an application from SQL Server to Oracle than from MySQL to Oracle.
Ah well. I suppose much of this will improve over time as they rewrite their engine to include more and more of that functionality that nobody wants.
But in all seriousness - there's not much to analyze about its popularity guys. It pretty much boils down to:
right place at the right time
It owned the light, easy and free niche four years ago. It's a different ballgame with postgresql now, but mysql has built up quite the following in the meanwhile.
My original post sounded sufficiently contrarian that I figured I should post a follow-up:
Link to a site dedicated to the ultimate sturdy vehicle: http://www.binderbulletin.org/ There you'll find stories from hundreds (maybe thousands) of current owners of 25-40 year old vehicles.
If you don't believe that a 30 year old vehicle can provide reliable and affordable daily transportation - just ask these guys.
sometimes old technology kicks butt. I've got a pair of 70s IH Scouts that I bought for a few thousand dollars years ago.
They're now over 25 years old, are driven every day, and never break down (well almost).
Advantages
- initial cost was very low
- labor is cheap & easy
- parts are very cheap and readily available
- most components are extra-heavy-duty, and so last hundreds of thousands of miles
- seven passenger convertible
- can use it to pull stumps on the weekend then commute topless during the week!
- gets better mileage than a new truck
- more fun to drive than most new trucks
Disadvantages
- no cup-holders
- no airbags
- no cup-holders
- loud on the highway
- even with extra emissions equipment, it isn't as clean or efficient as a new economy-oriented vehicle.
And the best part? After a day of listening to vendors describe how their shiney new product has made everything we're using from 2003 so obsolete...getting into a vehicle designed in the early sixties that still outperforms many new vehicles on the road. Screw disposable, build something amazing and folks will use it for decades.
One of the biggest problems with off-shoring is that it is only cheaper for large projects using outdated waterfall methodologies.
Sure, I suppose that moving these offshore could save costs (aside from extra risk & time incurred in communication).
But the REAL issue is that these projects shouldn't be moved offshore - they should be killed. Then restarted with small agile teams. I suppose there might be a project somewhere that doesn't fit the agile model - though in 22 years of development, I haven't come across it yet.
So the answer to offshoring is actually to change the question - not whether to offshore, but whether offshoring can work at all in an agile project. The common wisdom is *not at all*.
> Please go an get a decent, non-language specific book on testing before reading and listening to this
> article.
Sure usability, user acceptance, system, string, and integration testing are all valuable. But why can't developers without any knowledge of these start with built-in unit testing?
And keep in mind that few books on testing cover the fairly recent phenomenon of test-driven-development (or test-centered-development).
The use of built-in test harnesses and focus upon developing tests as documentation of requirements is probably the biggest single improvement in testing in twenty years.
In one fell swoop the debugger is rendered obsolete...
A few thoughts:
1. I've never had a problem with memory management in python. Can't say it doesn't exist, just never impacted my production applications.
2. Implementing great built-in test frameworks doesn't need to wait for memory management improvements. I'm seeing almost immediate pay-offs from this kind of built-in testing.
3. I'm implementing python as an alternative to java in large applications - with complete success. Easy to learn, easy to maintain, fast enough to handle millions of rows of data a day - what's not to like?
> The "problem" with mainframes is not so much that they are old, but that most of the applications didn't use relational databases.
DB2 was the second commercial database out there (following oracle by 1 year around 1983). It's been on mainframes since the very begining. I first starting working on relational databases in 1986 - and it was about 10 years before I finally began to meet a reasonable number of unix or windows developers with database experience. And oh yeah - I've gone through *hundreds* of resumes while hiring back in the 90s, so i'm not hallucinating here.
> If the applications used relational databases, then one could much more easily slowly replace
> COBOL applications with a more pleasant language of implementation in piecemeal.
Hmmm, on an IBM mainframe you've got rexx (like a simple functional version of python), c, java, pl1, etc. All reasonable languages. And they've been there for a while - I wrote C in MVS back around 1992, and Rexx back around 1990. They all talk to DB2. And btw, you can develop good systems with JCL & COBOL it is a little challenging, but not impossible. And I've seen COBOL systems that were far easier to manage and use than their modern counterparts. Of course, much of this has to do with the skill of the developers.
> A mainframe could make an ideal web server because of its security and multi-processing
> capabilities. If this is the case, then why is it not done often?
Reason #1: many mainframe shops are still running the software legacy from the early 90s. They don't have a linux lpar and old-timey protocols aren't ideal for this (often require expensive middleware). But for those organizations that set things up right, there's no real technical limitations.
Reason #2: simple economics. Web serves are the simplest applications out there - and it's awfully easy to deal with scalability & reliability through redundancy. They're probably the worst candidate for rehosting on a mainframe. Database servers, on the other hand, are good candidates.
However, in my experience the best candidates for hosting on a mainframe - are database standby servers. You can run a nearly infinite number of the things for nothing - since almost all are idle, and db2/oracle/sybase/postgresql/mysql/etc - all run just fine on suse/whatever linux on vm.
> Impressive, no?
I'm sure it's a nice solution - but the reason for it probably has more to do with reliability than performance.
On big unix hardware these days (i'm talking large IBM/Sun/HP clusters) databases typically sling *billions* of rows around. The cost / transaction is far lower than a mainframe, and in my experience they are far faster and more scalable.
However, they simply aren't as reliable either.
However, I don't think that any had the programmability or adaptability of the system 360.
The role of the 'system programmer' (the mainframe equivilent to today's 'system administrator') was created by the system 360. Prior to that every computer I remember lacked so much configurability - that it was a single-purpose machine.
you know, in some ways isp was way ahead of its time.
I remember back around '90 - 14 years ago that in diaglog manager I could create a complete database table update UI with all crud functionality (create/rename/update/delete) in about 150 lines of simple rexx code. That's with zero reuse.
Then I encapsulated that, it only required about 40 lines of actual code.
I've since worked in a lot of java shops and am accustomed to see a thousand lines of code for the same purpose - and everything is written from scratch.
Well, MVS & ISPF were pretty far behind in some ways, but Rexx & Dialog Manager are still ahead of java & j2ee in some ways. And that's from 15 years ago.
the only problem with users changing their minds - is the developers who insist that they don't.
These developers obviously are unaware that the single most unreliable & error-prone part of system development - is requirements gathering.
So, if you follow a waterfall methodology and deliver everything at once - you're almost guaranteed to have disgruntled users with major change requirements. If instead you provide them with a series of prototypes these problems are easily handled.
So, no - asking or expecting the users to understand the best way for their jobs to work is rediculous - since their jobs should be changing over time, and few have probably seen how it's done everywhere else anyway. The best thing is to develop adaptable systems in iterations - and go to work with the attitude that change is good.
Paper records will only help catch the most blatant of adjustments.
By distributing the shaving of hours - almost nobody will notice. And even if they do - you can chaulk it up to an accident or 'glitch' in the system - and almost everyone will believe you.
This is the same problem with paper voting receipts - they're almost completely useless. The only thing they help with is if the fraud is extremely blatant. But if that fraud is at all sophisticated, and just occasionally/randomly adjusts reality - it won't be noticed, and it won't be dealt with seriously.
Meanwhile, you've reduced your [payroll costs|voter results] by 10% with probably just 0.0001% chance of getting caught.
built a huge reporting system for a baby bell a few years back. Potential savings turned out to be up to be in the range of *hundreds* of millions / year - due to massive automation that system could perform.
Everything was going great - our little team got the entire system prototyped and working fine in six months. Going straight into production was mostly just a point of paperwork at that point. Total budget and project duration were both *far* under typical telco projects.
But then...
1.) New management team took over. Since IT stinks so badly at the telcos, they never deliver, and were terrified of high user expectations that we had developed. So, they tried to do everything to kill the project.
2.) Some functionality we planned to deliver was shifted to Bellcore (now Telcordia) - one of the most incompetant software shops in the world. We estimated a cost of $500k, Bellcore said $6 million.
3.) New alcoholic manager screamed at team members (just for the hell of it apparently) and forced us to switch from standard server platform to HP. Then spent much of year getting flown around to HP events at HP's cost.
4.) I left - tired of being screamed at by alcoholic. All other project members were also looking around.
5.) Telco discovered bookkeeping errors that required it to shelve all projects in process. Since purchasing & porting to HP slowed production down by 3 months - project wasn't in production yet. Was mothballed.
6.) Entire team let go.
*2 years go by*
7.) I find a posting in a job site for developers to work on system that sounds just like this one. I speak with consulting firm - yep, it's the *exact* same system, and they're trying to take it out of mothball. But they're paying an indian subcontractor almost nothing to do it. I tell them that I designed a vast amount of this complex system and knew it like the back of my hand. As the most valuable resource that they could possibly find - they offer me about $20/hour to join the team for 3 months.
I quickly decided that unpaid unemployment was better than the shoe-sales salary they were offering to fix the problem that their incompetence created.
JohnFluxx wrote:
> Could you tell me why you would use stored procedures?
> It just seems better to have another layer that handles that logic, seperate from the database. That way you can change databases easily.
Sure - that's a good approach - especially if you've got a product that you want to support on N databases. However, it's pretty difficult to implement a complex application with completely generic sql anyway. So, when working with complex apps - the use of simple stored procs can actually simplify porting.
But in a more typical situation -when you're developing an application for a single database, and your primary concern might be that the client may want to switch databases in the future (but it will still only run on one database product), *then* the case for stored procedures is much stronger.
Typically I'll implement stored procedures for four reasons:
#1 Parallel Development: stored procedures allow the database & application teams to develop in parallel. For example: as soon as the object model is created, the database team can approve it and commit to delivering stored procs that map to that spec. The same day the developers start writing code. While the developers are writing this code the dbas figure out the physical model and then map that to the procs. I've used this technique to often cut 3-5 weeks off project time-lines.
#2 Physical Model Adaptability: often in complex applications performance dynamics can change over time, requiring that the data model be tuned to handle new situations. With an abstraction layer of stored procedures the dbas are freed to easily make these modeling changes without significant interaction with the developers. This works better for some organizational structures, and even when the dba & programmer are on the same team, it still allows the one person with the greatest sql skills to perform the entire change - and does not require a team to perform impact analysis on both java and the database.
#3 Database Performance: again, in complex or performance-intensive applications, some queries can be extremely complex. However, the folks who are typically the best at tuning the queries are the dbas, not the developers. For example, I sometimes have to split a query into separate steps, with creation of a cartesian-product table as a first step, then joining against a few other tables as next steps, then pulling everything together in a 4-6th step. I can encapsulate that query behind the scenes and offer a relatively simple-looking table to the developers to work with. The programmer's best attempt at tuning their query took 2 minutes, and I got it down to 2 seconds - but there's no way that they can maintain the query I created. Other performance-benefits occur when the dba partitions data across a cluster, and includes logic to determine which node to run part of the query upon. This shouldn't be in the application layer - since it's very database system dependent.
#4 Miscellaneous - there are other potential benefits as well - such as keeping all the sql code where the dba can automate access plan creation, impact analysis, etc. Another misc benefit is in the creation of a security layer.
Of course, note that in none of the above scenarios am I recommending large or complex stored procedures. The kind I'm recommending are 99% sql - and easily port from one database to another.
But the problem with using java in the database is that the population of skilled developers and adminstrators of that solution drops to near-zero.
I know quite a few developers are experts with java, and quite a few with sql. But I only know one who's an expert at both.
Stored procedures are invaluable, though I mostly keep it simple. If I had a choice I'd go with a native impementation of python - something both easy to learn and powerful.
> For those Oracle lovers in the crowd, take a look at the benchmarks - MS SQL rules the lower and middle
> ground. It would rule the high end except lack of platform has held it back.
um, which benchmarks would those be? www.tpc.org doesn't have many benchmarks for desktop-sized servers (which is where sql server really does beat oracle/db2/etc). And as far as it being held back by its platform - without any of the parallel features of oracle/db2, and without any of the partitioning features - it has zero chance at the high-end.
It's basically *years* behind either of those two databases. This has nothing to do with windows, it has everything to do with lack of high-end database features in sql server. Microsoft has done a good job of improving the database client UI and adding usability features to low-end database functionality. But it hasn't added the high-end functionality, nor has it really delivered a great UI (for example: the SQL Server GUIs all sort date columns alphabetically rather than cronologically).
> Yukon is going to kill Oracle in the middle space because of development features.
Got news for ya, people pick databases for reasons other than development features.
The business press doesn't talk much about postgresql - but companies, vendors, and skilled database developers & administrators are.
It has plenty of momentum, and is clearly ready and getting used in a wide variety of applications now.
MySQL does have an incredible amount of momentum, more than most products out there. However, it'll have to be completely rewritten from the ground up before it really becomes a threat to commercial products. That will probably take years to get right. Postgresql on the other hand is just filling in feature gaps - and has a much more evolutionary upgrade path ahead.
Really?
.net stuff - integrating that into the database is a bad idea.
Is it the best database for a linux or unix shop?
Is it the best database for large reporting or search applications?
Is it the best database for projects or companies with a small budget?
Ah, the answer to all of the above is 'no':
- zero portability
- parallelism and partitioning is primitive
- licensing costs for a 4-way server can easily hit $100k, and in many configurations are more expensive than other top commercial products (db2 for example).
When it comes to prototyping, sql server is at the top of my list. However, when it comes to delivering powerful capabilities, automating operations, and scripting changes - then it's at the bottom of my list.
But I will agree with you on the
Sure Sun's in trouble - they optimized their business for the dot-coms, took advantage of the hype, and now everyone's far more interested in cheap than sexy.
.net. And I suppose it's just about the only language being taken seriously for large application development on open systems today.
But even their sexy servers aren't all that attractive anymore as the intel line gains more ground on them. And really, the the only reason for buying 24+ way SMPs was due to limitations in oracle clustering. And now they're moving away from that.
Java's fine - if you like heavy, ponderous languages. A few years ago I worked at a system integrator and performed a study of our productivity - it actually took us longer to create an application in J2EE/Oracle in 2002 than it did in CICS/DB2 in 1987. The only good thing I can say about java is that I suppose it's better for large application development than c or c++, and it runs on more platforms than
So, now what's Sun left with? Overpriced hardware and cumbersome software. Should we be deferential with them because of all that they've done for us? Please - they spin so much hype it's disgusting, and their sales team is almost as sleazy as that of sybase or oracle.