Slashdot Mirror
Monday, The Death of Websites
An anonymous reader writes "Developers implementing 'weekend inspiration' are more dangerous than hackers.
Vnunet.com has this article about how eager developers and administrators create more troubles than hackers and viruses do for websites. How about those of us who start the week with a cup of coffee and the morning online-news? My inspiration and new ideas for development are definitely not the cause of the Monday-crash hour ... I think."
You all died? Where are all the posts?
Does this mean I shouldn't expect anymore karma?
Is this truly the only Earth I can live on?
No boobies for you!
And if slashdotting causes more downtime than developer mistakes, couldn't one argue that interesting content is more harmful than bad code for website uptime?
Quack!Quack!.....QUACK!!
Has anyone done any sort of bandwidth study looking at sites like etrade and yahoo, for purposes of determining any correlation between bandwidth consumption and movement on the stock markets? Intuition says that Monday mornings ought to see some sort of correlated spike.
I log in, the story is a few hours old, and there are 4 posts. Slashdot implementing the theory?
Ok, I give up, why you?
The article suggests that developers come back from their weekends and start fiddling with websites, but I think this last paragraph is perhaps equally or more accurate. Managers get "inspired" over the weekend just as much as code writers.
Reminds me of the BBS days. Usually a few hours after the SysOp leaves on vacation, the BBS is guaranteed to go down.
This is nothing but unprofessional development - the old "Oh, this is soooo good and sooo simple, how can it possibly cause troub..... ".
/. crew this: while their spelling may be atrocious, their grasp of grammer poor, and their fact and dup checking next to non-existent, they will put major changes to the codebase into Banjo first, then after they've been abused put them into the main /. site.
Any codebase, be it a program, a web site, or a router's firewall rules, should be changed IN TEST FIRST! Then you do your best to break it, and only after you and several others have had at it do you move it to production/HEAD/whatever (and hold your breath).
If you had a wonderful idea over the weekend, GREAT! Implement it in a test branch, try it out, and then move it to production. But if you merge it into the mainline without testing you are not acting professionally.
I will give the
At least, some of the time.
www.eFax.com are spammers
OMFG I broke teh Intarweb this morning!
IAALS.
In any properly managed environment, developers don't get to [i]touch[/i] the production environment. If they do, it should be read-only. All changes are made in the dev environment (developers can do what they want), put into test (developers are seriously limited), and then finally into production. Prod should be a physically separate set of servers from dev & test.
If stuff breaks on Mondays, either someone is skipping steps, or there is more going on.
My server
Sounds alot more like lack of a proper devlopment environment to me.
I mean its easy for it to happen. We had problems like this with our monitoring system (tho it was manic friday where someone would attemtp to impliment something before the weekend because of course, the weekend is when you want pages the least so you want to get anything that causes false pages fixed on friday to maximize enjoyment of the weekend)
Now we have development and test servers where things live BEFORE they go production. I never had any idea that it would help so much until we finnaly implimented it.
-Steve
"I opened my eyes, and everything went dark again"
I guess these sites don't test anyting. Maybe they are talking about small sites. I work for a big car company. We have three stages of testing.
:-)
I'm not saying the artical is wrong. The developers are still the biggest problem with our web site. It just doesn't always happen on Monday. Some times it takes tell Wednesday to get through the system.
There are 10 type of people in the world, those who understand binary and those who don't.
Just a thought: The rest of the world lumps all of us IT people together; the distinction between, say, a "developer" and "sysadmin" means nothing to my non-geek friends.
I don't think stuff like this happens often to sysadmins or DBAs. How often do you come into work on a monday and decide to migrate to xfs because you read on slashdot over the weekend that SGI ported it to linux, and SGI is cool? Likewise, how often does an Oracle DBA decide on Monday to move some production tablespaces over to rawfs from cooked, because she read a whitepaper from Oracle on Saturday that talked about performance increases from raw filesystems?
I've written a lot of code, and also sysadmin'd an awful lot of servers, and in my experience probably 90% of "production outages" are software changes--exactly like the article said--poor change control, etc etc. So, what's the point of dynamic multipathing, patching, dual power supplies, etc etc, when most problems occur because someone got excited and forgot a semicolon somewhere?
Is it fair to say that sysadmins fix things and developers break them? What is different about a software engineer's brain than a systems engineers? Talk amongst yourselves :)
if developers hurt a website more than viruses, then surely sites with more developers crash more, and sites with fewer developers crash less. Thus, the site with the most developers working on it has the most problems, and all sites with 1 developer have incredibly few problems. My personal site doesnt have enough complicated stuff on it to really "crash" per se, so obviously less developers means less bugs in that way. So, whose site has the most programmers working on it? Hmmm.... of course, the largest software company!
stuff |
or do those developers need to possibly develop on a copy of the website not accessible to the public? I mean, it's not hat hard to hit cp -R and transfer your updated functioning website to the primary directory...Maybe I'm the only one that doesn't tinker with things that people are hitting as I type.
On retail and B2B sites, Monday is usually a busy day. Customers are rolling into their offices with articles to read, facts to research and stuff to buy. Out of the seven days of the week, it'd be the worst for making a change.
On the other hand, I'm not sure incremental development is that much worse than large releases. You're either releasing a bug or two a week or waiting eight weeks to release all your bugs at one time.
Monday, The Death of Websites
/. subscriber!
Yeah... Unless you are a
The tone of the article talks about shoot-from-the-hip developers acting irresponsibly, on impulse. They're taking a recognized and thoughtful practice and painting it as irresponsibility.
Monday is the best time to implement changes to most sites. The irresponsible coder implements on Friday, when errors might not be caught, or fixed, until the next working day, after a full weekend of downtime, bugginess, or insecure behavior.
But that wouldn't make for an interesting story. News flash: updating code often results in bugs that need to be fixed. When do the authors suggest we roll out new versions?
Kevin Fox
What about SUPPORT sites that continually change their links, I think that if there ever needs to be a LAW, it is regarding support websites. Front upper LEFT, a STATIC neverchanging link to PHONE SUPPORT. (With the real phone numbers)Right under that a RMA link, again STATIC, then and only then can the site talk about products!
I hate manufacturers who let their "webmaster" change links that people have come to depend upon. They are not webmasters, they are webBASTARDS.
i tend to spend the weekend trying to think of excuses to avoid doing any work on monday morning, somehow i figured other people did the same thing
p.s. our website hasn't gone down in 2 years, go figure
Hoist Number One and Number Six.
While working for a large nameless Telecoms Company,
I and my fellow Contractors had an unwritten rule to "hold off" on all "good" ideas generated in meetings etc on Monday & Friday. Almost inevitably they would
all be canceled within a couple of days. Not subjecting ourselves to post/pre weekend madness saved ourselves a ton of work and helped us bring the project in on time!!
My theory is much more simple. Everyone on the Western half of the planet is going back to work and they really don't want to be working, so they *gasp* - hit the internet. I also believe people are more likely to be home at New Year's and Christmas in addition to the developers.
Well thought out article. I put less thought in my article, which is why it is at Slashdot.
This space for rent.
In my experience we have had more problems with several of our intranet sites then internet sites. They just update the content on the internet websites whereas the intranet site they are always trying to make enhancements.
Losers whine about doing their best
Winners go home and f*ck the prom queen!
They talk as if the developer has an idea over the weekend, then comes in Monday morning and implements this idea without any testing. But if that were true, the websites would crash Tuesday. I mean, really, how many of you think these guys are really making the changes Monday morning and the websites are thus breaking Monday morning? Any changes you see Monday morning were loaded over the weekend, and are probably the result of all last-week's work. Whatever ideas anyone got over the weekend will be coded and tested this week and installed next weekend; they won't show up until next Monday at the earliest.
If all this should have a reason, we would be the last to know.
Log in.
Cup of coffee.
Browse online forums.
Read witty remark.
C|N>K
Change keyboard. Curse profusely.
Stéphane "Alias" Gallay
Now, where did I put this witty quote?..
I, as IT director, would fire my IT staff if they pulled this. Considering that I have some systems with uptimes in YEARS, a few going on a DECADE, and over-all the _entire_ network has worked 24x7 for the last 10 years. Our business operations isn't even Internet based (we just happen to use it -- primarily for email) and the operations of the systems isn't life-critical. We just like our computers/networks to work.
Of course I'm the one that implemented a testing domain (live on the Net) for just such purposes. NEVER, NEVER, NEVER "test" anything on a production system. I can't even think of any installations that were not tested for MONTHS "offline" before being implemented. When the day comes to install there usually aren't any shockers either. It just works.
Of course I'm the one that's NEVER allowed a Windows server to even be a consideration. "Are you NUTS?"
that is why everything they promised by friday (but havn't tested yet) is not installed until monday morning. that way they can relax on the weekend.
i know i NEVER update a customers site on friday (especially not on friday afternoon).
I'm going to start with pointing out that the first sentence of the article said "UK websites", not "US". Obviously that means the people across the Pond need to work on this.
And what a surprise that when people roll out changes sometimes things break. Oh My God. Have you cured cancer yet?
And I'd say more often than not the "problems" on Monday are caused by bug fixes that developers are rushing on to production to fix bugs that were found over the weekend. And, as we all know, sometimes bug fixes skip QA...
Seriously, most places I work have Go Live set to be Monday, or, more often, Tuesday. You go live when you have already tested it; its gone through QA; and you're sure the staff is there. Tuesday is the better date in order to deal with key people taking long weekends, and it gives you two or three days to fix issues before the next weekend. Besides, Mondays are already hellish without adding "release new version" to the list of torments.
Seems to me we are talking about several different things here.
First of all, presumably it is a good thing that people think, and get inspiration. Mon-Fri 9-5 is not the best time for thinking - this is the time for meeting deadlines, sitting in meetings, answering the phone, putting out fires, and so on. The only time most of us have to actually sit and think is the weekend. Personally, I think that should be encouraged.
The next step is implmenting what you have dreamt up. Obviously, most ideas fail - ask any patent officer. And obviulsly, implmenting a new idea without checking with colleagues, drawing it 0ut in a spec, getting that spec approved, then protoyping, testing, tuning is not ideal either. These procedures were invented for good reasons - not just to constrain the creative mind. This is where most developers fail - not in coming up with ideas, but in being disiplined in implmenting them. I hear "we cannot plan ahead, it does not work like that for us" all the time from my developers - this is always a misconception, and seems to me simply a combination of inexperience, laziness and inability... nothing that cannot be fixed!
Michael
---
BDOS ERR ON A:>
Sheesh, next thing you know they'll start spouting nonsense like "burning the midnight oil leads to more bugs."
http://pages.ebay.com/catindex/motorcycles.html as of 1246 pm most of the links on ebay motors section dont work! GO MONDAY!
This type of thing might happen on small websites, where developers work on code, unit test, then publish. But any large code-base will have a cycle where things are tested first, and then rolled. Typically these rolls are scheduled for the best possible time, which often is monday morning. Everyone is in house, and you've got a whole week to fix anything that went wrong.
M@
Krispy Cream is people
When I was responsible for the Internet site of a rather large national bank, we only accepted change requests for Tuesday and Thursday mornings. It was just easier for the operators to get hold of a sober developer/administrator at 02:00 on a Tuesday or a Thursday than any other time. And getting a contact on the business side to ok a rollback that caused contract issues on the weekend was near impossible.
It doesn't matter if the website was made on saturday , or wensday. anytime a website comes out with new code, its going to fuck up in the first few days. there is just no cheap way to test a website with a full load of users all with difrent OS's, web browsers, and connections. how many times has slashdot craped out when they update the slashcode
Having been working on a company that grew from a 1999 Internet startup with 5 employees (me being the only programmer to work along two consultants) to a profitable Internet company with 40 employees in 2003 (inlcuding the two former consultants), I've seen quite a bit of change in the IT procedures.
:-(
We have an 8 people tech team now (manager, programmers, support, QA). Whereas before we programmers would just use a development environment somewhat similar to the production (live) environment, test it a bit, deploy at will and monitor if anything went wrong, things have progressed a lot. Now we develop on a development environment as close to the production one as possible, then this is released to a test environment (also as close as possible to the production one) to be tested by QA, and that is finally released on the production (live) environment after it all tests ok (including regression testing).
Moreover, all the code changes are now under CVS, and we have automatic tools for monitoring the site, emailing errors, etc. QA is also done by separate people. IMHO it is conceptually flawed to allow the developers to do the final testing, by definition. (Though of course this is not always possible for cost reasons, it should be a goal).
The quality of our site is much better now. Problems almost always only arise when people want to bypass QA or force things through for emergencies.
IMHO, what is needed is:
1. Professionalism by the developers.
2. Testing, testing, and testing -- by the developers.
3. QA, QA and QA -- by someone other than the developers!
4. Managers must know the test/ QA process should never by bypassed -- this unfortunately is probably the hardest point.
I taught a couple of ecommerce classes for MBA students and had them actually do hands on development (in a limited sense of course) so they could get an appreciation of this process. Hopefully if some of them are managers they will remember that and not try to shortcut the due process.
/* TAANSTAFL */
Sounds like someones got a case of the Mondays.
How about web designers that choose formats that make their websites completely unusable to non-standard browsers?
vnunet
idg
.. Have never defaced my site with the goatse guy nor deleted critical files. They may not work great at first (or at all) but they're in no way malicious. More dangerous than hackers or virii? I think not.
Everyone is entitled to their own opinion. It's just that yours is stupid.
this is when developers come in and implement ideas they had over the weekend.
:-) Sure it's a cool idea, but I think Slashdot would end up Slashdoting itself!
I don't know about some of these development teams they are talking about but around here, you don't just implement "ideas" you might have had over the weekend. "Hey! Wouldn't it be cool if it did this... !" If it's not a requirement, it doesn't go in.
If that were the case... Wouldn't it be cool if Slashdot loaded a random pr0n image with every article posting!
Ascalante: Your bride is over 3,000 years old.
Kull: She told me she was 19!
"Looks like someone has a case of the Mooondays"
There must be a course somewhere for developers - how to piss-off sysadmins. Highlights:
1. Make changes last thing on a Friday.
2. Or before a 2 week holiday
3. Change Management does not apply to developers
4. CVS is for wimps
5. And if you must use CVS, wait a week before committing fixed code.
5. Don't bump version numbers
6. Don't update init scripts
7. Ecept if they are correct
8. If anyome is aware of what you are upto... go to lunch.
Do you mind, your karma has just run over my dogma.
When you make changes to websites, they sometimes break. Anytime you introduce change into a stable system, you open the door for instability.
And generally business websites don't see as much traffic on the weekends, so naturally the weekend is the time to make changes.
So wow, it's no shock that Mondays are when you're most likely to see problems with a website. But these problems and hiccups are the price you pay for progress.
If you don't want to chance any disruption in your life, then I guess you should never change. Otherwise, get over it.
You need to get the code monkey off the production box.
They need a Dev environment. And THAT's ALL they touch. They deliver their code to UAT.
QA needs 2 environments:
- Unit Acceptence testing (UAT) and all bugs go back to Dev
- Integration Testing (IT) and all bugs back to Dev or you need SysAdmins who need to hack the OS middleware &| environment)
Production where NOHING is allowed until its gone through UAT & IT.
MSBPodcast.com The opinions expressed here are my own. If you don't like 'em... Think up your own stuff.
You mean, other companies' web programmers get to take weekends?!? Man, that must be nice...
"In a 32-bit world, you're a 2-bit user. You've got your own newsgroup, alt.total.loser." -Weird Al
Clearly we need a 5 day waiting period on developers. How many more of our websites must go down before congress gets the message? We should also launch a class action lawsuit against schools and staffing agencies for negligence in putting dangerous developers in the hands of unsuspecting companies. Its the republicans fault.
[/sarcasm]
Skiers and Riders -- http://www.snowjournal.com
I am an employee of the state, and the government of Georgia is cheap! That, and our old-as-creation computers give us hell at random intervals, and let me tell you, they don't discriminate based on the days of the week.
The way I look at it, if you have anyone doing active development against your production server then you are just asking for it anyway.... geeeeez a test server, change management, such novel ideas :>)
Of course you also provided your IT staff with lots of people who do the testing of the webpage? Or do you suppose they should do so themselves?
Lawrence: No. No, man. Shit, no man. I believe you'd get your ass kicked saying something like that.
(atleast thats the saying around my company)
Desktop applications sucked because the developers were smart and the users were stupid.
Web applications suck because the developers are stupid and the users are smart.
Solution: have the desktop developers design web sites and fire the webdevs! I mean, I've been waiting WAY to long for a boss key on e2, anyway.
Arg. I just got back from vacation, so I don't seem to be hitting on all 8 yet.
s/grammer/grammar
www.eFax.com are spammers
Is this really a case of "Weekend Inspiration", or a case of management pushing changes that haven't been thouroughly tested?
I find it quite disturbing how these companies are blaming downtime on developers. This means that:
a. You have no change control over your environment, and developers can do as they please, hence poor management.
b. Developers are implementing changes that haven't been thouroughly tested. Again poor management.
Technology and competition isn't moving so quickly that you cannot take the time to use a test/qa environment.
Awesome!
What kind of mickey-mouse operation is it that allows someone's (whether management or developer) mistake to take down their web site? Have they heard of QA and testing? You'd have to be insane to allow any changes to a production system without it being tested on the test system first. In all the places I've worked at (I'm a back-end hacker, using C/C++ and java) anyone who made a change to the production system without following all the test procedures (regression tests and QA signoff) would be canned in a second. (Unless it's the VP of Engineering -- but that's another story.) Or these personal sites they're talking about?
Unlimited growth == Cancer.
Change controls.
Any place that doesn't use them deserves all the shit it causes itself.
- A.P.
"Remember when the U.S. had a drug problem, and then we declared a War On Drugs, and now you can't buy drugs anymore?"
Developers! Developers! Developers!
Isn't mondays 'patch your ms box' day?
Maybe it isn't the web developers....
J
Developers don't have beepers going off when the server goes down. All the sysadmins I have known basically live with a server lo-jack on their hip, unable to go outside a certain radius from the server (or a terminal).
Development staff are now a bigger threat to website uptime than hackers and viruses combined Is it going to stop ? Ignas Mikalajunas
In other news, more people die in hospitals than at McDonalds, so if you get appendicitis, you should go to McDonalds.
I'm shocked, I say, shocked! You mean to say that webhosts are more likely to crash the more they get used? No! It's sounds so counterintuitive! Seems to me a site would be MUCH more likely to fail at, say, 4AM Christmas Eve.
Nothing to see here, folks, move along.
Mod down people who tell people how to mod in their sigs
Correct is not necessarily what is mandated to you from above. Sure, it'd be nice to be able to take two weeks for a single idea, but reality is a different story. By two weeks later you are probably expected to be working on a new project or at the very least moving on...
This is my digital signature. 10011011001
Not only does my company have an extensive change control system in place, but until very recently, we had a "no changes on Monday" policy. Since Monday is our busiest day, it made good sense. In fact, we couldn't even run network cabling for new servers on Mondays. There were little signs all over the building that said, "A Monday without change is like money in the bank!" Kinda cheesy, but it seemed to work.
We recently dropped the policy, but I'm not sure if there has been any fallout from doing so.
I don't care what the question is, but the answer is FileMaker. --HarvDog
IMHO, what is needed is:
1. Professionalism by the developers.
2. Testing, testing, and testing -- by the developers.
3. QA, QA and QA -- by someone other than the developers!
The personal sites are all updated during "weekend inspiration", which generally involves consumption of mass quantities, as the Coneheads would say.
It's practically impossible to tell when the average "personal web site" is functioning as intended anyway.
"Obviously, I'm not an IBM computer any more than I'm an ashtray" (Bob Dylan)
When I was an admin at Univ. of Texas at Austin, we upheld the same idea, calling it the 'Friday Rule'. Unless something was outright broken or a big shot professor needed something done, no hardware work, software installs, config file changes, database rebuilds, or anything else potentially destructive would be allowed to happen on a Friday to a production machine. Test machines and desktops not running any simulations (we got a lot of VLSI / circuit sim jobs) were fair game though.
2. Testing, testing, and testing -- by the developers.
:( This is one are where having *restrictive* per-seat db licences bite.
:-(
probably not a good idea. while I would get them to test their archictecture, algorythms and performance I would leave any real testting to a QA team (hey thats if you really have one).
3. QA, QA and QA -- by someone other than the developers!
very true. *monkey-testing* for input screens, navigation and design. Load and stress-testing may reduce any performance bugs. But the one that may bite is the RDBMS for database intensive sites where developers have made changes to stored procecures on the db. Is the SQL code in CVS? Can we roll back the db on the live site if needed? Opps we make changes to the live DB
4. Managers must know the test/ QA process should never by bypassed -- this unfortunately is probably the hardest point.
the phb problem. no amount of testing, development rigour will avoid the *monday morning* crash with 100's hasty ill-defined of cvs commits in the weeks previous *mandatated* with/without poorly defined specifications. Of course you have to balance this with a business being able to adapt quickly. But I remember one marketing clown thinking *development was easy* and could be learnt in 15 minutes.
There are no *silver bullets*.
Working with fragile web based systems almost warrants XP unit type testing approach. Other agile development approachs might be useful.
peterrenshaw ~ Another Scrappy Startup
Since when is it good policy to impliment changes to a site without first going through a test server?
Any serious outfit should put anything and everything through z test server BEFORE it goes live.
I take mondays off. Nothing like having a software release on a Saturday to ruin the weeks start.
Where am I going and why am I in this handbasket?
Attenda
Hey Neal...your developers are about to logon
Go get em slashdotters....
I work with a guy who used to work (as a contractor) for, um, a well-known cablE SPorts Network . His group was responsible for maintaining the code that populated the sports stats ticker that runs along the bottom of the screen, reporting the scores of various games. This data comes in from very disparate and non-standardized sources.
He has a couple war stories about how they would recompile code during the commercial breaks of a live broadcast!
How's that for pressure?!
Software Wars
...The problem is that somebody forgets to make a copy and work on that instead of the original?
Linux, you magnificent bastard, I read the fucking manual!
Remember in school you'd get the same "A" grade whether you put in a perfect 100% job or a 90% job? Sometimes coding is like this too. I work in a micro-startup (2 of us) and managed operations (read: I am coder sysadmin, etc. and tester, etc.). Given the too many things that need to be done on a daily basis, extensive testing is often just not feasible. So I shoot for 90% in my QA work and hope that it all hangs together. Most of the time that is still "good enough for government work". It's great to have teams of testers when you have resources to allocate - if I had a team to sick on this I'd do the testing in a heartbeat - but in our situation, we do what testing we can manage and move on to the next thing. If we crash (and it does happen from time to time) - we fix it quickly (grin!). One of these days we hope to be able to have more resources and I'll sing a different song. But for now we're bootstraping.
True, some developers tend to shoot from the hip, uploading "simple" or "poorly thought through" or "I don't really understand the implications of what I'm doing but look, it flashes and the marketing department loves it!" code to the live environment. Chaos usually ensues. The programmer acted unprofessionally, making an unsanctioned change without getting appropriate permission and following the correct processes.
In my experience, however, more often than not these changes aren't made at the whim of the programmer but at the request, pleading, and sometimes demand or direct order of a manager who does not understand the grand scheme of things. So many times I've heard "Can't you just add a field to the database?" or "Isn't it just changing the text in this one place?" or "What if we just [insert hacked/crazy/incomplete/unworkable solution here]" while the people making these statements hardly ever mean harm, a lack of understanding of the big picture can often turn a seemingly common-sense solution into a ticking time bomb.
Not every programmer is up to the challenge of saying "no" to the marketing director or customer service manager when approached with such a request. Some see it as way to earn kudos or respect from these people; to advance their career, show off their skills, and sometimes "save the day". As you move up the chain of command resisting becomes harder and harder. Heaven forbid the CEO walks in and asks a programmer to handle this "critical issue" ASAP. It seems so simple; our customers/clients/team will love it; it should only take 5 minutes, right?
There are certainly cases which demand an immediate reprioritization of resources. Legal issues, critical issues with technology ("No one can log into the site!"), and even making a small change to close a deal barely even scratches the surface of issues that often demand immediate attention. The trick is assaying the importance of each issue and responding accordingly. If Chicken Little walks in from the call center saying the sky is falling and by the way, they're flooded with calls from customers who can't access the website, make sure it's not just a single phone call that reached a stressed service manager at a busy time of the day. With proper attention and prioritization,
If not carefully managed, however, giving in to off-the-cuff requests can spell disaster for both the short term function and long term viability of any system. A programmer goes out-of-process to quickly respond to a direct request from management once, and the next time a not-so-important issue arises, they're more likely to do the same. They feel like a hero. They're saved the day. On the flip side, management loves to get an immediate response, so they'll go back to that same programmer.
The developer shirks or rushes regularly prioritized tasks to get another feather in the cap... so now we have 2 hacked solutions. In both cases, the amazing 5 minute solution that looks great on the surface can cause problems in the future. Even worse, because of the "ASAP" nature of many such requests, necessary processes such as testing, QA, and proper implementation procedures can be fudged or ignored completely, making the likelihood that an unintended bug is introduced even greater. In addition, failure to plan for capacity, follow a design specification, properly document work, or just plain missing a detail leads to statements like "Oops, we did what with the SSN's?" or "The data hasn't been captured for how long?". The 5 Minute Solution had its day of glory, but has festered out of sight and now is reborn as 2 Weeks of Damage Control.
If the problem is tracked back to the programmer who created the bug with the quick fix for the CEO, who turns out looking bad? It's certainly not the CEO for putting pressure/an unrealistic timeline/an out-of-process request on the developer, who might not have even been the right person for the job. In the end, both the developer, who thought he was going the extra mile to be a team player, and the techno
If it ain't broke, don't fix it.
However there's not much choice since much web (and other) software development seems driven by a high priority need to have new content, exciting developments, and enhanced features. If you have a stable system, then introducing any changes can have unforeseen consequences, especially if it's a particularly large and complex system.
The article blames developers for Monday morning bugs, and it's right. If I change some code and deploy it to the live server that I work on, then it is my fault if it breaks something (and yes I have done it on occassion). However this shouldn't be the case. Inspiration that I have on Monday night should be no more likely to cause problems than ideas I have over the weekend. In both cases any code changes should be deployed to a staging server and run through a test suite.
If the ratio of bugs to changes is higher on a Monday morning, then this is a problem - developers are not following quality control or change control processes on Monday mornings. That would be an interesting finding. On the other hand, if the ratio's the same, then more bugs just means more changes are made on Monday mornings. And you could argue that that implies a higher level of productivity!
If any Project Managers out there have statistics on this sort of thing - please post and let us know!
This is not a sig
Anyway, what the hell are you software developers doing thinking about work at the weekend?
Stop it now. Go out. Yes, outside. Have some fun!
This is not a sig
Sounds like another "twenty year man" speaking - Amen.
But again, that is one of the things that seperates a GOOD senior software engineer from a new hire - the knowledge of what happens when one bodges features in, and the courage to say to the senior VP of marketing "NO, I am not going to just rush that in. The risks are too great. If you don't like it, you can talk to the head of the department, and see if he is willing to accept my resignation, because this feature will not be rushed in while I am the lead on this project. I am perfectly willing to begin work on the feature, and once it has been designed, implemented, and tested, we can release it. Now, do you wish to continue to delay the day it is done by wasting my time, or do you want be to begin working on it correctly now?"
NOTE: Kids, don't try this at home UNLESS you really ARE the senior software engineer on a project - you have to have developed some cred with management to get away with this.
And this is also why senior software engineers need to constantly pressure management to make sure the company policy allows you to do this, and that you get backing from management. This is one of the (few) places an ISO-9000 cert is useful - make sure your policies are written to require planning and test, so that when the VPMarketing tries to do an end-run around it, you can say, "Well, if you want to void the company's ISO-9000 cert, I guess I can start...."
www.eFax.com are spammers