Programming Pearls by Jon Bentley - old as the hills by now (he talks about the location of data on tape....), but full of very good insights into writing "good code"TM.
You might also like "The pragmatic programmer" - Hunt and Thomas - which is another "meta-programming" book with a lot of ideas and insights you could actually sell to your pointy-headed boss.
The section on "zero-tolerance" coding is a great "why and when to refactor" argument. There's also a good section on how to design the units of which your software is composed, how to reduce the coupling between those units, and how to test em when (you think...) they're done.
My father - as an undergraduate - got an award for math. He basically got the second highest score in the country for his exam, and used his skills every day navigating ships across the oceans in the days before satelites and calculators.
The weird thing was he could not get his head around the Commodore 64 I got for Christmas. He never worked out how to use a calculator - he didn't need one, because his mental arithmetic was faster than you could type numbers into a machine. He never even worked out how to set the video recorder...
My wife, a succesful woman who has been in charge of large teams and managed multi-million dollar projects can't get to grips with email.
What this boils down to is that all devices - digital, analogue, physical - rely on some pattern of understanding in their users.
If the world view is not shared, or not communicated clearly, no matter how "smart" your users are, they won't be able to use that lovely gadget.
The problem is - of course - that you can usually only afford to implement a single metaphor. Your VCR can either behave like a tape recorder (remember those ?) or like a TiVo ( i havent got one, so I have no idea how they behave), or like your cable digital set top box.
That means that you will nearly always have a whole bunch of people go "it doesn't make sense". Live with it.
We are often faced with immovable deadlines and limited resources; we nearly always end up negotiating about the scope of what we can deliver within those constraints. Of course, this is not in a consulting/outsourcing environment - we're an internal team, so our clients are allegedly on the same side (hah). Nevertheless, we agree with the PM what's in scope for the current phase, and what they need to do to increase the scope if they're not happy with that. We can do this because we have a habit of meeting those commitments - sure, we screw up, but never without letting the PM know as soon as we know what the situation is.
A lot of responses to this thread seem to indicate a lack of trust and respect between the techies and the project managers. I don't care how "technical" the PM is, if that basic trust and respect is lacking, you will never fix the problem. Of course, trust and respect need to be earnt, and sometimes, we techies have to take the first step....
Until recently, I was the development manager at a fairly large internet company; we had project managers who knew very little about software development, database design or how to run software projects. Do you know why ? because that was my job.
Whatever you may think, technology is not the most important part of the project - delivering what the business wants and making the right trade-offs to get that done is what matters. The intellectual purity of our great code is wonderful, but who cares if it gets delivered 6 months after it's needed ?
The project manager's job is to work out what needs to get built and by when; they need to get all the external dependencies sorted out, ensure the requirements are either known or the person(s) who controls those requirements is available when required, get the money and resources sorted out, and work with a techie on how to get the deliverables built in time.
I was that techie - and it worked pretty well. The project manager asks for stuff by a certain date, I work with the rest of the team to see what we'd need to do to make that happen, I negotiate with the PM on what is and is not in scope, and get the techies to start with whatever needs to happen to get the project done.
Every couple of days, I sit down with the Project Manager to agree out where we are, re-negotiate dates/resources etc. if required, assess new requirements, maybe work out in more detail what the plan for the next phase looks like. If we have to cut corners - and this does happen, coz we don't live in a perfect world - I work with the developers to see what we can cut that will have the least effect on the quality - the PM doesn't make that call, I do.
Project Management requires skills I don't have - I don't understand the commercial pressures on the company, I don't understand the legal framework we're working in, I don't have the patience to build and update Gantt charts, I don't enjoy endless meetings or chasing people for every little detail of their deliverabes. The project managers know this - they don't think any less of me, just as I respect the fact they couldn't design a database schema to save their lives.
So, I would suggest trying to form a good working relationship with your project manager by trying to understand what they do for a living, understand that there is more to a project than the technical deliverables, learn to speak their language, and offer alternatives when they ask for the impossible.
The attitude of most of the posts in this subject has been "huh, we're 200 times smarter than those idiots running the project, they're so stupid they couldn't blah blah blah". Hey, if you're so smart, it's your job to use that intellgence to move the project forward, not whine about how what a bad job everyone else is doing.
(and it wasn't improved by the millions of dollars we spent on a top-brand consulting firm who implemented it), but the problems are pretty generic. Vignette does a particularly bad job of solving them, but content management is inherently tricky. I guess most of the history of computer science has been concerned with maths and databases, so we know pretty well how to model an accounting system or calculate PI to a zillion digits. THe softer stuff - how do people actually create "words" and how do they access and manage those words is something you can't look up in an algorithm book or a pattern repository. I had a look at OpenACS - it looks pretty cool. After working with Vignette though, I have sworn of TCL as a web language for ever.
I was the development manager for a big site using Vignette as a content management system. What they don't tell you when you buy these products is that you will inevitably invest a huge amount of time in customizing the application for your requirements, and that - at the end - it will still not feel "quite right".
Besides the purchase price, you have to assume you will need signifant training and/or consulting support, and you will inevitably end up rebuilding your site from scratch to fit into the new framework. The standard consulting fee seems to be roughly 10 times the cost of the software (amazing how that always seems to work out...).
If your content creation processes are in any way complex, you will spend a significant amount of time and effort trying to create a tool that facilitates or at least accomodates this process; if workflow features are expected, you will have a serious amount of work to tackle.
You will also need data entry time (or at least the ability to convert all the exising content into whatever format the Content Management Server expects) for all 2500+ pages.
If what you're doing can be classified as business critical in any way, you will need at least one administrator for the Content Management System and whatever database it uses.
Because content management systems involve a lot of "dynamic" activity when serving pages, it's common for them to experience performance problems, so you may need to invest in additional hardware to serve the same number of pages. You will definitely be looking at how to cache your content - this is a whole bundle of joy in its own right.
In short - make sure you want to go down this route ! It may free you up from the tedium of cutting and pasting into a static HTML file and FTP-ing it, but instead you'll be feeding and watering this content management beast.
Unless your company makes its living from its content, the costs are unlikely to be recouped.
Of course, if you want to do it, I would suggest writing one from scratch; I'm working on a JSP/servlet based content management server right now, and it's a lot of fun.
You say you're finding too many bugs - it sounds like there is something fundamentally wrong with either the product or the development process.
First thing to do : look in your bugtracking software ( you DO use bug tracking software, right ?) , and try to isolate hot spots. Is there a particular piece of code that generates more bugs than others ? Is there a common pattern to the bugs (ie. memory not being freed, of-by-one errors etc.) ? Are they _really_ bugs or mis-interpretations of the requirements or the design ? In my experience, the 80/20 rule applies to bugs in spades - it is just hard to find the patterns.
If you need to, make the bug categorisation in your bug tracking software more specific. Once you get an idea of what your hotspot is, you can work at fixing the cause of the bugs.
If it's a particular piece of code, make sure it's reviewed by the best developers/architects you have, and consider refactoring it. At the very least, insist that it is reviewed and tested thoroughly before chec-in to the source code control system, and consider adding a hurdle to jump prior to check in (e.g. get the manager to sign it off).
If the code was written by one developer, consider swapping them out and giving it to someone else - it may be they're in over their head.
Make sure you increase the number of test cases for this piece of software, and check for "edge cases" religiously - if the code is broken at all, it is likely to be broken in more ways than you realized.
If it turns out that the problems tend to have a common cause (memory leaks, of-by-one errors,etc.) consider a structure which forces developers to concentrate on those issues before checking in code; again, consider the hurdle ("software must be signed off by the off-by-one guru prior to check in"), and hone your tests to check for these kinds of errors if possible.
If the bugs stem more from misunderstood requirements or designs, beef up those areas. Work on your requirements and analysis processes; consider training courses for the developers to get them up to speed on interpreting these nebulous documents, and look at improving the review process by having designers present. Frequent "mini-deliverables" (another concept stolen from XP) will help here too - get your team to deliver a working piece of code - it need only be a minimal sub-system - and get it reviewed by designers and analysts. If the bugs tend to occur on the boundaries - i.e. invalid API calls, invalid parameters etc. - consider design by contract or aspects.
Finally, there's a bunch of hygiene stuff:
coding standards (ideally with tools to check for compliance) help by allowing you to specify how to deal with commonly bug-prone operations
invest in a decent bug tracker, request tracker and source code control system - it really helps if you can trace a bug back to a particular requirement or previous bug fix if you're trying to find out where all your bugs are coming from
seating arrangements - I worked on a project once where the bug rate was halved when the developers were moved to sit in the same room as the database team. Get everyone in the same room if at all possible
automated test tools - already mentioned - help to ease the pain of testing. If practical, insist that the developers run the automated tests before checking in code
religiously review all work products - designs, specifications, documentation, project plans, code, tests, the works - before declaring them "ready for use". It's the single cheapest way of finding problems, and a great way to spread good practice.
In any development effort involving more than 1 person, comments are not enough.
You should seriously consider:
coding standards outlining naming conventions, indentation, common usage, commenting requirements etc.
design and code reviews of all non-trivial work products
documentation standards (see the book review on agile modelling) - you should document the stuff that isn't obvious, and keep the documents up to date
automated unit tests which allow any changes to the code to be tested (yes, I stole that one from the eXtreme Programming guys).
The problem with comments is that they explain what the code does, but all too often the "why" - the structure of the program - is not obvious by looking at the comments - it's like trying to work out a streetmap by looking at the names in the phonebook.
I took over a large project from a major consulting firm; much of the code was immaculately commented, but the overall structure of the design was almost impossible to fathom; the documentation was out of date and incomplete, and everybody had a slightly different view of how things worked. Whenever we fixed a bug or made a trivial change, we'd hold our breath just in case the trivial change had unforeseen consequences somewhere else in the system. A simple class diagram and database schema would have been more useful than most of the comments. Unit tests would have saved us literally hundreds of hours of pain...
Code Complete, a book by Steve McConnell is a great read on this subject; I also recommend "Agile Development" by Cockburn.
Some people have asked what this modeling thing is all about; as a public service, here's the PinglePongle version.
Some academics regard the process of building software as a descent from the abstract to the concrete. Typically, it starts of with "hey, we need a thingumajig - it must streamline our widget creation process". This is usually followed up by a requirements spec which says what the thingumajig should do, but doesn't go into how should be constructed. The next phase typically involves the developers working out how to build a thingumajig which meets the requirements spec. Once the developers have worked out how to build the thingumajig, they start to code it; if you're lucky it gets tested and eventually it's released.
All these phases do occur, but the way in which they're executed varies enormously. Frequently, they overlap; frequently people move back and forth between the phases several times; frequently it's hard to tell which phase you're in.
Modelling general concerns mainly the "how do we build a thingumajig" design stage. A model can be anything that helps to convey the intention of the design without being actual code - a crayon drawing, a piece of prose, an email, a memo, a flowchart, a wall-sized database diagram, a haiku. There are a bunch of methodologies which adress the modeling phase, most famously the Unified Modeling Language (UML) which defines a standard vocabulary for object oriented design using around a dozen different diagram styles.
The problem with models is that they can become very cumbersome and end up leading a life of their own (here's a previous/. thread...), and it's not uncommon to have projects with many times more paper than code.
As it goes, I believe that anyone who takes programming seriously should be at least passingly familiar with the most common modelling practices - UML use cases and class diagrams, database E/R diagrams, flowcharts; in my projects I ask the developers to model the tricky parts of the project, but only with the explicit purpose of understanding the problem better before starting to write code. You can change a model around easily and quickly - changing a class diagram to put a method in a different class takes seconds - but if you realize halfway through coding there's a better way to do it, you almost never have time to fix it.
I've ordered the book (no, I'm not going to tell you where from) - and if it's as good as Cockburn's other work, it'll be money well spent, even from bn.com !
I am a development manager with a large (20+) team; I am gradually introducing some aspects of extreme programming into the organisation, and so far, it's working pretty well. The concept of user stories is great for getting our multiple customers to get to grips with their requirements; the tactile nature of the cards allows the planning sessions to be participatory and inclusive, the customers understand the trade-offs and the developers get to understand what they're _really_ being asked to deliver. Pair programming and continuous testing have produced decent results; the main problems we have are accomodating sickness and holidays into the project plan. On the other hand, our organisation does insist on some of the more traditional aspects of development management - a formal "Project Manager" role, regular progress reports to the executive team, functional and technical specifications etc. I don't see how you could realistically hope to convert an entire team to XP overnight. The gradual introduction of some of the basic concepts - esp. those which can be explained as "quality oriented" - is a lot more likely to succeed. I've read the book, and it's okay; the editing could have been a lot tighter (there's page after page of code printed out which outlines the application to be tested, not the tests themselves, there is a huge chunk devoted to the documentation for the tools which is available online etc.), and it kinda drifts away from the point now and then. If you want to get up to speed with Ant, jUnit etc. it's worthwhile, but don't expect to become an XP xpert.
Content Management Systems are golf-course-ware. It gets sold to senior executives by smooth-talking sales executives who claim their products solve every conceivable business problem, is a doddle to install, standards compliant, holographic user interfaces, everything.
The reality is that this is an inherently complex field, which requires a huge amount of business-thinking before the technical solution even becomes relevant - how do you want to manage your content ? Do you have a requirement for workflow-style solutions ? Is revision control important ? Do you need collaborative features allowing several people to work on a document at the same time ? Do you have a knowledge management infrastructure so you can re-use an accepted taxonomy ? What are your security requirements ? Where does your content reside - is it largely "document" based, or is it mainly database-driven ? How technically sophisticated are your content generators ?
Only after you have worked out what you want to do with your precious content should you consider what the technology can do for you - I suggest using any one of the myriad requirements gathering techniques used in software engineering and specify your "ideal" content management system, then drawing up a list of candidate technologies.
If you start with "what can the technology do for me", you almost certainly will end up spending a lot of time and effort (and money !) and getting very little in return....
I joined a company about a year ago; they'd spent several million dollars on getting a website built by a big consulting firm, using a cutting edge content management system. "Is there any documentation ?" I asked, on my first day, and the project manager from the consulting firm proudly pointed at the shared drive with nearly a gigabyte's worth of stuff in the "Documentation" folder.
I poked around - weeding out the huge screenshots and photoshop files the designers had created - and tried to find out how the system worked; I found a folder called "architecture", and started to read. Funny thing was, most of the documents were more than 6 months old; several were more than a year old. Hardly any were written by current project members. There was no road map, no versioning. In short - the documentation - though plentiful - was badly out of date, often actively misleading, and totally pointless without the writer to explain the context and intent.
This is not an uncommon occurence; there are people with (parts of) the answer.
Check out "Agile Development" by Alastair Cockburn; he has very strong ideas on the topic of documentation and how to keep it relevant.
Next, buy "Code complete" by Steve McConnell, and make sure everyone in your development team reads it and understands it.
You might also want to check out the ideas behind Xtreme programming - it sounds to me like you're starting to find it impossible to maintain code written by other people with different coding styles and are hoping that documentation will save your bacon....it won't.
Most long-term projects rely on coding standards, a common vision and understanding of the architecture, effective communication and well-written code. Documentation is usually done because the customer insists on it, or the management team believe it's going to solve hand over problems. It rarely helps the development team to get to grips with what is really going on...
If you've ever been in the position where you had to recruit 20 or 30 people in a few months, you'll appreciate the help of a good HR specialist. Sure, they might not bother to forward the CV of the eccentric but inspired genius who submits his resume in object pascal. On the other hand, they will shield you from the deluge of bozos who think that because they can spell "www" they can code. Or the "I know you asked for skills in code wrangling, but I feel my experience in the catering sector uniquely qualifies me for this role" hopefuls.
I also think a good HR team can help the rest of the company to be fair about who gets rewarded, who gets promoted, and what to do about those who don't make the grade.
All this stuff is annoying, distracts from the real goal of the team, and frequently clueless - "duh - how can you compare the unique skills of a 3d artist with whatever the hell a C++ programmer does when dishing out bonuses". But just try and run a team for more than 6 months without this kind of support. Try and explain that you blew your annual bonus budget on the guy who wrote a cool screensaver, so you can't reward the team which delivered the space shuttle software on time.
It doesn't always work like this - I've been in lots of places where HR were a self-sustaining bureaucracy with no clue as to what the real organisation was doing. But when it works, the HR team is as much a part of the special teams as the people who keep the laser printer supplied with toner.
not longevity. I do think it's a good idea to have a defined career structure - I have interviewed a lot of developers in my time, and most of them want to know how their contributions are recognized. A simple hierarchy of "junior/senior/principle" or something at least shows that we have a clue on how to recognize what people contribute and that there are visible signs of that recognition.
As part of a reorg, I am trying to change my team's job titles so that instead of reflecting their specialism, it reflects their seniority. So, I've moved the "Senior designer of buttons and fonts" to a role called "Senior Developer",the "Junior designer of buttons, fonts, and logos" to "Junior Developer" etc.
This only reflects the reality of how we work - everyone moves between projects and uses their specific skills to contribute where they need to, it shows that there is a career path that reflects your contribution, rather than some org chart structure (oh, I am the Junior Database Column-adding specialist - there's no position on the org chart for Senior Database Column-adding Specialist - how do I get promoted ?).
The main risk to this strategy is that it makes it harder to explain to the senior management team what all those people do - they rarely understand what it takes to build software. So in these uncertain times, it's a lot easier to say "You want to fire Joe ? Look at the org chart - Joe is our Senior User Interface Designer - we'll always need a user interface designer - you can't fire Joe !", rather than "You want to fire Joe ? He has really good skills at User Interface Design ! Well, yeah, so does Jane - but Jane is our lead Object Specialist ! Well, okay, Gunther knows a bit about objects, but we really need his skills as..." etc.
It seems that in most organisations, HR and the executives like to know what people do for a living, and the easiest way for them to achieve that is to insist on an org chart with specific titles. The way to ease this desire is to help them to understand the way the team works, show good results on major projects, and build trust.
Nearly all the legislative approaches I am aware of concentrate on prosecuting perpetrators of security breaches or those who create/posess the tools to do so. This is ultimately not very likely to yield satisfactory results - the Internet being a global medium and all, you might find it hard to prosecute that albanian script kiddie, or the Afghani virus writer.
I believe that the best thing the government can do is to help establish a credible full disclosure infrastructure. A "not-for-profit" security organisation managing the release of vulnerability information, together with patches etc. By creating a formal infrastructure, run at arms' length, the government could signal how important Internet security is, and help establish best-practice.
It could be backed up by some kind of deal which requires all government IT suppliers to disclose to this body, and all other suppliers to have a nominated security adminstrator who monitors the new vulnerabilities and takes appropriate action.
The key issue with Internet Security is that the majority of incidents are due to poor security postures by many organisations. The inter-connected nature of the internet means that one organisation which is compromised provides a spring board for attacks against many others. Surely the common weal is served better by encouraging (though not forcing) those companies to "patch up" than by trying to outlaw tools/actions/thoughts.
The Register will no doubt notice their immortal prose is being stolen, and will employ Microsoft to create a DRM solution for text. Oh, wait, no, Adobe already did that. Now, what was that Skylarov guy doing again ?
DRM technologies are based on crypto; crypto tends to be open - a well-publicized algorithm is usually favoured over an obscure home-grown one. The products I've seen so far (and I have implemented some DRM solutions in the field) are pretty obvious. It's unlikely the media companies could control a DRM scheme - it would be too easy to crack, and you could easily imagine a napster-like revolution based on publicly available crypto and a handful of clueful bands.
I'm not saying it will happen, just that it could, and if it did, it mightn't be the worst thing in the world....
maybe effective DRM will allow content creators - writers, musicians, artists, programmers (?) - who operate at a human scale to bypass the distribution network. After all, the reason record labels, publishers, studios etc. exist is :
they have a distribution network which allows artists to get paid for their work
they have a marketing network which can generate sales by advertising and PR
The internet can provide a marketing and PR medium - a word-of-mouth campaign works better than any number of radio plugs (Blair Witch). A meaningful, user-friendly, consumer-driven DRM mechanism can ensure content creators can get paid for their efforts, and effectively bypass the distribution network altogether. What then is the role of the middlemen ?
This works at most "human" scale projects; movies are typically too big and expensive for individuals or small groups to be able to put together, although the a DRM-enabled broadband internet distribution network might work well for smaller productions.
It is a possible outcome - not a likely one, given the clout and political buynig power of the big media companies.
(a big consulting firm, btw), they sent me a form letter reminding me that any "inventions, novel ideas, recipes, designs, concepts and business models" I had developed during my time there belonged to them, and that they would protect their property yadeyadeyade.
I wrote back and said I had come up with a pretty cool recipe for chocolate cake, and the only novel ideas I had developed involved Irene in accounting; I had come up with a business model for a company that doesn't treat its staff like thieves, and my designs could be found on the borders of the notes of the interminable meetings I had forced to sit through. It wasn't very mature, but it made me feel a bit better....
Oh, and Irene from accounting actually emailed me 2 days later.
P
The ability to book an appointment without having to go through a receptionist; links to information sites on common afflictions; links to local support organisations and healthcare businesses (e.g. pharmacies); a repeat prescription system, and of course, linux advocacy. 9 out of 10 doctors prefer it !
One of the reasons software engineering is hard is because you can not easily draw analogies. People have been building houses and bridges for thousands of years; people have been building websites for less than a decade.
The methods that work well for software projects - like XP, the spiral development cycle, patterns, OO - are young, and constantly evolving at a rapid pace. My PC from 4 years ago is a doorstop; the house I live in is 130 years old. Our local MacDonalds is full of Cobol programmers (many with CS degress) serving fries.
One of the main reasons my work can be difficult is that there is no easy way to explain to non-software folk why certain things are hard and other things are easy - to them, it all just looks like misspelled typing. The comparison between "software engineering" and "civil engineering" is partly valid, but in many ways a sofware developer is more akin to the guy laying the bricks than to Frank Lloyd Wright.
I'm in exactly the same position - cept I already resigned.
Why did I resign - even though I faced the same dilemmas ?
After cycling through half a dozen business models in as many months, I reckoned the company was doomed.
While technology was/is crucial to all of those business models, I did not feel I could save the company from itself.
While I like and admire a lot of the people I work with, I felt that the strongest message I could send them was to lead by example. They can do better, an I know that "the company" will screw them over with little compunction.
So, leave, and make sure that you explain (in private, and discretely) to the rest of the firm what is going on - and try to take them with you or find them other jobs.
Don't be the last rat to leave the sinking ship - nobody wins.
in principle, it should be relatively easy to write a VST plug-in which uses TCP/IP to farm out work to a music rendering farm (just as 3d graphics apps use render farms).
However, you would almost certainly never get real-time performance - you could imagine a fractal "reverb" program, maybe, where the local processor provides a coarse version, farms out the detail to a render farm and moves on, but performance would be seriously variable - a single network glitch like a request for new tasks could cause the effect to coarsen as performance degrades.
Most of the time, you would rather have no effects at all than randomly variable ones.
However, clusters would be great for apps like winchimes etc. which generate random music based on complex math - although the law of diminishing returns may set in rather early....
Slashdot Mirror
Programming Pearls by Jon Bentley - old as the hills by now (he talks about the location of data on tape....), but full of very good insights into writing "good code"TM.
You might also like "The pragmatic programmer" - Hunt and Thomas - which is another "meta-programming" book with a lot of ideas and insights you could actually sell to your pointy-headed boss.
The section on "zero-tolerance" coding is a great "why and when to refactor" argument. There's also a good section on how to design the units of which your software is composed, how to reduce the coupling between those units, and how to test em when (you think...) they're done.
Nev
My father - as an undergraduate - got an award for math. He basically got the second highest score in the country for his exam, and used his skills every day navigating ships across the oceans in the days before satelites and calculators.
The weird thing was he could not get his head around the Commodore 64 I got for Christmas. He never worked out how to use a calculator - he didn't need one, because his mental arithmetic was faster than you could type numbers into a machine. He never even worked out how to set the video recorder...
My wife, a succesful woman who has been in charge of large teams and managed multi-million dollar projects can't get to grips with email.
What this boils down to is that all devices - digital, analogue, physical - rely on some pattern of understanding in their users.
If the world view is not shared, or not communicated clearly, no matter how "smart" your users are, they won't be able to use that lovely gadget.
The problem is - of course - that you can usually only afford to implement a single metaphor. Your VCR can either behave like a tape recorder (remember those ?) or like a TiVo ( i havent got one, so I have no idea how they behave), or like your cable digital set top box.
That means that you will nearly always have a whole bunch of people go "it doesn't make sense". Live with it.
We are often faced with immovable deadlines and limited resources; we nearly always end up negotiating about the scope of what we can deliver within those constraints.
Of course, this is not in a consulting/outsourcing environment - we're an internal team, so our clients are allegedly on the same side (hah). Nevertheless, we agree with the PM what's in scope for the current phase, and what they need to do to increase the scope if they're not happy with that. We can do this because we have a habit of meeting those commitments - sure, we screw up, but never without letting the PM know as soon as we know what the situation is.
A lot of responses to this thread seem to indicate a lack of trust and respect between the techies and the project managers. I don't care how "technical" the PM is, if that basic trust and respect is lacking, you will never fix the problem. Of course, trust and respect need to be earnt, and sometimes, we techies have to take the first step....
N
Until recently, I was the development manager at a fairly large internet company; we had project managers who knew very little about software development, database design or how to run software projects. Do you know why ? because that was my job.
Whatever you may think, technology is not the most important part of the project - delivering what the business wants and making the right trade-offs to get that done is what matters. The intellectual purity of our great code is wonderful, but who cares if it gets delivered 6 months after it's needed ?
The project manager's job is to work out what needs to get built and by when; they need to get all the external dependencies sorted out, ensure the requirements are either known or the person(s) who controls those requirements is available when required, get the money and resources sorted out, and work with a techie on how to get the deliverables built in time.
I was that techie - and it worked pretty well. The project manager asks for stuff by a certain date, I work with the rest of the team to see what we'd need to do to make that happen, I negotiate with the PM on what is and is not in scope, and get the techies to start with whatever needs to happen to get the project done.
Every couple of days, I sit down with the Project Manager to agree out where we are, re-negotiate dates/resources etc. if required, assess new requirements, maybe work out in more detail what the plan for the next phase looks like. If we have to cut corners - and this does happen, coz we don't live in a perfect world - I work with the developers to see what we can cut that will have the least effect on the quality - the PM doesn't make that call, I do.
Project Management requires skills I don't have - I don't understand the commercial pressures on the company, I don't understand the legal framework we're working in, I don't have the patience to build and update Gantt charts, I don't enjoy endless meetings or chasing people for every little detail of their deliverabes. The project managers know this - they don't think any less of me, just as I respect the fact they couldn't design a database schema to save their lives.
So, I would suggest trying to form a good working relationship with your project manager by trying to understand what they do for a living, understand that there is more to a project than the technical deliverables, learn to speak their language, and offer alternatives when they ask for the impossible.
The attitude of most of the posts in this subject has been "huh, we're 200 times smarter than those idiots running the project, they're so stupid they couldn't blah blah blah". Hey, if you're so smart, it's your job to use that intellgence to move the project forward, not whine about how what a bad job everyone else is doing.
(and it wasn't improved by the millions of dollars we spent on a top-brand consulting firm who implemented it), but the problems are pretty generic. Vignette does a particularly bad job of solving them, but content management is inherently tricky.
I guess most of the history of computer science has been concerned with maths and databases, so we know pretty well how to model an accounting system or calculate PI to a zillion digits. THe softer stuff - how do people actually create "words" and how do they access and manage those words is something you can't look up in an algorithm book or a pattern repository.
I had a look at OpenACS - it looks pretty cool. After working with Vignette though, I have sworn of TCL as a web language for ever.
I was the development manager for a big site using Vignette as a content management system. What they don't tell you when you buy these products is that you will inevitably invest a huge amount of time in customizing the application for your requirements, and that - at the end - it will still not feel "quite right".
Besides the purchase price, you have to assume you will need signifant training and/or consulting support, and you will inevitably end up rebuilding your site from scratch to fit into the new framework. The standard consulting fee seems to be roughly 10 times the cost of the software (amazing how that always seems to work out...).
If your content creation processes are in any way complex, you will spend a significant amount of time and effort trying to create a tool that facilitates or at least accomodates this process; if workflow features are expected, you will have a serious amount of work to tackle.
You will also need data entry time (or at least the ability to convert all the exising content into whatever format the Content Management Server expects) for all 2500+ pages.
If what you're doing can be classified as business critical in any way, you will need at least one administrator for the Content Management System and whatever database it uses.
Because content management systems involve a lot of "dynamic" activity when serving pages, it's common for them to experience performance problems, so you may need to invest in additional hardware to serve the same number of pages. You will definitely be looking at how to cache your content - this is a whole bundle of joy in its own right.
In short - make sure you want to go down this route ! It may free you up from the tedium of cutting and pasting into a static HTML file and FTP-ing it, but instead you'll be feeding and watering this content management beast.
Unless your company makes its living from its content, the costs are unlikely to be recouped.
Of course, if you want to do it, I would suggest writing one from scratch; I'm working on a JSP/servlet based content management server right now, and it's a lot of fun.
N
First thing to do : look in your bugtracking software ( you DO use bug tracking software, right ?) , and try to isolate hot spots. Is there a particular piece of code that generates more bugs than others ? Is there a common pattern to the bugs (ie. memory not being freed, of-by-one errors etc.) ? Are they _really_ bugs or mis-interpretations of the requirements or the design ? In my experience, the 80/20 rule applies to bugs in spades - it is just hard to find the patterns.
If you need to, make the bug categorisation in your bug tracking software more specific. Once you get an idea of what your hotspot is, you can work at fixing the cause of the bugs.
If it's a particular piece of code, make sure it's reviewed by the best developers/architects you have, and consider refactoring it. At the very least, insist that it is reviewed and tested thoroughly before chec-in to the source code control system, and consider adding a hurdle to jump prior to check in (e.g. get the manager to sign it off).
If the code was written by one developer, consider swapping them out and giving it to someone else - it may be they're in over their head.
Make sure you increase the number of test cases for this piece of software, and check for "edge cases" religiously - if the code is broken at all, it is likely to be broken in more ways than you realized.
If it turns out that the problems tend to have a common cause (memory leaks, of-by-one errors,etc.) consider a structure which forces developers to concentrate on those issues before checking in code; again, consider the hurdle ("software must be signed off by the off-by-one guru prior to check in"), and hone your tests to check for these kinds of errors if possible.
If the bugs stem more from misunderstood requirements or designs, beef up those areas. Work on your requirements and analysis processes; consider training courses for the developers to get them up to speed on interpreting these nebulous documents, and look at improving the review process by having designers present. Frequent "mini-deliverables" (another concept stolen from XP) will help here too - get your team to deliver a working piece of code - it need only be a minimal sub-system - and get it reviewed by designers and analysts. If the bugs tend to occur on the boundaries - i.e. invalid API calls, invalid parameters etc. - consider design by contract or aspects.
Finally, there's a bunch of hygiene stuff
N
You should seriously consider
The problem with comments is that they explain what the code does, but all too often the "why" - the structure of the program - is not obvious by looking at the comments - it's like trying to work out a streetmap by looking at the names in the phonebook.
I took over a large project from a major consulting firm; much of the code was immaculately commented, but the overall structure of the design was almost impossible to fathom; the documentation was out of date and incomplete, and everybody had a slightly different view of how things worked. Whenever we fixed a bug or made a trivial change, we'd hold our breath just in case the trivial change had unforeseen consequences somewhere else in the system. A simple class diagram and database schema would have been more useful than most of the comments. Unit tests would have saved us literally hundreds of hours of pain...
Code Complete, a book by Steve McConnell is a great read on this subject; I also recommend "Agile Development" by Cockburn.
Some people have asked what this modeling thing is all about; as a public service, here's the PinglePongle version.
/. thread...), and it's not uncommon to have projects with many times more paper than code.
Some academics regard the process of building software as a descent from the abstract to the concrete. Typically, it starts of with "hey, we need a thingumajig - it must streamline our widget creation process". This is usually followed up by a requirements spec which says what the thingumajig should do, but doesn't go into how should be constructed.
The next phase typically involves the developers working out how to build a thingumajig which meets the requirements spec.
Once the developers have worked out how to build the thingumajig, they start to code it; if you're lucky it gets tested and eventually it's released.
All these phases do occur, but the way in which they're executed varies enormously. Frequently, they overlap; frequently people move back and forth between the phases several times; frequently it's hard to tell which phase you're in.
Modelling general concerns mainly the "how do we build a thingumajig" design stage. A model can be anything that helps to convey the intention of the design without being actual code - a crayon drawing, a piece of prose, an email, a memo, a flowchart, a wall-sized database diagram, a haiku. There are a bunch of methodologies which adress the modeling phase, most famously the Unified Modeling Language (UML) which defines a standard vocabulary for object oriented design using around a dozen different diagram styles.
The problem with models is that they can become very cumbersome and end up leading a life of their own (here's a previous
As it goes, I believe that anyone who takes programming seriously should be at least passingly familiar with the most common modelling practices - UML use cases and class diagrams, database E/R diagrams, flowcharts; in my projects I ask the developers to model the tricky parts of the project, but only with the explicit purpose of understanding the problem better before starting to write code. You can change a model around easily and quickly - changing a class diagram to put a method in a different class takes seconds - but if you realize halfway through coding there's a better way to do it, you almost never have time to fix it.
I've ordered the book (no, I'm not going to tell you where from) - and if it's as good as Cockburn's other work, it'll be money well spent, even from bn.com !
I am a development manager with a large (20+) team; I am gradually introducing some aspects of extreme programming into the organisation, and so far, it's working pretty well.
The concept of user stories is great for getting our multiple customers to get to grips with their requirements; the tactile nature of the cards allows the planning sessions to be participatory and inclusive, the customers understand the trade-offs and the developers get to understand what they're _really_ being asked to deliver.
Pair programming and continuous testing have produced decent results; the main problems we have are accomodating sickness and holidays into the project plan.
On the other hand, our organisation does insist on some of the more traditional aspects of development management - a formal "Project Manager" role, regular progress reports to the executive team, functional and technical specifications etc.
I don't see how you could realistically hope to convert an entire team to XP overnight. The gradual introduction of some of the basic concepts - esp. those which can be explained as "quality oriented" - is a lot more likely to succeed.
I've read the book, and it's okay; the editing could have been a lot tighter (there's page after page of code printed out which outlines the application to be tested, not the tests themselves, there is a huge chunk devoted to the documentation for the tools which is available online etc.), and it kinda drifts away from the point now and then. If you want to get up to speed with Ant, jUnit etc. it's worthwhile, but don't expect to become an XP xpert.
The movie stinks, but there's an excellent suggestion about how to get into the zone under pressure. Our HR manager objected, though....
Content Management Systems are golf-course-ware. It gets sold to senior executives by smooth-talking sales executives who claim their products solve every conceivable business problem, is a doddle to install, standards compliant, holographic user interfaces, everything.
The reality is that this is an inherently complex field, which requires a huge amount of business-thinking before the technical solution even becomes relevant - how do you want to manage your content ? Do you have a requirement for workflow-style solutions ? Is revision control important ? Do you need collaborative features allowing several people to work on a document at the same time ? Do you have a knowledge management infrastructure so you can re-use an accepted taxonomy ? What are your security requirements ? Where does your content reside - is it largely "document" based, or is it mainly database-driven ? How technically sophisticated are your content generators ?
Only after you have worked out what you want to do with your precious content should you consider what the technology can do for you - I suggest using any one of the myriad requirements gathering techniques used in software engineering and specify your "ideal" content management system, then drawing up a list of candidate technologies.
If you start with "what can the technology do for me", you almost certainly will end up spending a lot of time and effort (and money !) and getting very little in return....
I joined a company about a year ago; they'd spent several million dollars on getting a website built by a big consulting firm, using a cutting edge content management system. "Is there any documentation ?" I asked, on my first day, and the project manager from the consulting firm proudly pointed at the shared drive with nearly a gigabyte's worth of stuff in the "Documentation" folder.
I poked around - weeding out the huge screenshots and photoshop files the designers had created - and tried to find out how the system worked; I found a folder called "architecture", and started to read. Funny thing was, most of the documents were more than 6 months old; several were more than a year old. Hardly any were written by current project members. There was no road map, no versioning. In short - the documentation - though plentiful - was badly out of date, often actively misleading, and totally pointless without the writer to explain the context and intent.
This is not an uncommon occurence; there are people with (parts of) the answer.
Check out "Agile Development" by Alastair Cockburn; he has very strong ideas on the topic of documentation and how to keep it relevant.
Next, buy "Code complete" by Steve McConnell, and make sure everyone in your development team reads it and understands it.
You might also want to check out the ideas behind Xtreme programming - it sounds to me like you're starting to find it impossible to maintain code written by other people with different coding styles and are hoping that documentation will save your bacon....it won't.
Most long-term projects rely on coding standards, a common vision and understanding of the architecture, effective communication and well-written code. Documentation is usually done because the customer insists on it, or the management team believe it's going to solve hand over problems. It rarely helps the development team to get to grips with what is really going on...
Where's the documentation for the linux kernel ?
though of course, it can be.
If you've ever been in the position where you had to recruit 20 or 30 people in a few months, you'll appreciate the help of a good HR specialist. Sure, they might not bother to forward the CV of the eccentric but inspired genius who submits his resume in object pascal. On the other hand, they will shield you from the deluge of bozos who think that because they can spell "www" they can code. Or the "I know you asked for skills in code wrangling, but I feel my experience in the catering sector uniquely qualifies me for this role" hopefuls.
I also think a good HR team can help the rest of the company to be fair about who gets rewarded, who gets promoted, and what to do about those who don't make the grade.
All this stuff is annoying, distracts from the real goal of the team, and frequently clueless - "duh - how can you compare the unique skills of a 3d artist with whatever the hell a C++ programmer does when dishing out bonuses". But just try and run a team for more than 6 months without this kind of support. Try and explain that you blew your annual bonus budget on the guy who wrote a cool screensaver, so you can't reward the team which delivered the space shuttle software on time.
It doesn't always work like this - I've been in lots of places where HR were a self-sustaining bureaucracy with no clue as to what the real organisation was doing. But when it works, the HR team is as much a part of the special teams as the people who keep the laser printer supplied with toner.
not longevity. I do think it's a good idea to have a defined career structure - I have interviewed a lot of developers in my time, and most of them want to know how their contributions are recognized. A simple hierarchy of "junior/senior/principle" or something at least shows that we have a clue on how to recognize what people contribute and that there are visible signs of that recognition.
As part of a reorg, I am trying to change my team's job titles so that instead of reflecting their specialism, it reflects their seniority. So, I've moved the "Senior designer of buttons and fonts" to a role called "Senior Developer",the "Junior designer of buttons, fonts, and logos" to "Junior Developer" etc.
This only reflects the reality of how we work - everyone moves between projects and uses their specific skills to contribute where they need to, it shows that there is a career path that reflects your contribution, rather than some org chart structure (oh, I am the Junior Database Column-adding specialist - there's no position on the org chart for Senior Database Column-adding Specialist - how do I get promoted ?).
The main risk to this strategy is that it makes it harder to explain to the senior management team what all those people do - they rarely understand what it takes to build software. So in these uncertain times, it's a lot easier to say "You want to fire Joe ? Look at the org chart - Joe is our Senior User Interface Designer - we'll always need a user interface designer - you can't fire Joe !", rather than "You want to fire Joe ? He has really good skills at User Interface Design ! Well, yeah, so does Jane - but Jane is our lead Object Specialist ! Well, okay, Gunther knows a bit about objects, but we really need his skills as..." etc.
It seems that in most organisations, HR and the executives like to know what people do for a living, and the easiest way for them to achieve that is to insist on an org chart with specific titles. The way to ease this desire is to help them to understand the way the team works, show good results on major projects, and build trust.
Nearly all the legislative approaches I am aware of concentrate on prosecuting perpetrators of security breaches or those who create/posess the tools to do so. This is ultimately not very likely to yield satisfactory results - the Internet being a global medium and all, you might find it hard to prosecute that albanian script kiddie, or the Afghani virus writer.
I believe that the best thing the government can do is to help establish a credible full disclosure infrastructure. A "not-for-profit" security organisation managing the release of vulnerability information, together with patches etc. By creating a formal infrastructure, run at arms' length, the government could signal how important Internet security is, and help establish best-practice.
It could be backed up by some kind of deal which requires all government IT suppliers to disclose to this body, and all other suppliers to have a nominated security adminstrator who monitors the new vulnerabilities and takes appropriate action.
The key issue with Internet Security is that the majority of incidents are due to poor security postures by many organisations. The inter-connected nature of the internet means that one organisation which is compromised provides a spring board for attacks against many others. Surely the common weal is served better by encouraging (though not forcing) those companies to "patch up" than by trying to outlaw tools/actions/thoughts.
The Register will no doubt notice their immortal prose is being stolen, and will employ Microsoft to create a DRM solution for text. Oh, wait, no, Adobe already did that. Now, what was that Skylarov guy doing again ?
DRM technologies are based on crypto; crypto tends to be open - a well-publicized algorithm is usually favoured over an obscure home-grown one. The products I've seen so far (and I have implemented some DRM solutions in the field) are pretty obvious. It's unlikely the media companies could control a DRM scheme - it would be too easy to crack, and you could easily imagine a napster-like revolution based on publicly available crypto and a handful of clueful bands.
I'm not saying it will happen, just that it could, and if it did, it mightn't be the worst thing in the world....
The internet can provide a marketing and PR medium - a word-of-mouth campaign works better than any number of radio plugs (Blair Witch). A meaningful, user-friendly, consumer-driven DRM mechanism can ensure content creators can get paid for their efforts, and effectively bypass the distribution network altogether. What then is the role of the middlemen ?
This works at most "human" scale projects; movies are typically too big and expensive for individuals or small groups to be able to put together, although the a DRM-enabled broadband internet distribution network might work well for smaller productions.
It is a possible outcome - not a likely one, given the clout and political buynig power of the big media companies.
(a big consulting firm, btw), they sent me a form letter reminding me that any "inventions, novel ideas, recipes, designs, concepts and business models" I had developed during my time there belonged to them, and that they would protect their property yadeyadeyade.
I wrote back and said I had come up with a pretty cool recipe for chocolate cake, and the only novel ideas I had developed involved Irene in accounting; I had come up with a business model for a company that doesn't treat its staff like thieves, and my designs could be found on the borders of the notes of the interminable meetings I had forced to sit through. It wasn't very mature, but it made me feel a bit better....
Oh, and Irene from accounting actually emailed me 2 days later.
P
The ability to book an appointment without having to go through a receptionist; links to information sites on common afflictions; links to local support organisations and healthcare businesses (e.g. pharmacies); a repeat prescription system, and of course, linux advocacy. 9 out of 10 doctors prefer it !
One of the reasons software engineering is hard is because you can not easily draw analogies. People have been building houses and bridges for thousands of years; people have been building websites for less than a decade.
The methods that work well for software projects - like XP, the spiral development cycle, patterns, OO - are young, and constantly evolving at a rapid pace. My PC from 4 years ago is a doorstop; the house I live in is 130 years old. Our local MacDonalds is full of Cobol programmers (many with CS degress) serving fries.
One of the main reasons my work can be difficult is that there is no easy way to explain to non-software folk why certain things are hard and other things are easy - to them, it all just looks like misspelled typing. The comparison between "software engineering" and "civil engineering" is partly valid, but in many ways a sofware developer is more akin to the guy laying the bricks than to Frank Lloyd Wright.
N
Why did I resign - even though I faced the same dilemmas ?
So, leave, and make sure that you explain (in private, and discretely) to the rest of the firm what is going on - and try to take them with you or find them other jobs.
Don't be the last rat to leave the sinking ship - nobody wins.
N
in principle, it should be relatively easy to write a VST plug-in which uses TCP/IP to farm out work to a music rendering farm (just as 3d graphics apps use render farms).
However, you would almost certainly never get real-time performance - you could imagine a fractal "reverb" program, maybe, where the local processor provides a coarse version, farms out the detail to a render farm and moves on, but performance would be seriously variable - a single network glitch like a request for new tasks could cause the effect to coarsen as performance degrades.
Most of the time, you would rather have no effects at all than randomly variable ones.
However, clusters would be great for apps like winchimes etc. which generate random music based on complex math - although the law of diminishing returns may set in rather early....