Large Dev Teams Do Not Make For Quick Dev Cycles

Josh Bennett writes "1UP has a recent interview with Splinter Cell Chaos Theory Producer Mathieu Ferland where he talks about the difficulties in developing the game. In the article, Ferland said there are 120 people working on the game. That's not unheard of for a big budget EA game, but those games come out every year and the new Splinter Cell is taking more than two years at this point. Interesting read."

Comments

Mythical Man-Month Anyone

[Anonymous Coward]

It's not exactly an unheard of concept. Sheesh.

So that's why...

[KDR_11k]

...it's called Splinter Cell Chaos Theory?

sure, ask a carpenter

[typhoonius]

You can't build a house in a week no matter how many men you throw on it. After a point, your returns diminish.

Re:sure, ask a carpenter

[mcmonkey]

I've got nine women lined up; I'll let ya know in a month if we've made a baby.

Re:sure, ask a carpenter

[El]

My favorite quote is "You can't make a baby in one month by getting 9 women pregnant." All projects can be broken up into smaller tasks, but most tasks simply do not parallelize very well, so your critical path remains the same no matter how many bodies you throw at the problem. Also, the interface between one person's area of responsibility and everybody else's must be clearly documented. With a single developer, he spends 100% of his time getting things working and 0% of his time documenting interfaces. With a large group, most spend 90% of their time documenting and explaining their own interfaces and learning other people's interfaces.

Re:sure, ask a carpenter

[vasqzr]

You can't build a house in a week no matter how many men you throw on it. After a point, your returns diminish.

Have you ever seen Habit for Humanity build them in one day?

The only thing you have to wait on is the cement and paint to dry.

Re:sure, ask a carpenter

[Jerf]

You can't build a house in a week no matter how many men you throw on it. After a point, your returns diminish.

Have you seen that makeover show, I think it's on ABC, that has done just that? One house in particular actually had to have foundation work done on it. (I don't watch it routinely, just caught it a couple times.)

I actually don't say this to disagree with you. One of the reasons neither my wife nor I can really stand to watch that show regularly is we both know you can't build a house from the foundation in a week... but you can build a television set from the foundation in a week. We have a rather strong suspicion that as neat as these houses look on TV, and as cool as they look on the surface (eliciting the cries of joy from the new owners), that these people are really just getting television sets. And those are no fun to live in.

I don't know, I'd love to be wrong, but the suspicion that these make-over-ees are getting boned wrecks the show for us. If 20/20 or equivalent show from another network followed up on one of these homes after a year or two, and everything was peachy within reason, maybe I wouldn't feel this way. But I suspect "peachy" wouldn't be the right word.

Re:sure, ask a carpenter

[LordNimon]

True, but if you can get nine women pregnant, you get to say, "Yeah, baby!"

(with apologies to Austin Powers)

Re:sure, ask a carpenter

[El]

Just remember: class action paternity suits are a bitch!

Re:sure, ask a carpenter

[Anonymous Coward]

i think it usually takes them a while to get a single house done... unless you're talkin about a house w/ no foundation and stuff like that..

Re:sure, ask a carpenter

[Anonymous Coward]

Worse: lass action paternity suits are more than one bitch!

Re:sure, ask a carpenter

[OwlofCreamCheese]

I was like "they are gonna live in a television set?"

then I was like OOOOHH! a set! that kind of set!

Re:sure, ask a carpenter

[N3Z]

From what I've seen, three weeks would be a more accurate description. They seem to work three shifts.

It would be nice to see what kind of problems show up in a year though. There must be some substantial shrinkage in some of the products used.

Re:sure, ask a carpenter

[Fulcrum of Evil]

Have you ever seen Habit for Humanity build them in one day?

Wow!

The only thing you have to wait on is the cement and paint to dry.

Oh, wait, wasn't that the whole point? Concrete still takes a few days to dry and all that.

Re:sure, ask a carpenter

[Anonymous Coward]

If 20/20 or equivalent show from another network followed up on one of these homes after a year or two, and everything was peachy within reason, maybe I wouldn't feel this way. But I suspect "peachy" wouldn't be the right word.

I recall an episode of Monster House where they did exactly that.

C'mon

[Anonymous Coward]

a) The title has nothing to do with the blurb

b) The blurb has next to nothing to do with the article

Re:C'mon

[Anonymous Coward]

c) The article has nothing to do with the comment.

d) The comment has nothing to do with the mod.

And all is well in /.land.

Re:C'mon

[Anonymous Coward]

Slashdot Rejection Form

This article is about:

( ) bashing MicroSoft
( ) music or movie piracy
( ) hackers
( ) somebody patenting a worthless idea
( ) what Sun is doing wrong

Although it fits one of our major categories (above), your article
was rejected because it:

( ) wasn't rabidly pro-Linux
( ) wasn't sufficiently ironic
( ) made too many Star Trek references
( ) failed to link to a low-bandwidth server
( ) promoted your own crappy website too prominently
( ) was too right-wing or too centrist
( ) had correct grammar and spelling
( ) referenced Fark, Kuro5hin, The Reg or the Mainichi Daily News,
all of which are frequently more entertaining than we are.

On the positive side, your article had some of the characteristics
that we are looking for:

(x) creates a certain sense of deja vu
(x) makes a wild speculation unjustified by the referenced article
( ) mentions an outrageous lawsuit or criminal trial
( ) announces a minor BSD patch release
( ) explains a new idea for space travel
( ) uses the word "teraflops"

Keep sending and sooner or later we will mangle one of your
submissions.

Thanks

Re:C'mon

[edittard]

Although it fits one of our major categories (above), your article
was rejected because it:

(x) wasn't submitted by michael's bum-chum Roland Niquetamere

Computer games have been around...

[Conspiracy_Of_Doves]

for something like 25 years, and people are just now figuring this out?

No duh, you friggin idiot.

[Safety Cap]

In any group, the number of communication paths is

C =
n(n=1)/2

Obviously, the larger the group, the more communications events that it will require to get the job done, but it is not O(n).

A team of two developers only has 1 communication path.
A team of 10 has 45.
A team of 20 has 100.

News for social misfits, stuff that is glaringly obvious.

Re:No duh, you friggin idiot.

[MalleusEBHC]

That's why you have *gasp* management! I know the word management generally only conjours up images of PHBs to most people here, but the only way you are ever going to get a group of 120 people to work towards a communal goal is to break them into units (and most likely sub-units) and have the managers handle the inter unit/sub-unit communication. That way the people on the bottom can go about working on their little section of the game while their manager makes sure their section fits in with the rest of the groups.

I fear the day anyone thinks this is a load of crap and sticks 120 engineers together on a project with no sort of leadership hierarchy.

Re:No duh, you friggin idiot.

[El]

Yes, but can't most information be broadcast, instead of communicated point to point? E.g. send a single email to everyone on the team, don't compose a separate email to answer a single person's questions. (Ok, this still adds overhead on the receiving side, where each receiver has to filter out all the stuff they already know or didn't want to know.)

Re:No duh, you friggin idiot.

[sporty]

You don't have the developers talking to each others directly. You have lead developers or architects talking to each other. THey agree on deliverables and communicate changes that way. Instead of having a * structure, you have a small star with offshoots.

Re:No duh, you friggin idiot.

[vhold]

But it sure would be an interesting experiment though.. Assuming there was already a common goal, what do you think would happen? Assuming there is some quality to the engineers, I think they'd immediately recognize the need for organization and the first problem they would solve is establishing that by some kind of engineered process. Without a formal process, I think the natural leaders would emerge as the people who spearhead the most agreeded upon initiatives, people would naturally align themselves with the projects they feel most comfortable with.

But doing that with -120- people would be pretty nuts. I've seen natural hierarchies emerge in pools of 10-15 people working on a project, but 120 would definitely be something else. A big factor that I've seen with natural leaders emerging from a pool of previous equal individuals was the leaders' initiative to acquire neccesary resources. Whoever acquired neccesary resources first is

If you ran an experiment with preacquired resources that nobody involved was responsible for acquiring, I think the natural method of acquiring a leader would become seriously muddied by politics instead of practicality. Acquiring resources is something that can be demonstrated immediately. Spending them is something you can't judge until it's too late if you are wrong.

Re:No duh, you friggin idiot.

[vhold]

err.. weird typo.. meant to say that whoever acquired resources first is most likely to be trusted with managing those resources.

Re:No duh, you friggin idiot.

[ChaosDiscord]

Several people have pointed out that management is supposed to solve the problem. Indeed, with good management you can have a team of 120 working quite effectively.

However, mediocre to bad management is far more common than good management. With mediocre or worse management the team ends up routing around management problems and reestablishing those hundreds of communication paths. Thus, your pessimistic estimates are reasonable for real world situations.

The key is good management. Finding and retaining good management is left as an exercise for the reader.

Re:No duh, you friggin idiot.

[Darth_Burrito]

Additionally, the notion of bad management seems pervasive within the technical community even though it's not always justified to blame management. Some projects are simply very hard to manage effectively. The more people you have on a project, the more distributed general and specialized knowledge becomes.

I guess what I'm trying to say is project management is hard and it gets harder the bigger the mess becomes. Blaming managers, as tempting as that may be, is not always the answer. More to the point, in any system, if you find yourself routinely blaming a particular part or type of part (eg a manager), it is likely the fault lies with the system and not the part.

I destroy my enemy when I make him my friend.

Re:No duh, you friggin idiot.

[arose]

That only works if get Linus as your manager. :-P

Re:No duh, you friggin idiot.

[edittard]

E.g. send a single email to everyone on the team, don't compose a separate email to answer a single person's questions.

So everyone gets 900 emails a day, of which three actually concern them. Has anyone ever told you that you're a genius?

Re:No duh, you friggin idiot.

[Hognoxious]

You don't have the developers talking to each others directly. You have lead developers or architects talking to each other. THey agree on deliverables and communicate changes that way. Instead of having a * structure, you have a small star with offshoots.

That's all well and good in theory, but a "chinese whispers" effect soon puts the kybosh on that.

Imagine this scenario: programmer in the widgets team talks to team leader (widgets) who talks to team leader (I/O) who talks to programmer in the I/O team. Even if only 20% of information is lost at each stage (some say that's a conservative estimate [gnnnnn ... must ... avoid ... cheap ...shot ... at ... managers]) it's nearly half being lost on the way.

Re:No duh, you friggin idiot.

[sporty]

Documentation documentation documentation, especially in a large group. If something is described as working one way, and then it is changed, the written documentation must be cahnged and made clear what is going on. This includes unit tests and examples if necessary.

Re:No duh, you friggin idiot.

[Hognoxious]

Again, fine in theory. Of course, in the real world you run into "I'm too busy coding to do documentation" (sometimnes found with "I'm too busy coding to read documantation" or (my personal favourite) the source code pasted into a word document.

Re:No duh, you friggin idiot.

[sporty]

Then its the manager's fault for not following a protocol for a large development group. *period*

And I've worked in small and large groups, where it's failed and succeeded all that random times.

Re:No duh, you friggin idiot.

[sjames]

That's why you have *gasp* management!

Competant functional management can indeed handle this, but it takes more. The project itslef must be designed to facillitate modular sub-division. Each module can then be developed by a much smaller sub-team werer the number of communications paths will be small. In turn, the sub-team leaders must meet as a group but will also have fewer communications paths to deal with.

Poor management can sabotage that sort of thing by regularly 'inviting' the whole dev team to a meetings or failing to respect the communications groundrules (likely unwritten) within or between sub-groups, perhaps by attempting to formalize water-cooler talks rather than simply facillitating them.

If enough of the developers are sufficiently experianced, especially if they have experiance with community development of Free software, they will likely self organize into subgroups and will consider the need for modular development to be obvious. In that case, the worst thing management could do to productivity is attempt to impose a structure on the team.

Re:No duh, you friggin idiot.

[Hognoxious]

Then its the manager's fault for not following a protocol for a large development group. *period*

Never said it wasn't - and if you've only ever worked for good managers, on projects with good processes, then you're lucky. The same loss of meaning with spoken communication also arises with written communication - especially in a multinational. Sure you can then QA the documentation and specifications, but that adds even more overhead.

Maybe I'm getting cynical in my old age, maybe it's because I'm working on a project that has a "code first, think second" mentality, but I don't see a solution that isn't almost as bad as the problem, or that will reqire a massive fight. YMMV.

gasp, programmers don't scale!

[evilmousse]

could it be, that adding programmers is a lot like adding processors.. doubling doesn't quite result in 200%, as more and more resources go towards coordination?

btw, the article SUCKED, i went in expecting to read even just a little bit about large-scale-development and all i got was a crappy 2-page splinter-cell advertizement. that and firefox asked to install 2 different plugins so i can better experience further, flashy-er advertizements on my plain-text advertiziment.

Stunning revelations from the 1960s!

[cbiffle]

Sounds like some more people should read Brooks' Mythical Man Month. There's a reason this 40-year-old book still inhabits my bookshelf at work.

'Course, from how EA seems to treat their programmers, it sounds like they're not really considering any human aspects of the cycle, so I suppose this is not surprising.

Mythical Man-month still applies

[xmas2003]

Slashdot reviewed Fred Brook's classic The Mythical Man Month way back in 1998. This book was actually written in 1975 based on is experience in the 1960's ... so while the /. review is 6 years old, it still holds true today and applies in this situation IMHO ...

RTFA (yawn)

[emiddlec]

I don't understand why this article is worthy of discussion. If you actually RTFA, there is very little to discuss..

• With a recently announced delay pushing the release to March, a staff of roughly 120 people, and the development cycle pushing past the two year barrier, Splinter Cell Chaos Theory isn't exactly being pieced together by two kids in a garage. But according to the game's Producer, Mathieu Ferland, that doesn't mean it's been an easy game to make. We recently sat down with Ferland to find out about the development, what got left on the cutting room floor, and the possibility of a spin-off multiplayer game down the road.

  ...

  Much of that technology research has been aimed at overhauling the game's graphics engine. Though the previous two Splinter Cell games received praise for their graphics, the developers decided to use new normal mapping techniques to create shinier surfaces and add detail in the backgrounds. But according to Ferland, this technology wasn't as easy to implement as the team hoped. "The biggest surprise [in developing the game] was the management of the data," he says. "The integration of normal mapping and shaders changed the whole pipeline for development, and it's been much more complex than what we expected originally. A lot of people say, 'Oh yeah, we've got normal mapping in our game and blah blah blah.' Wow, hold on -- if you're adding a layer of shaders and all this, and you want it to be consistent in a realistic world, it's very, very complex. This is one of the reasons why there were 60 people working on Splinter Cell 1 and 120 working on Splinter Cell 3."

Okay, let's see here.. we have..

1. 120 people
2. 2 year cycle
3. technical difficulties (shocking)

I assume they started with 60 people, hit technical difficulties, then increased to 120 people. So what?

Re:RTFA (yawn)

[gl4ss]

the _REAL_ reason why they have 120 against 60 is that they can afford it with their shiny bigger budget because splinter cell is now a 'hit' franchise.

the sad thing? if previous games by anyone are to go by.. humongous teams don't make imaginative games... it's not even a guarantee that it's polished(unless you coun't the ea's yearly games.. they're pretty polished but then again they've been polishing them for the past 6 years..).

blablabala buzzword1 blablba buzzword2 blabla so we need double the people, give us budget!!!

Re:RTFA (yawn)

[xerocube]

I assume they started with 60 people, hit technical difficulties, then increased to 120 people. So what?

That's exactly the "what." If you start with 60 ppl and then hit technical difficulties -- throwing people at the problem will not necessarily increase production time.

You would have to take into account the time it would take to orient all the new staff and to train them to the culture and technology in use. So, doubling the staff would force you to invest a hefty chunk of resources (read: time, money, coffee...) before you'd realize any significant benefit to the project.

Just my opinion... I could be wrong.

I'm not buying EA again after they did.

[Anonymous Coward]

I just won't support a tyrannical company who uses their employees like a pirate ship used pirates. Go steal the treasure (Do all the work, get killed), sail the high seas (get scurvy, die), and when the treasure is buried you all get disposed of because you aren't needed anymore. EA worked them to the bone, disregarded their families, and probably gave them various mind sicknesses from lack of sleep. I hope the ex-workers in the class action lawsuit take EA for all they can.

Re:I'm not buying EA again after they did.

[Anonymous Coward]

That's great and all but... this is a Ubisoft game.

Re:I'm not buying EA again after they did.

[Anonymous Coward]

Your analogy is a little flawed. Most pirates didn't bury loot, they went and spent it on whores and booze at port. It was also a quick way to make a buck in its heyday.

Re:I'm not buying EA again after they did.

[Anonymous Coward]

If you really feel that way... don't buy games at all :)

Ion Storm?

[All_Star25]

Wasn't Ion Storm already testament to this fact? If memory serves, the Daikatana team was large, and everyone knows that game's saga.

In related news...

[tsm_sf]

Battleship not as nimble as speedboat, say boffins.

Hmmm

[THCLothar]

Maybe the develpment team has been worked to the bone and they're just not as productive as when say they get sleep. If you don't know what I'm talking about, read the previous and more recent articles on working conditions for EA. I'm starting to think EA really stands for "Everybody takes it up the Ass"!

damn

[Sv-Manowar]

too many cooks spoil the broth... maybe ID could tell you about it

The Tao of Programming

[RayMarron]

From the Tao of Programming (http://www.canonical.org/~kragen/tao-of-programmi ng.html):

A manager went to the master programmer and showed him the requirements document for a new application. The manager asked the master: ``How long will it take to design this system if I assign five programmers to it?''

``It will take one year,'' said the master promptly.

``But we need this system immediately or even sooner! How long will it take if I assign ten programmers to it?''

The master programmer frowned. ``In that case, it will take two years.''

``And what if I assign a hundred programmers to it?''

The master programmer shrugged. ``Then the design will never be completed,'' he said.

Re:The Tao of Programming

[iminplaya]

Paraphrasing:

The band costs $500 with reheasal.
How much is it without the rehearsal?
You can't afford it.

I think I should point out

[ikkonoishi]

That all of them aren't coders.

120 people seems to include all the artists and map designers as well.

Art works a lot more smoothly than coding when you have a large number of people.

And in other news..

[rjshields]

Large herd of buffaloes takes more time to cross river than small herd..

No real or imagined similarity between programmers and buffaloes is implied.

Yeah, that's pretty YNSA

[smileyy]

YNSA == Yeah, no shit, asshole