NYT Paywall Cost $40 Million: How?

An anonymous reader submits this musing from Philip Greenspun's blog: "Aside from wondering who will pay more than the cost of a Wall Street Journal subscription in order to subscribe to the New York Times, my biggest question right now is how the NY Times spent a reported $40-50 million writing the code (Bloomberg; other sources are consistent). Google was financed with $25 million. The New York Times already had a credit card processing system for selling home delivery. It already had a database management system for keeping track of Web site registrants. What did they spend the $40-50 million on?" Maybe the folks behind CityTime were free on weekends.

Comments

Large organization doing something simple

[Anrego]

I can actually see how this happens. Large organizations spending millions and taking years to do something a small team could whip up (and probably do a better job of) in a few months.

Different team sizes are required for different tasks. Some companies get this and put small teams together and have flexible processes that can scale to project size. Other companies can only do things one way, and that’s where you end up with insanity such as this.

You end up with layers and layers of process controlling huge unwieldy teams. You spend months just drafting the process by which you’ll operate under, and then it needs to be reviewed and this is before development even begins! You end up with 5 layers of management, each providing no real value to anything... but adding lots of time and cost.

You’ll need to gather metrics of course, so you need to figure out what metrics you need, and how you will analyse them, and how they will feed back into the dev process. And of course you’ll need someone to actually facilitate all this with some kind of metric crunching tool (which has to be bought and admined as well).

Re:Large organization doing something simple

[headhot]

I'm a consultant in telecom. I see this every day. I'm convinced that any project, no matter how big can be done by 6 people.

Re:Large organization doing something simple

[fruey]

Two well identified principles at work here (and the bigger an organisation, the more likely they are to happen, especially without strong leadership)

1. Parkinson's law : basically, work spreads out to fill the time that was earmarked to complete a project
2. Brooks' law : Adding people to a project increases lateness, because the number of communication channels to manage increases as a square of the number of people on a project

Only very sound management and trusting delegation - along with having a reasonably competent project team in the first place - can make things happen quickly.

Re:Large organization doing something simple

[rwven]

I'm a consultant in telecom. I see this every day. I'm convinced that any project, no matter how big can be done by 6 people.

QFT. In my experience it seems like, for the most part, a small, highly skilled, highly focused team can accomplish at least as much (or in some cases far more) than any large team of developers/architects. Decisions are easier, faster, and cheaper to make when you have a group of people with industry experience and know-how. The amount of code needing to be laid down for most web projects really isn't THAT large....especially when it comes to a project like this NYT example.

I also think (and this probably goes without saying around here) that a top-heavy management structure is an instant doubling (or probably worse) of time and budget for any project. I was on a development team once that had twice as many people-management and project-management positions as there were developers, and it was an absolute nightmare. Developers ended up sitting around twiddling their thumbs waiting for someone to actually make a decision and call down the order to act.

Re:Large organization doing something simple

[PsyciatricHelp]

I actually use to work for the Times as a tech and all I can say is, man you nailed it.

Re:Large organization doing something simple

[dlingman]

I'm a consultant in telecom. I see this every day. I'm convinced that any project, no matter how big can be done by 6 people.

I guess you don't have any kids then. (What am I saying. This is slashdot.)

Re:Large organization doing something simple

[bjourne]

Hah requirement analysis! Aka doing a lot of mind numbing paper work that if you are lucky the engineers are able to ignore and still produce a quality product. Or, if you're not so lucky, have to follow it because clueless managers don't know how software design works resulting in releasing something which is a veritable disaster. People are notoriously bad at reasoning about abstract things, especially if they don't have the technical know-how on how those things really work, which is why "requirement analysis" is a waste of time.

Re:Large organization doing something simple

[jimbrooking]

Agreed. Long before most of you were born, i.e., in the 1960s-70s, it was a given that if IBM told you (a customer) that they were putting another 100 people to work on a piece of late software, they were really telling you the project had been killed. And of course there's the classic Mythical Man Month (http://en.wikipedia.org/wiki/The_Mythical_Man-Month) which, of course, no one reads any more.

Re:Large organization doing something simple

[dkleinsc]

I should point out that the NYT Paywall is apparently a much more complex beast than a simple "pay up to see the articles". What they're trying to do is allow search engines, Twitter, and other social media to drive traffic to them, but at the same time not allow people to regularly read their content for free.

My guess as to the approximate cost breakdown:
- consulting fees to convince the top brass to go along with this plan even though the last attempt failed miserably: $20 million
- project management and business analysis: $10 million
- profits for the purchase manager's brother-in-law's IT contracting firm: $9 million
- 8 developers and 2 testers to do the actual work: $1 million
- Watching online readership plummet again: Priceless.

Re:Large organization doing something simple

[SharpFang]

Prerequisite: the team must be equipped with powertools to cut through the red tape.

Say, open line to the CEO who just says "yes" to anything they say and authority to fire whoever stops them from performing their task.

Depending on company structure, 10-60% of the time of any "revolutionary" change is spent actually developing the change, the remainder is asking, waiting, begging, urging, pressing, explaining, escalating and generally overcoming people who while aware of the necessity of the change and futility of their resistance, will resist the change as much as they can (or see it as the opportunity to exercise their decision-making power, which is totally unneeded and unwelcome there but by no means anyone could ever notice that.)

Re:Large organization doing something simple

[Anonymous Coward]

The depressing thing about the "big budget" development efforts is that the bigger the budget, the bigger the likelihood that it will fail.

Some of the most successful software I've ever seen was fairly ugly stuff that was knocked out in short order by less than 5 people. Years later, as it finally begins to seriously creak at the seams, the corporation puts together a massive team to bring it up to date. They spend massive amounts of time and money on consultants, fad "silver bullet" development tools and fad "silver bullet" project management techniques. 18 months into the 2-year schedule, they realize they've spent a year and a half generation UML Actor diagrams, panic, and immediately set everyone to work coding, without any coherent blueprint at all. The whole thing falls apart, lots of people get laid off, and maybe 2-3 people go in and hack the original warty system to keep it going. With luck, one can repeat this cycle several times.

Re:Large organization doing something simple

[betterunixthanunix]

And of course there's the classic Mythical Man Month (http://en.wikipedia.org/wiki/The_Mythical_Man-Month) which, of course, no one reads any more.

Funny, it was required reading in my software engineering course. Then again, perhaps it is not the developers who need to be reading it; more likely, it is management.

Re:Large organization doing something simple

[steelfood]

What they lack is vision.

You can pull off a big project with a large group of people. Countless people in the past have shown us you can do it, Steve Jobs being the most visible currently. But you need to have vision. You need to have somebody at the top going, "This is what it needs to look like in the end. This is the part you need to work on. Now go do it."

Most management meetings are more about answering the question that a single, lone visionary would've answered in two minutes, than about actually getting there. Rule by committee isn't only inefficient, it's the perfect way to get nothing done (which is why there are heads of state even in democracies). Management meetings are a form of rule by committee. Is it no wonder then that everything crawls?

Re:Large organization doing something simple

[TheRaven64]

The common reason that we're not taken seriously is that we don't have a big enough team to deliver on time, and if these larger companies can't stay on schedule then we have no hope in hell.

Next time you have a project, you can hire me as a consultant. You won't have to manage me, and I won't do any work, and will only require a token salary, but I'll increase the size of your team by 50% and therefore make you seem 50% more credible!

Its easy

[MichaelSmith]

A lot of it will have gone into executive information components of the system. Ways of showing the guys in charge exactly how much money they are making from the paywall this minute. Then you have the configuration interfaces and the teams to design datasets to control how the paywall works. Then you have the engineering which actually implements the paywall. They probably wrote a proxy from scratch to do that. Then they put it through validation. This created 10000 bug reports. Thats a lot of bugs so they outsourced the bug fixing to four companies in India who approached the solutions in 223 different ways. Then the resulting code changes were merged back into the mainline with bugs closed. Nobody wanted to do the tests again which was probably a good idea for the sanity of the people involved. Then they went live.

Well, thats my guess, anyway.

Re:Its easy

[gl4ss]

you actually forgot one bit.

hiring 'shadow developers' (also known as retards) and inflating the team size from the 10 needed to the unnecessary 100+ with testers. the times still receives just the output from 5 good guys but gets a bill for 100+. and every new person introduced to the team is more profit, as the there's more hours and persons to bill for(so they hire an unnecessary new guy/gal to inflate the team, the new person costs say 5000$ a month, the client is billed for 6000$, the company ruining executives keeping the difference . and universities have been just inventing new professions so it's pretty easy to inflate the team with sheep experts, marketing advisors, accessibility destroyers etc. and every single one of them will try to come up with output that would change the end product somehow, to justify their job).

What did they spend the $40-50 million on?

[Anne_Nonymous]

Consulting.

What else offers so little for so much?

Re:What did they spend the $40-50 million on?

[SharpFang]

The sad part is that there *are* good consultants out there. I'm one of them. I'm extremely skilled, knowledgeable, and I bring a lot to the table.

...but that's what they all say.

A simpler way.

[khasim]

Follow the money.

Someone is getting paid. Find out who and what that person's connection to the person signing off on that expense is.

Re:A simpler way.

[Reeses]

You mean they're overbilling and using the overage to fund black ops projects like unmanned shuttles.

"You don't actually think they spend $20,000.00 on a hammer, $30,000.00 on a toilet seat do you?"

Having worked for the government, yes, I do. If it's a one off project. Or only a handful.

There's no "open market" for the government to allow to absorb research and development costs or to achieve economies of scale.

If you need a special inertialess (non-rebounding, not physics-violating) sparkless hammer that can be used in an explosive gas-filled environment, and you need, say, 3. The government will commission and buy 10. The initial development and testing costs all get rolled into the cost of those 10 hammers, no matter how many man hours and resources it takes.

And, having watched the USMC test a piece of equipment in a "shaker box" designed to imitate driving a HumVee over rough terrain, hearing the salesperson from the company say, "Oh shit, there's no way this is going to....." and watching the piece of equipment explode 6 seconds into the test, there's a lot of engineering involved in meeting weird military requirements.

Is there money hidden and wasted in there? Yes, probably.

Does that hammer really cost $20,000 to develop? Yes, probably.

Re:A simpler way.

[_Sprocket_]

Follow the money.

Someone is getting paid. Find out who and what that person's connection to the person signing off on that expense is.

Sure. But don't expect to always find some nefarious link. The world of bureaucracy is bigger than just corruption. Although bureaucracies do make a breeding ground for corruption and they become more unwieldy the more the system is adapted to eliminate corruption.

I've had a lot of direct experience dealing with government bureaucracy at various levels and organizations (and some experience with corporate bureaucracy). I've seen relatively simple tasks turned in to months-long projects and couple months worth of thorough effort turned in to a multi-year outsourced contract. This isn't because someone was getting a pay-off. This is because The System, of which every good Bureaucrat serves and follows, demands levels of effort far beyond anything anyone not serving The System would think sane. So while the tasks themselves can be simple, performing the tasks within the bureaucracy requires many more additional steps that require many more man-hours to accomplish.

I should stress that corruption can still rear its ugly head. I haven't viewed it very often myself. But I've dealt with rules that have come in to place to close a loophole exposed by someone who had figured out how to game the system and got caught doing so. There will be people that are gaming the system according to these new rules and the process will repeat itself (as well as the occasional case of someone who thinks they won't get caught doing something others got caught doing).

Corporation

[moberry]

$45,000 for the implementation, and $39,955,000 in management bonuses.

repurposed joke from recently read comment:

[circletimessquare]

http://politics.slashdot.org/comments.pl?sid=2067216&cid=35703166

An investment firm is hiring mathematicians. After the first round of interviews, three hopeful recent graduates - a pure mathematician, an applied mathematician, and a graduate in mathematical finance - are asked what starting salary they are expecting. The pure mathematician: "Would $30,000 be too much?" The applied mathematician: "I think $60,000 would be OK." The math finance person: "What about $300,000?" The personnel officer is flabberghasted: "Do you know that we have a graduate in pure mathematics who is willing to do the same work for a tenth of what you are demanding!?" "Well, I thought of $135,000 for me, $135,000 for you - and $30,000 for the pure mathematician who will do the work."

thank you, SilverHatHacker (1381259) for the joke

Stupid comparison

[Anonymous Coward]

Apples: Creating a search engine from scratch. The main hook is that it is simpler than existing products. User workflow involves typing in a term and clicking a link, and users are interested in it because it's different than competing products. No money (remember, this is before Google Ads) is changing hands.

Oranges: Changing the user experience for a major existing site. Users are already familiar with the existing site and already inclined to react negatively because you're now charging for what was free. Money is changing hands, so a complete system for handling disputes and showing purchase history is required. The whole system has to hook into existing customer service systems. Customer service systems behind the scenes have to be extended. Support personnel have to be trained. Legal considerations for multiple states or possibly nations may be involved. Management needs reporting features.

Re:Why? Rupert likes to keep money moving around

[charlievarrick]

Great theory but Newscorp does not own the New York Times, it owns the New York Post.

Re:Why? Rupert likes to keep money moving around

[wombatmobile]

Rupert Murdoch doesn't own the New York Times. He owns the New York Post.

1 million for the code

[DrXym]

39 million for the oracle license.

Dude, shut up!

[jayhawk88]

Look, we told them we had to program an autonomous artificial intelligence agent to proactively scan cyberspace for hackers looking to bypass the firewall using port cross-scripting. They bought it, don't screw this up for us.

It's easy

[HangingChad]

I watched the Navy burn $27 million on a glorified CRM that used Siebel and never got any working components. While that clusterfuck was going on a small team of four people built a prototype type system that was eventually rolled out to production because it was the only one that worked.

The person responsible for the $27 million dollar disaster got promoted and took over management of the working system, which they promptly turned over to EDS to manage.

When it comes to software development, spending more doesn't necessarily get you more.

Uhh...

[Panaflex]

Well it's quite possible to "book" the value of a project much higher than you may actually outlay at any present time. Remember, these costs get deducted, deprecated and ultimately reduce tax burdens.

Or maybe it was just hookers and blow...

$20k is cheap for a hammer

[name_already_taken]

Supplier says "pay us $20k for a hammer and we'll throw in $15k of spare helicopter parts."

That's not how it works at all.

You want some helicopter parts thrown in? The military guys know to not even try that. That's not what happens at all. Everyone is covering their asses. If some military helicopter part fails, you can bet that the procurement chain will be examined. There has to be a paper trail for everything.

Supplier says "pay us $20k for your crazy over-spec'ed* hammer, and we'll jump through your stupid purchasing hoops, go through all kinds of extra work certifying things that have nothing to do with the performance of the item, fill out reams of unnecessary paperwork, send an employee to a special course so they can learn how to enter invoices in your arcane billing system (btw, commercial invoice ~1 page, government invoice ~30 pages), wait thirty days for our first billing to be rejected because of some minor issue (100% chance first billing is rejected, btw), submit corrected billing, wait 30 days for someone to tell us that the contract was shifted to another department and so it has to be resubmitted again (they knew 2 days after we billed them, but they're not required to respond until 30 days, so they don't), wait another 30 days to find that the billing was accepted, then wait 60 days for the payment to show up".

Many companies turn government business away, because the documentation requirements are onerous, the payment terms are ridiculous, and the project may be cancelled halfway through anyway.

*The requirements on military items would make your head spin. Making a tiny design change to a part to make it easier and cheaper to manufacture can trigger everything from having three government people sign off on the revised drawing all the way to having to run a live fire test at some proving ground where they strap your whatever it is to a tank and drive it around, attach it to whatever gun it's supposed to work with and fire 1000 rounds, or shoot at it, depending on what it is. All for a really minor change that was never going to affect how it worked in the first place.

You may think a $20k hammer is ridiculous, and it is, but once you see the paper and testing trail, it starts to look reasonable, assuming it's not an off-the-shelf-item (very few are). Now, if they're buying more than 10 hammers, that price had better come down, but for a one-off, $20k is a bargain. Heck, it'd probably win an award for cost savings.