The Billion Dollar Startup: Inside Obama's Campaign Tech

Nerval's Lobster writes "A presidential campaign is many things to many people: a reason to hope in the future, a wellspring of jokes and debate fodder, an annoyance to tune out, a chance to participate in the civic process. But for a couple dozen software engineers and developers involved over the past two years in President Obama's re-election effort, a campaign was something entirely different: a billion-dollar tech startup with an eighteen-month lifespan and a mandate to ship code under extreme pressure. Speaking to a New York City audience, some of Obama for America's leading tech people—those involved in the all-important Dashboard and Narwhal projects, as well as fundraising and DevOps—characterized the experience as 'insane,' filled with unending problems and the knowledge that, at the end of the whole process, nearly everything they worked on would likely end up tossed away. This is the story of what happened, and how technologies on a massive scale can make or break campaigns."

Comments

Content free campaigning

[ShanghaiBill]

What bothered me the most about the 2012 campaign was the lack of almost any discussion of actual issues. There was almost no discussion about the fiscal cliff, entitlement reform, gun control, or any other issues that the country is now dealing with. Unfortunately, the lesson seems to be that keeping campaigns content free, and instead focusing on social media, turnout, and the "ground game", is the way to get elected, even if it isn't good for the country.

Re:Content free campaigning

[cayenne8]

What bothered me the most about the 2012 campaign was the lack of almost any discussion of actual issues. There was almost no discussion about the fiscal cliff, entitlement reform, gun control, or any other issues that the country is now dealing with.

Of course not...that wasn't in their best interest.

And the masses would actually have to *think* and try to understand tough things like "issues".

I don't think we've actually had an election where the candidates have actually addressed issues since maybe the early 80's or slightly before.

Re:Content free campaigning

[Anonymous Coward]

Nobody wants to hear how similar real candidates are. They want a football game. Someone to hate and someone to cheer for.

Really it's more like a very long, very complicated chess match with an unequal number of pieces. You take an endorsement from this group, let the other guy take the endorsement from that group. Up the ad budget in these four states and concede these other ones to the other guy. Spend those dollars elsewhere. Say you're pro-choice, but you're for parental notification. That gives up dollars and votes from these people but gets you dollars and votes for these other ones. Everything is negotiable. Not because they're open to changing ideas, because they're open to slightly different, more effective strategies.

Nobody runs for president because they really, really believe in certain things. They just run for president, and use the party with the platform that casually aligns with their own. The rest is bullshit politics.

Re:Content free campaigning

[Bill, Shooter of Bul]

Yes there was. The republicans relentlessly talked about reforming social benefit programs, Obama relentlessly talked about the need for higher taxes. They didn't mention gun control, though.

Re:Content free campaigning

[Stiletto]

Why focus on changing people's minds (difficult) when you can simply focus on voter count (easy). Democrats tend to win when voter turnout is high, and Republicans tend to win when voter turnout is low. So, depending on your party, it's more effective to invest your efforts into Get Out To Vote or into Voter Suppression, than it is to try to change peoples' minds.

Re:Content free campaigning

[danbert8]

They were eliminated in favor of minting trillion dollar coins.

Re:Content free campaigning

[datavirtue]

Nice talking point, but we must acknowledge that no matter what we collect from the rich it will never keep up with what the government is spending. Fun fact.

Re:Content free campaigning

[jjohnson]

And yet, the U.S. went over the fiscal cliff, clawing back only at the last second. And only be delaying the debt ceiling fight for several weeks. Do you think that's what O wanted? Of course not. He wanted a grand bargain, and didn't get it. So where's his omnipotence now?

Re:Content free campaigning

[able1234au]

The result would be the same with a popular nationwide vote and the plus side is that there is no shrub! But in practice the electoral vote does mean that states that would be ignored in a nationwide vote do get focus. And that focus moves around. Missouri and Indiana one election, Colorado and Ohio this election, Pennsylvania in the next election

Re:Content free campaigning

[thesandtiger]

Well, the first problem I see is the thinking in terms of "sides" - that encourages bullshit grandstanding and that whole rah rah we can't ever say we were wrong about anything because then THE OTHER SIDE wins. It makes us all stupider as a result - we can't ask questions without the other side thinking we're trying to trap them or saying that we're ignorant, we can't admit fault because HOLY SHITBALLS we were wrong about one thing and therefore we must be wrong about EVERYTHING EVER and all that nonsense. Fuck that noise, it's stupid, and we can do better - and I expect people to be better than that.

The good news is that if we're smart we can do small changes that will gently push us onto the right track. The bad news is that we have people who are too fucking stupid to understand any of this, and many of those people are in positions of power. Anyway.

We have 2 parts to this: Raising Revenue and Cutting Spending.

Raising revenue is, by and large, pretty easy: Raise taxes a couple of points, more on people who can afford it, less on those who can't afford it, putting more money in the hands of the people who will most likely spend the money they aren't paying in taxes. I'm not talking about soaking the rich or anything of the sort - a few points, that's it. Also raise revenue for social security by removing the income cap, but still keeping a cap on maximum benefit payout. I know, it's not "fair" to the people who make millions of dollars a year but who will only receive benefits as if they made a hundred k a year or so, but somehow I think they'll be just fine. And no, as long as we don't get stupid ("OMG LOL LETS JUST TAKE EVERYTHING THEY HAVE!!!!!") there won't be some massive "Gone Galt" extravaganza - by and large, living in the US is pretty fucking amazing compared to many other places in the world, and even with higher taxes, living in the US and being rich is HUGELY FUCKING AMAZING compared to many other places in the world. You'll still be able to live the lifestyle you want, and even better you can do so with a slightly lowered risk of being executed by a mob of armed peasants with nothing left to lose. It's a win-win!

But for cuts:

I'm hugely in favor of cutting the fuck out of spending on social programs and defense but we have to do it INTELLIGENTLY. Simply saying "hey, welfare, you have 25% less money to spend, figure it out" is fucking retarded because it winds up being done in the most brutal way - reducing services or cutting some people in need off of services. Simply saying "Hey, DOD, you have 25% less to spend, figure it out" is fucking retarded because there are so many vested interests involved that we would wind up with, like, 3 hot-shit pilots flying the most amazingly expensive fighter jets ever, a couple of billion dollar cigarette boats cruising around the gulf, and I don't know what the fuck else. In any case, I used cuts in my initial comment as a shorthand - and I regret that I did. What I mean to say is that we need to be more efficient in how we spend our money - we need to figure out what mission we want to accomplish with whatever we're spending money on and then really REALLY look at whether or not our dollars are actually contributing to that mission.

One problem we have with many cuts is this: Things that are obviously fucking stupid and can be cut with impunity tend to be actually pretty small portions of any budget. But, yeah - cut them anyway, because it all counts. The things that are REALLY expensive, though, aren't so obvious and easy to cut, and come with trade offs.

Another problem is that so many programs are inextricably linked together, where changes in one have an effect on another. Ex: law enforcement and mental health services. Turns out, if you cut mental health service budgets, you wind up with an increase in crime - many people who are seeking health for substance abuse treatment or depression or other, more serious, issues wind up getting turned away and then committing crimes. It turns out that, when these people commit

More like a 10M startup ad brokerage

[Overzeetop]

Yes, but they don't have to deal with 50 individual, winner-take-all races over several hundred markets with three hundred million voters.

And, to be fair, most of the 1.1 Billion spent by the Obama campaign was spent on advertising slots and ground game (rental, printing). This wasn't really a $1 Billion startup, but rather a conduit for $1B in spending. It's like saying your stock broker is a billion dollar operation because he directs clients 401k money for a 10,000 person corporation.

when the infrastructure and software was complete

[schlachter]

Obama turned to his team and said..."you didn't build that!"
I keed, I keed.... :)

Always has been, always will be.

[Overzeetop]

You're just now getting this? Every election is about the little shit that doesn't really matter much. It's about emotion and flash. It always has been. Look back 100 years and it will be the same thing. Look back 200. Mudslinging, character assassination, out-of-context quotes, outright lies have always been part and parcel of the political election process. Sure, we can do more and make more convincing fakes with technology (autotune the news, anyone?), but it's also easier to fact check.

Contentless politician banter is anything but a recent phenomenon.

Re:Always has been, always will be.

[ShanghaiBill]

Every election is about the little shit that doesn't really matter much.

Plenty of elections, even recent elections, have put big issues in front of the voters. Even Obama, in 2008, made health insurance reform an issue in his campaign. This election the only issue seemed to be whether taxes would go up on 1% of the taxpayers or 0%. This campaign was unusually content free.

Look back 100 years and it will be the same thing. Look back 200.

Read some history books. Look at the election of 1864. You think that was content free? "Continued the civil war till victory" vs "peace through negotiated secession" seems like a pretty big issue to me. Big issues were raised in the presidential campaigns in 2008, 2004, 1992, 1980, 1972, 1968, 1964, etc.

Re:Always has been, always will be.

[datavirtue]

Becasue none of the candidates wanted to talk about their "accomplishments" which would have certainly led to their blatant incompetencay and failures. Romney technically was the better choice but no one wants to debate technical merit as has been pointed out. In all seriousness you need a practical business man to head the government becasue their leadership and ability to make decisions has been proven, everything else is but the flip of a coin. Instead we got inbroiled in class warfare which Obama stoked and continues to stoke. This is not leadership, it is devisive and irresponsible. Perception still rules the day--forget facts.

Re:Always has been, always will be.

[TapeCutter]

Are you for real? You complain about "class warfare" and in the same paragraph basically claim that a class of people (successful business men) are the only class of people who can be trusted to lead properly. And you don't see a glaring contradiction?

Please don't take the above as an insult, my intention is to be blunt, not insulting. Politics requires "critical thinking skills" more so than science but rarely are they applied with the same rigor. The reason you fall for the "class warfare" bullshit is because your vote with your gut, not your brain. You actually need to use both, know your emotional triggers and question your own basic assumptions, such as the assumption you display in the post that says something like "governments and corporations are very similar", once you accept that assumption as gospel, what do you then do to balance the power that you have handed over to wealthy merchants?

Just remember one fact when people talk about "class warfare" - ALL governments redistribute wealth and they all claim their particular formula is for the "common good", it's not "class warfare" it's their purpose, it is the very definition of civilization itself. Sometimes the wealth piles up in great mounds on the "elite", sometimes it stolen and squandered by corrupt officials or angry mobs, every now and then a "booming middle class" appears as it did in the US after WW2, and is now doing in China after Mao's famines.

Yes, I have an assumption that a "booming middle class" is a GoodThing(TM), but I don't think anyone really knows exactly how to create one (please don't send me newsletters). Truth is, if you take the time to look there are good and bad ideas from all sides, politics should be about shaping society in OUR own image, that image should not be preemptively restricted to the economic heroes of of the day.

Re:Always has been, always will be.

[dkf]

Sometimes the wealth piles up in great mounds on the "elite"

Actually, the place in society where the wealth piles up pretty much defines who is part of the elite. There are many possible reasons for the accumulation to happen (inheritance, business, kleptocracy, all sorts of possibilities) and many people think that the reason for it is terribly important. I'm not so sure it is though: I prefer to abstract all that away and use a thermodynamic model where the basis of economic activity is random exchange of money (totally an approximation!) The amazing thing is that *just that* (and the fact that money isn't infinitely divisible) gives an accumulation of wealth, an elite, and a middle class too. Of course, once you've got a thermodynamic model, you can measure its temperature (with many thanks to Ludwig Boltzmann). What I don't know, but suspect is true, is that healthier economies have higher monetary temperatures, that recessions are when the temperature is falling and booms are when it is rising.

Of course, it's vastly oversimplified. But it is an interesting thought experiment that any true geek ought to appreciate as it is rooted in universal truths and abstracts away from vast amounts of confusing detail. Probably scares the pants off social scientists though...

Huge contrast from Republican IT op.

[Anonymous Coward]

The election politicians themselves were boring and predictable. (Well, once the primaries were over. I'm still convinced Herman Cain is actually a comedian who's work rivals that of Andy Kaufmen)

The stories about the IT side of their campaigns was pretty interesting. Obama's crew put together a really interesting and very modern piece of software that scaled up and scaled down in a way pretty much unique to it's purpose. Think about it. You need a piece of software that goes from zero users, to literally nation wide in every corner of every part of the country in a span of a few months.. And then it all ends in one day. How do you do that? How do you pay for that? Well, I remember at one point someone mentioned that it took a significant chunk of the entire EC2 cloud at it's peak usage.

Interviewers with developers made it obvious that these were very competent and enthusiastic engineers. They were involved in campaign ops from day one and the whole op was considered a huge sucess. .. Contrast that from Romney's op. It was clearly a subcontracted piece of software written by an outsourced developer with no little from the campaign. It ran from a single datacenter on a few fixed servers. Very compartmentalized. Very businesslike. And it failed miserably. It was late, crashed under nowhere near it's needed peak load, and left their organizers stranded with no information.

Re:when the infrastructure and software was comple

[Anonymous Coward]

I was at first taken aback when I heard the clip of "You didn't build that". I just had to go youtube the Obama speech that it came from. Wow, he was talking about the infrastructure - roads, post offices, other public services... that all businesses count on to run. His point was that some things are best done by businesses, and other things are best done by Public Sector. For example, what company would take on building and maintaining Interstate highways throughout the US? Funny the statement sounded when taken out of context.

Re:SIGH....

[Anonymous Coward]

Yes, a brighter future... Since Bush left office, the deficit is down, unemployment is down, the economy is up and healthcare costs have gone down (my insurance provider sent me a refund this year, and as a result dropped my premiums. People who tell you they will increase are lying. Obama has pretty much undone the path to complete economic failure that Bush set us up for.

Don't pick a team. Look at the facts and figure things out for yourself. The country as a whole is doing far better than it would have under any current Republican candidate. The current Republicans only care about short-term gains for themselves and their cronies, while the Democrats are actually trying to make things work long-term.

Re:Their system really worked

[coinreturn]

Juicy. Of course, if you want to play "what if" and hold suppositions as hard evidence, we could also say that the only reason Democrats did as well as they did this last cycle was because they are so successful at class warfare.

Successful class warfare? So nudging upper tax brackets barely above their historic lows is successful class warfare? Successful class warfare is having a salary cap on FICA tax, little or no tax on capital gains, historic-low tax rates, increasing the retirement age, record-high executive pay, bank bailouts on stupid loans, sole-source wartime contracting, pay freezes for civil servants and teachers, etc.