The Hacking Contest Nobody Tried to Win

When Slashdot reader magenta was invited to participate in the Loki Hack Contest attached to Atlanta Linux Showcase, his first thought was, "There must be someone more deserving of this high honor; after all, I'd never even seen Civ:CTP. But after a few exchanged emails and a bit of chatting with various friends who all told me it would be a good idea (thanks, Ross and Joel), I reluctantly decided to accept the invitation, and booked myself a cheap flight to Atlanta." (More below)

The Hacking Contest Nobody Tried to Win -by magenta

I am very glad that I decided to enter.

First off, nobody there, including myself, cared about the potential for prizes. There was a great feeling of camaraderie, of friendship. Thirty people (including Loki Entertainment's representatives; very few people, as it turns out, actually signed up, probably due to some of the FUD spread on Slashdot regarding Loki's intentions) talked to each other as though they were old friends even though we'd never met each other online, much less in real life. We all exchanged ideas for hacks, bounced ideas off each other, and discussed means of implementation. The attitude was the opensource mentality at its best.

And then we saw the source.

The official Loki Hack '99 t-shirt has a slogan on the back. The slogan is "I saw the source and lived." This slogan is incredibly apt. At the risk of offending Activision (which I certainly don't intend to do), Civ:CTP isn't exactly the best-engineered program around. Imagine the original release of Mozilla, put into a blender on frappé, shoved into a toaster oven and let to sit on medium heat for a few days. And then microwave it. The individual pieces of code weren't too bad (if not inelegant), but the entire package wasn't arranged in any semblance of order. The first hack I wanted to perform was XaoS-like smooth zooming on the map, so that when someone clicked on a zoom button, it would do it smoothly. I couldn't find, nor could the Loki people tell me, where any sort of useful callback would be. After tracing through about 10 levels of indirection, going past hard-coded constants and multiple redundant and nonsensical calls to seemingly-random parts of the source, I gave up and decided to work on other things.

But I saw the source, and I lived.

That is such a cool concept. Namely that I saw the source. The source to a very commercial, very proprietary program, from a very un-opensource company. Activision put a lot of faith and trust into this experiment, and hopefully they will find that it was well worth it. In just 48 hours, a couple dozen programmers who had never seen the source before added in some incredible changes. I rewrote the map generator to generate quasi-fractal terrains with randomly-combined algorithms. The person sitting next to me changed the game logic so that different civilizations would have different abilities (rather than just being different in name, one tribe could run faster but be weaker, while another one could produce technology more quickly but not generate wealth, or whatever), which was no mean feat considering the network-play mechanism. Various people ported SDL (the low-level graphics API written by Loki for the purpose of having DirectDraw-esque functionality in Linux) to varying other APIs, such as Clanlib and GGI. One person, whom I now consider myself friends with thanks to this experience, added in hooks so that during a network game, you could launch an arbitrary program (such as tetris) while waiting for other people to finish their turns, and also added in hooks for publishing the gamestate to an external server, so that people can watch the game from the outside, getting a play-by-play summary for example.

I saw some very cool things. One of the reasons I chose to work on the map generator was that unlike the main program, which took several hours to build fresh and 10 minutes to link (on a quad Xeon-600 with a gigabyte of RAM), the map generator was implemented originally by Activision as a COM object. Sam of Loki decided to take the next logical step when porting this code to Linux by making it a separately-linked set of shared libraries to be linked at runtime. Of course, Linux doesn't have any good COM implementations, certainly none which are readily-available and behave the same way as MSVC's, so what did he do? He basically reimplemented the most important functionality (QueryInterface, AddRef, and DelRef) in about 10 lines of C++, confirming my suspicions that without DCOM, COM is just a C++ refcount garbage collector.

The competition (or lack thereof) was basically the embodiment of open source. Twenty or so talented hackers teamed up to make sweeping improvements on someone else's code, without any real desire for glory or money. Most of the Slashdot naysayers were saying that Loki was obviously trying to take advantage of talented programmers by conning them into giving up quite a bit of labor for free. Well, Loki certainly didn't get it "for free," as they went out of their way to be as helpful and giving to the "competitors" as they could. They provided everything, short of travel; they even rented a hotel room that people could take turns crashing and showering in, and when the hotel-catered meals weren't enough, they happily went and got food for us as needed. And of course, the best thing that Loki did was deciding to do this to begin with. That and coming up with a very flexible NDA which only restricts our usage of the physical code for the next five years, which of course, we weren't allowed to leave with anyway.

I can't properly describe how incredible this competition was to anyone who wasn't there. Particularly I know I can't describe it to people who are driven by money and recognition, rather than the love of simply coding. Again, there was no feeling of competition there. Nor did there need to be. As bad as Activision's code was, it wasn't bad enough to stop people from simply having fun and doing very cool things. It surprised everyone there how productive they could be under the circumstances. It certainly surprised me.

Hopefully this experiment will be a positive lesson to Activision and other companies that opening the source can lead to wonderful consequences.

In any case, I would definitely like to thank the wonderful people at Loki Entertainment for enabling this to happen, especially Sam, who tried his best to help me whenever I was stuck on a piece of code, and Scott, who stroked everyone's ego in nice ways. I would like to thank the people I met there, especially Chris and Ryan, for being so cool to work with and knowing what this contest was really all about. I would definitely like to thank Activision, who made this possible to begin with; I look forward to seeing the results of this experiment, on every scale.

People who don't know any better sometimes think magenta's real name is Joshua Shagam.

Comments

Second runner-up's view on Loki Hack...

[netfunk]

WARNING. THIS IS A LONG POST. BUT I WAS THERE, SO I KNOW WHAT I'M TALKING ABOUT. :)

(Man...I wish I had seen this a few hours ago! :) )

Uh, hi. I was runner-up #2. My name's Ryan. I wrote the "server mod", which isn't really a server mod, and I'll explain in a minute. First, I want to discuss Loki Hack and ALS.

I wasn't going to ALS at all, originally. It was another Linux conference that I couldn't schedule into my life (like all Linux conferences...sigh...) but I saw Loki's advertisement for the 'Hack on Slashdot, and figured "Hhm...no prayer of beating the Slashdot masses with MY application, but I'll enter it anyway." I actually screwed up my 'Hack application, referring to the contest as "48 contestants doing 30 hours of programming" ...my bad. :)

So, needless to say, I was somewhat shocked when I checked my email one Friday night to find that I was invited to participate.

I later found out that there was only ~31 entries...20 of which agreed to show up. So much for that Slashdot effect, eh? Realistically, it was VERY short notice to put a lot of demand on a contestant's time and money...I suspect next year this may be less of an issue.

Still, I quietly took a week off from work (I wonder if I still have a job? Oh well...) Transfered some flow to my check card, and hopped in my little blue Saturn to make the drive from Charlotte to Altanta...only 3.5 hours down I-85 South. Not bad at all.

One contestant was there before me, having "hitchhiked" from Salt Lake City. Smart guy...works for MyComputer.com. Met Scott Draeker and Sam Lantiga of Loki, got my case of CD's flipped through quickly by the security guard (which was the extent of the "police harrassment" any of us got...which was MUCH less than expected by me...but I'm paranoid.)

So I popped in my custom CD-R of important data: source to FTE, glibc reference in HTML, Limp Bizkit in MP3. :) Copies of every relevant webpage I could find; the original slashdot posting, Loki's homepage, SDL's documentation...

Others began to arrive. The guy next to me noted there was a nametag waiting for a contestant named "Eric Raymond." As the contest started, Mr. Draeker annouced that whomever was the contestant from IBM, he need not worry; IBM's lawyers said the NDA was okay for him to sign. After a (VERY) brief overview of the code ('the "gfx" directory is for graphics, "ui" is the user interface, etc...any questions?'), we all went around the room and introduced ourselves and what we wanted to do to the code. There was a contestant from Zendragon software, who was an old-school XFree86 hacker. There was a guy from Cygnus software. Someone else was from Creative Labs. San Mehan from VA was there. Eric Raymond was not, yet. One person wanted to port to Clanlib. One wanted to redesign the tile properties to include stealth, etc. for "more realistic" tank warfare (he was a programmer from Florida working on defense contracts or something). Others talked about AI, GGI cubes, and other more-than-impressive stuff. And they were all confident this would be accomplished in 48 hours.

Needless to say, my little punk ass was feeling more than slightly underqualified at this point.

"Uh, hi. I'm Ryan Gordon, I'm a programmer in Charlotte, NC, and I haven't the SLIGHTEST clue what I'm going to do with Civ, yet. But it'll be REALLY cool when it's done."

I gave a goofy smile and a thumbs up. Everyone (myself included) probably considered the possibility of mental retardation.

Admittedly, I'd played the Win32 version of Civ:CTP for the first time the previous night. Never the Linux version. I hoped they were similar. Remarkably, they appear to be identical, minus a splash screen from Loki at the start of the Linux port.

NDAs are passed out. They are a JOKE. We may not physically remove the source from the room, but we may discuss it at our leisure publically. Anything we retain IN OUR MEMORY can be used for personal, public, commercial, and non-commercial uses. I interpret this to mean that if my memory was good enough, I could leave the hack, type in 0.5 million lines of code, and legally own it. Maybe I'm wrong. Scott Draeker is a software licensing lawyer when not running Loki, and he called the document an "Anti-NDA." He had never seen the memory clause before, and suspects he never will again. No one knows why Activision allowed that one to get by. Somehow, I don't see myself as a threat to their business, though.

The hack begins. We tore through code like we tore through the cases of Pepsi. No one says much beyond the standard jokes about the quality of the code.

A word about that. The code to Civ:CTP is...well...I've seen worse. But it wasn't great. There wasn't any documentation (but comments in the code existed...I wish there was more, and an overview or whatnot would have helped a lot...) Specifically, internally, a good portion of Civ is scripted using the "slic" engine...no documentation exists for this engine at all, in source comments or printed form; indeed, Loki didn't have this either. Script syntax and usage could only be grokked from the source code itself. This principle applies to the entire codebase. Loki developers noted to me that it took them three days to get a dialog box to draw using the aui (activision? abstract? user interface) code from civ:ctp. I didn't have that much time, but being stubborn, spent about 30 of my hours trying to grok it. I failed on that one. :) This was my biggest tactical mistake, for those that really believed this to be a contest.

This brings us to a good point. It occurred to us all, subconsciously and early on, that the best help resource was not Loki's developers (who didn't know where to point us for the REALLY obscure details we needed for really strange hacks) but rather EACH OTHER.

Okay, group hug.

Seriously, we found that the obscure details we needed were usually found by someone else, or that they were willing TO STOP WHAT THEY WERE DOING, and help us search. Amazing. Anti-contest.

Eric S. Raymond -DID- show up, and gave up on adding a new government type...no one (especially Eric) has that much free time to spend on something of that magnitude. Later on, he added more cities to the game, but mostly had other things to do with his time. I can't say I blame the man...Linus Torvalds himself would have more free time than Eric at ALS.

So the contest continued. I found myself asleep under the food table at one point around the 18 hour mark...crawled out from under there and back to my seat blurry-eyed to continue working...no one thought this was unusual. :)

Around 24 hours, Cartman (the quad Xeon, 1 gig RAM build server) crashed. It took forever to get it back, and the solution seemed to be disabling NFS, as it was devouring memory for those multiple copies of 80-megabyte debug binaries and such...we continued then by telnetting into Cartman and doing our work there instead of exporting the devel directories to our individual boxes. That seemed to fix it.

As the contest reached the 10-hour-left point, I -FINALLY had a good way to do my hack. The original idea remained: I wanted to play Tetris (or pong...or whatever) in between network game turns. In a 32 player game, you have TIME TO KILL between turns. I thought I'd add this to the game in a Civ:CTP dialog box. You see the failing? Writing tetris from scratch in 48 hours seemed trivial in comparison. In reality, it probably was trivial in comparison.

Also, a friend had previously suggested to me that I add an "addiction meter" to the game. Something like a popup that says "You've been playing for 6 hours and 19 minutes...this rates you as: -JUNKIE-" ...

So I used an old Unix philosophy: why make one BIG program, when you can have a bunch of little ones that exec() each other?

So here's my hack: I added code to Civ that will spawn external programs at the beginning and end of the local player's turn. These are specified by the environment variables CTPBEGINTURN and CTPENDTURN. The values of those variables are the file to exec(). If the variables don't exist, then no fork()/exec() is done.

But wait, there's more.

At the start of each turn, Civ can now dump out a text file of game state information. This includes network game info (can I dynamically join this game?, etc) and player info ("Brian Boru", isAlive, etc...)...uh, and other little things, like if anyone's built a space city yet.

Nothing too insecure for cheating's sake.

This file is written to $HOME/.civctp/civchain.txt (or wherever you specify with $CIVCHAINFILENAME), before the process is spawned. The spawned code can be anything with the executable bit set; a shell or perl script, binaries, etc...

This leads to a primary benefit: extensibility BEYOND the hack. Users in the open source world can add new features, roughly, through a rough interface to the game. It sure as hell isn't ideal, but it's better, in my opinion, than a slightly improved, but hereafter untouchable, AI algorithm would have been.

Some uses for this?

1) The user can spawn xtetris/xbill/xcetera at the end of his turn, and have a shell script kill off the game when his turn comes around again.

2) The user can run a program that reads civchain.txt and pops up a dialog if he's been at it too long (the addiction rater I mentioned before.) The difference is now that this isn't limited to closed-source, and it's not limited to C++ and the AUI classes.

3) The user can parse the details of civchain.txt to a pretty HTML file, and dump that onto a webserver so people can see the current status of a game by hitting the website. Indeed, I wrote this program as proof-of-concept during hour #48.

There are deficiencies, and even bugs, especially in the civchain.txt creation, but it's usable. Email me for details. My address is at the bottom of this reply.

Thanks for reading this far if you just wanted the details of the "server hack." :)

Eventually the judges came around. I apologized to Hemos for not having any Nanotechnology in my hack. :)

Lots of cool was created. Eric got all the US capitals added to the game (and we all stopped to help...taught us all how piss-poor our georgraphy skills were...we couldn't even think of all fifty states for quite awhile...of course, we were mostly on our third day without good sleep.) Daniel, who later kicked my ass at BattleTech at Dave and Busters, ported (miraculously and without a prior test run before judgement) the program to clanlib. The dude from Creative Labs got Civ:CTP running on a side of the infamous GGI cube. Joshua got a bunch of cool things done. There was a lot more, but these were the ones I remember best.

As for the future use of these, obviously the Clanlib and GGI guys should be crapping their pants when they hear about the port, and the Aztecs are a great (re)addition to Civilization. I hope my addition is at least flexible enough for some creative soul to make use of...many things (like the addition of a porcupine race...sorry, Josh!) will probably NOT be folded back into the game, but some details (like the wandering players hack) could even make it as far as the next win32 release...who knows? I doubt Loki could even be ACCUSED of getting "free labor" out of this...but Activision and Loki definitely got "free ideas"...and that's cool. Open Source isn't about code so much as good suggestions from peer review...fresh perspectives...maybe Activision will see that Intellectual Property is expanded in a new way much more so than it is diminished in the common ways through open source.

But maybe not. That's okay too; I had fun. Call me a selfish detractor from the open source movement, but I don't care if Activision opens this game up. I also don't care that I "consorted" with the enemy for 48 hours, and gave them free peer review. It was fun. I met all sorts of cool people from the Loki guys to Phil Hughes to Illiad to Eric Raymond to Liz Coolbaugh to Taco and Hemos to Scoop to that dude from Cult of the Dead Cow to...countless others. I got offered some jobs (and Scott, I'm still sending you that resume...Charlotte isn't TOO far from California, right?) by some very varied sources.

And I learned a lot. It was a growing experience. Not only did I learn more about how (not) to manage a massively large project, but I saw what great minds do when they connect. I learned that you needn't be TOO ashamed when you've been up for 45 hours and forget how a C++ constructor works. (oops. :) )

So in the first and last Loki Hack of the century, I'd say the result was entirely positive. The only thing I would seriously change, if I had it all to do over again, would be how I attacked the code. The best plan seemed to be to plant some hooks and get out of the code base as quickly as possible.

As for next year, there's been some grumblings of plans for a Loki Hack '00, but nothing is definite. Let's just say if it goes anything like what we all discussed at the open bar after this year's event, it'll be a) VERY different, b) much better, and c) have wider-reaching effects and implications. I think you'll be impressed, whether you liked Loki Hack this year or not...I just don't want to promise things for Loki that aren't nearly definite yet, but you can get some hints from other replies to the original article.

In closing, big thanks again to Loki for the opportunity. I enjoyed every moment and got so much from it...and it only cost me gas money...everything after that, I spent of my own free, impulsive will. :)

Comments, questions, and everything else that isn't spam is welcome at ryan_gordon@hotmail.com.

--ryan c. gordon.
second runner-up, loki hack '99.

Interesting query..

[Kitsune Sushi]

But on the other I keep hearing echos of the gpl in my ear telling me that it's a moral imperative to not support proprietary software.. and a waste of one's effort to do so anyway.

While I'm not precisely a big fan of proprietary software (about the only non-GPL stuff I ever bother to use are PICO and PINE.. which I see no real benefit to GPLing in the first place =P), I've never been all that concerned about non-free game software. Most of the software that I prefer be free are general purpose or otherwise a means to an end (and most especially device drivers ). I don't really care about the newest Quake being free software the moment it comes out any more than I am drooling all over myself to see the PlayStation 2 get "open sourced" (as someone complained that it wasn't.. the argument against closed consoles was quickly follow by retorts of "is something worthless because you can't program it? just because your toaster isn't open source you're not going to use it?" or something like that..).

I guess my unanswered question is - should you have fun even if the people you associate with would disapprove of it? For me, the answer would normally be yes. But the moral dilemma found with the free software / open source philosophies leave me with a bad taste in my mouth.

When it's just a game (especially since most games have a short life expectancy), I really don't see a problem with it. I don't really think generating proprietary software in general is "evil" per se, but I certainly wouldn't waste much time on such things (aside from games.. ;). While it might be nice to see id release the code for their games a while after they come out, I don't see it as any sort of pressing issue that they be free software to begin with, or at all. Stuff you sort of have to rely on, like your operating system or other damn useful tools, I see a problem with helping along proprietary systems that simply don't work like they should (and yet get marketed as the best thing to happen to the world since.. um.. sex..? ..certain religious orders apt to disagree.. ;).

Ok, now that I've beaten my point to death and just about bored myself senseless.. Loki really isn't that bad of a company (haven't they been releasing libraries and other tools they used to create games as free software?), and since games are only for play, it's not quite the same as pushing along "the corporate machine". Now, if a bunch of hackers were invited to Redmond to get all the wrinkles out of Windows 2000 and were offered a neat new computer to the one who did the best work (and lots of notable individuals actually accepted!) then you'd have much more cause for that bad taste in your mouth.

Until then, focus on what's more important. ;)