EverQuest/Sony Fights Code Wars With Latest Expansion

Perlmonkey has written a summary on the latest Everquest Expansion, and Sony's efforts to thwart those who might wish to to tap into the packets and do things that maybe aren't exactly fair to other players. Or they just want a map that should have been in the first place. In anycase, hit the link below to read his piece on the subject.

The most recent expansion for EverQuest (Planes of Power) adds a lot of problem-solving quests to the game, so Sony beefed up the (long-since broken) encryption that they used for the client protocol. The expansion has been a major hit, pleasing some of the most critical voices in the EverQuest world, but one week later, the anonymous development team of ShowEQ had broken the new encryption. Read on for details of the ongoing battle over keeping secrets in plain sight.

First, the skinny on the latest EverQuest expansion, Planes of Power (PoP). Because this is an expansion chock-full of content for only the highest level characters in the game, Sony added some features that everyone would want (and thus, pay for): the ability to progress to level 65 (60 was the cap before); a new zone called the Plane of Knowledge which allows characters to moved freely to all of the old game areas and a feature that allows large groups to coordinate more easily. That's the carrot for the lower-end users, but really this is the first expansion to lock out even moderately experienced players in favor of large, strong in-game guilds.

Even so, the response has been almost all positive. Some players complain about the last-minute changes (especially the changes that made monks and druids less powerful in the high-end game), but those who are taking advantage of the new game areas are happy with the reduced time required for encounters and the fact that the game rewards strategy more than ever.

Planning, attention to detail and a fanatical focus on getting past every challenge that Sony presents are important in-game, but Sony is less than pleased by programmers who are just as happy to approach those challenges from outside of the game. Using Linux and Qt, ShowEQ is a packet sniffer that watches the EverQuest client protocol and displays a map of everything that the Windows client is privy to, but may not disclose to the player. Years ago, the ShowEQ developers discovered a weakness in the encryption that the client uses, and they have been able to reliably interpret the data ever since.

With the PoP release, Sony improved the encryption so that it used a larger key which was more securely chosen. At first, the talk on the ShowEQ IRC forum was gloomy and the normally secretive developers cloistered themselves off from the the group, returning only rarely to proclaim the difficulty of breaking this new scheme. The protocol is not unlike that used by ssh or SSL. A public key is sent from Sony to the client, and the client uses that key to encrypt a random session key and send it to Sony. Theoretically, this approach is open to only a limited number of attacks, all of which run the risk of being detected by the client.

A former ShowEQ developer who was hired by Sony was reported to have said it's over, "you'll never break this"... One week later, the new version of ShowEQ was available via CVS and was working again. The new keys were vulnerable, it seems, to an even simpler form of analysis and the result was simply that ShowEQ worked significantly faster. In many ways, this seemed to simply be a "bonus quest" that Sony threw into the PoP expansion, and it had been beaten.

On Thursday, October 31 ShowEQ broke once again. The protocol now compresses key data to prevent the analysis that was limiting the keyspace that has to be searched. As of this writing, ShowEQ no longer works passively, but this escalation is not over. The latest version allows a user to input the key directly, and developers are hard at work, trying to find further weaknesses in the key generation and/or exchange. The developers are even starting to question the long-held, unwritten truce that they maintained with Sony. The idea was that if Sony did not make decryption require a Windows-side component, there would never be a Windows version, limiting the use of ShowEQ to those capable of getting ShowEQ working under Linux. Now, the party line is, "there is absolutely, positively no reason not to have a WinSEQ."

The technical details are interesting, but the social and legal details may take center-stage for a while. The seq team is trying to figure out what they could put on the client-side without being detected and that brings into question the legality of Sony scanning running processes and reporting back. There's also the matter of Sony's rather astoundingly harsh EULA that tries to preclude activities like this in every way that it can (though the legality of click-through EULAs is still a hot topic).

One problem with this escalation is that, like another product (TiVo, which is partially backed by Sony) the very people subverting the product and making it more than the creator wants it to be are the best customers. In terms of EverQuest, they are often the ones maintaining several accounts and/or spending extra money for the "Legends" service. How does a company contend with a market where your best customers are also your most resourceful? With the TiVo, there was an uneasy understanding between the company and its modders. Sony has broken that balance with EverQuest.

Now that Sony has crossed this Rubicon, it is quite likely that ShowEQ will be ported to Windows and hundreds if not thousands of new users will be introduced to it. Was that Sony's goal? Certainly Prof. Felton showed us that such a battle is ultimately futile. Why does Sony want to fight it again on yet another front (remember that they are an RIAA member)? Is there any financial justification, here? Does mapping software really threaten the game more than the many in-game exploits that the high-end encounters suffer from?

PoP is a finely crafted fantasy gaming experience, but Sony has once again chosen to spend extra time and money hurting themselves and their market. Perhaps their competition will not make the same mistakes.

Comments

Ha ha ha.

[Perianwyr Stormcrow]

Raph Koster's rule of "the client is in the hands of the enemy" seems to have been forgotten by EQ's developers- if ShowEQ is such a problem, it's time, perhaps, that they stopped telling the client all these nasty things they didn't want it to know. I mean, I first remember ShowEQ coming 'round *3 years ago*. Why they haven't simply made the client ignorant of things it shouldn't know in all this time is beyond me.

Re:Ha ha ha.

[Dillon2112]

The client has to know certain things to run. To alter what it knows would to make a thin client game even thinner, and would alter the balance of the client/server load. If the servers are picking up slack for the clients (or the sole purpose of players not sniffing that info), then the servers have to be redesigned and beefed up. Not likely, I say.

Re:Ha ha ha.

[Saxerman]

the servers have to be redesigned and beefed up. Not likely, I say.

For games that have already been around for awhile, you're right that they're not going to be making large changes to the server and client to hide those things they shouldn't have been sending to the client anyways.

I have great hope that the next generation of Massively Multi-player games will incorporate the idea of 'the client can not be trusted' into the game design early on. I agree that it requires a lot more work server side to process things it would be much easier to have the client work on. But the only real way to prevent people from accessing things you don't want them to see, is to NOT SEND IT.

I also think games like Everquest should keep the server closed, but open the client so third party groups can create them and add features. There only real way to allow this is to only send information to the client that you want them to have.

Re:Ha ha ha.

[cwebster]

(not anon, and i am a seq dev)

>SOE (or Verant previously) *has* removed lots of data from the client-side. For example, a long time ago, the client used to be aware of every "mob's" (monster's) loot.

ShowEQ has never been able to tell you a monsters loot. Ever.

You could deduce what they might drop from things hey have on them (the old favorite was telling which wisps had lightstones, not because we could tell thier loot, but because you could tell what level of light they gave off).

ShowEQ has never known loot.

Well written!

[Buran]

A very well written and informative article. I'm intrigued by the story of ShowEQ -- and the fact that Sony seems to want to protect their system by deciding what users can and can't run on their computers. It reminds me of the video player that uninstalled Ad-aware automatically, claiming that running it wasn't allowed under its license. I don't believe that such things are really legal -- are they not an invasion of privacy and an illegal search? (Alas, the Constitution's protections against this in the US are only applicable to the government itself, but such practices, I feel, are going too far) Sooner or later, I think, Sony will be dragged into court over this and sued. What if, for example, they use their little spy applet to suck data off someone's computer that gives them an advantage (a competitor's computer that had confidential information, for example?).

I am, though, a little surprised to see that the ShowEQ people haven't been sued under the DMCA. And I'm glad they haven't, because what they are doing is reverse engineering and they are not actually hacking into the client or the server, so it is legal. They are not trying to pirate the game. They are merely trying to see what exactly their computer is saying to the server and what it is receiving from it. A perfectly fair thing to do, I think, because it IS their own property and the owner of it has the right to know what exactly is happening, so they can choose whether or not to run a given application instead of having to trust the programmers. Checks and balances.

Note: I don't play Everquest. I call it EverCrack, actually. I don't play games that I have to pay a monthly fee for. I prefer to play single-player games, or sometimes multiplayer, offnetwork, with people I already know.

Re:Well written!

[UberOogie]

A perfectly fair thing to do, I think, because it IS their own property and the owner of it has the right to know what exactly is happening, so they can choose whether or not to run a given application instead of having to trust the programmers. Checks and balances.

Surely you kid, right? ShowEQ isn't used as a security program. It is a cheating tool. Even in its most beneficent uses, it is for cheating. Period. This has got to be the most sad apologies for cheating I've ever seen.

Sony made a game. Someone made a cheat program that unbalances the playing field. Sony has every right to try and disable this cheating program. However, their rights end where ours begin. But if they want to change the encryption in their program or make a client that monitors game traffic or the use of a specific cheat program manditory for using the game, guess what? That's their right. You don't have to play the game.

And your analogy with ad programs that uninstall Ad-Aware is both faulty and inflamatory. Those programs are unistalling a security program from your computer so that their spyware will work. Sony is just not letting you play their game if you have a known cheat tool running on your computer. Huge difference.

Clickthrough License

[man_ls]

In EverQuest, the license agreement must be accepted each time you start up the game.

No auto-yes option. No "I understand and agree, don't tell me again" option. You [b]must[/b] click "I Accept" every time the game EXE is called.

If you've accumulated any significant level or /played time, you've seen that license hundreds of times. Whether or not you've read it is illrelevant, but you've willingly clicked yes to it hundreds, if not thousands of times.

I'd say that makes it pretty binding.

Sony just needs to fix what they send...

[Jerrith]

The big advantage people get from decoding the zone information is the name, and to some extent, the position of monsters all over the zone. If Sony changed EQ to only send data about monsters that are near you (within a reasonable distance) there would no longer be a big reason to decode the zone data. 99% of what you'd be seeing you could also see by turning in place, and clicking on each monster.

Due to how the servers are setup however, it seems to be more efficient for them to send out all the monsters, rather than do the range calculations and just send the nearby ones.

My prediction, if a Windows version is released and becomes widespread (and I consider the latter likely if the former occurs), is that Sony will, finally, bite the bullet and change the code. It's not quite as straight forward as I may have made it sound, as there are some other systems (such as tracking) that will have to be significantly rewritten as well. However, if they really want to stop people getting at this data, really the only way to do it is to stop sending the parts that aren't needed. :)

Re:Sympathic view of cheating?

[UberOogie]

Exactly.

I have never understood online gamers whining about their right to cheat. This article is talking about this as though it is some brave stand against a corporation doing something bad. This is about people cheating at a game. It ruins it for everyone playing fair. I fail to see why this should be applauded or supported, and I'm fairly shocked Michael wasn't the one posting this story.

Sadly, this is why consoles are going to take over for a while: The majority of players are simply sick with the cheaters. It was amazing to watch people immediately start to whine when MS disabled modded X-Boxes from Live. Sure you can say there were "legitimate" reasons to mod the boxes, but come on.

Of course, this only lasts until some "worthy" individual hooks up a box between the cable and the X-Box to start parsing out material.

More information on the story...

[Valen0]

According to the Sony developer that everyone talked to, the changes that Sony makes to the encryption only takes 20 minutes or so. I believe that part of that time included remaking about 5 different binaries with the new code. Sony just has to change the 5 or so #define's on the encryption and everything breaks. Also, according to the same developer, Sony will not spend much time on breaking ShowEQ until management decides otherwise.

How ShowEQ is fighting back is very interesting. Encryption information is stored at a preset offset in the client. About a month ago, SOE changed it so that, on NT/2000/XP boxes (this didn't effect Win 9x), other programs (even on accounts with "Administrator Access") couldn't read that memory space. However, ShowEQ developers eventually got around the limitation by making the key reader run as a service on the NT LocalSystem account. This service can then send the key information to the decoding system.

As for WinShowEQ, I have sources that have told me that WinShowEQ is an easy port to make. If they are serious about making it and releasing it, expect to see it sometime soon.

Re:Sympathic view of cheating?

[Buzz_Litebeer]

One of the weird things is, maybe they didnt "want" you to know how much damage things did, and not from a balance standpoint either. I have created modifications for the game medal of honor, allied assault spearhead. and I do not want people to know all the exact values i have set for my weapons. to those who are curious and know what they are doing it is a relatively simple matter, but i feel it takes away some of my intent for everyone to know that the smg does 45 damage or such. imbalance is not a practically "bad" thing, even in a mmorpg. No you wouldnt know your singing steel boots or whatever didnt work well until you wore them for a long time, and realized that they arent purhaps as strong as your lambent whatevers. But that would be part of the interesting part of it. When you get into finding out direct stats you get to having players that are, sadly, like me. who will statisically approach the game, and quite possibly ruining the intent of the creators of the game and how the game is played. I am not decrying the fact that many things have been fixed because of SEQ, not at all, I am just saying that if it had not been around the game would be entirely different, the only way you would know things is from relative tested heresay. which might even be more interesting.

Man in the Middle attack?

[kakos]

As I understand it, ShowEQ is usually run on a different computer on the same network as the EQ box. It sniffs packets and then does its thing accordingly. Why not use that ShowEQ box to block the initial key exchange and generate and send its own key. The EQ server will then encrypt the key and send it to the ShowEQ. ShowEQ can then just decrypt it. It would only be a matter of figuring out the protocol and type of encryption that the servers use.

The Man in the Middle attack is one of the oldest and still most effective attack on public key encryption.

I don't know all the details, unfortunately. They may very well have tried this already.

Cheaters = Wankers

[Czernobog]

Flame me all you want. Mod me down.
But anyone who doesn't condemn the actions of this group is no better than them.
This game belongs to Sony. They make the rules, so either play the game as its creators intended the game to be played or don't play it at all.
If you think that this app is a valuable addition to the game, convince Sony to accept it and help those wankers develop it. If they say no, then just go away. It's their game.
Otherwise, you're no better than the people who exploit the in-game weaknesses. A cheater.

Re:Sympathic view of cheating?

[UberOogie]

To me, the right to cheat exists (although it is socially despicable to do so outside of single player games, unless every player agrees beforehand to cheat) and opposition to this right generally takes the same form as the opposition of my right to media shift (ie rip CDs to portable MP3 player).

Yes, the right to cheat exists. Yes, the right to make and run programs that lets you cheat at games exists. All Sony is doing is preventing people who are running known cheating tools from playing on their world servers. They are not disabling the software. They are not modifying your computer in any way.

Your mp3 analogy is thought-provoking, but I think not valid. They are not taking the same forms. No one at Sony is legally going after users using the cheat tools. No cops are breaking down doors. No nastygrams are being sent to schools where the cheat tools are being run. The users right to make and use cheat tools are not being infringed. What Sony is doing is preventing people using cheating tools from playing on their servers. Seems well within their rights to me.

Your solution of playing only with friends makes sense in other online games, but not MMP games. You can't just play with your friends because there are only official servers to play on. But as a moral question, is it really okay for a minority of cheating players to ruin the entire online play experience for everyone else who doesn't want to cheat, and paid exactly the same amoun t of money for the game? That doesn't seem to add up to me.

That's the proper solution

[Perianwyr Stormcrow]

Years ago, Ultima Online had a pretty egregious cheating application called UOExtreme. It let you do all kinds of special stuff- run faster than normal, see hidden people, get an automatic readout of damage you did to players, and have general interface improvements that allowed you to play the game more efficiently.

Well, people got banned for UOE use for quite a while, but the thing that killed it was that the UO dev team simply emasculated it and made it no more than a device for the delivery of trojans.

How did they do this? Clever engineering and greater awareness of the needs of their playerbase.

Fastwalk was fixed by making walk packets require a response from the server before moving the player.

See hidden was fixed by just not telling non-GM clients where hidden players were, and disallowing attacks and other operations on hidden characters. Invisibility was handled in a pretty slipshod way beforehand- the server just told the client, "hey don't show this guy."

The automatic damage readout was just integrated into the client, with the addition of Starcraft-style health bars showing the damage level of your current target.

The interface improvement issue was solved by the legalization of a similar program called UOAssist. Many operations in UO rely on an extremely clumsy interface requiring many mouse clicks and movements for actions that should be far simpler. UOAssist changes this, offering somewhat of an "expert interface" for the game. UOAssist's author sends all program changes to UO's developers to be examined before release.

Come to think of it, you probably know all this stuff already...

If you can't beat 'em, join 'em.

[LostCluster]

The comparison to TiVo is an interesting one. TiVo's policy is that they will gladly look the other way for some hacks, and in fact even make some of the hacks as easy as possible to pull off, in exchange for being able to declare certain hacks off-limits.

Specifically, they make it easy to upgrade a TiVo with a large hard disk by designing their single-disk designs to have a place where the second disk can fit nicely in the box. They also it possible for users who want to accomplish their "daily call" over the Internet rather than a phone line by just happening to leave the server that handles those sessions at an Internet-accessable location rather than requiring that the only way into their network is through their chosen dial-up providers. The company sponsors (but does not actually run) message boards at TivoCommunity.com where hacking discussion is encuraged, and people can compare notes and share experiences.

The tradeoff is that there are certain hacks that the company does not want to see made, and will not allow the TivoCommunity.com boards to discuss. There are the hacks that would either harm the company, like any hack that would provide another source of listings, which would eliminate the need to subscribe to TiVo's listing services, or any hack that would allow content to be extracted from the device which would surely bring down the wrath of the MPAA and friends.

By allowing wide open back doors into their system, TiVo has been able to direct hacking efforts into the areas the company wants to see them go. Yes, there are a few people trying to drill through the concrete and get the "forbidden hacks" to work, but their numbers are few and they operate in obscurity compared to the company-sponsored forums.

It's a total 180 from Everquest's "Thou shalt not hack us!" perspective. TiVo's offering carrots, Everquest is using rather ineffective sticks.

Not Quite True

[The Raven]

There are many things that the client MUST know, for performance and usability reasons.

Every request the client makes for information means another round trip access to the server. Another complex decision by the server whether the client should be allowed that information. When you have 100K users online, making requests like that dozens of times a second, it can get JUST A TINY BIT LAGGY.

To alleviate these issues, the server sends some information to every client. The game map. The nearby creatures that the client may need to render at any moment. The current statistics of all the players equipment. All this information the client NEEDS to know.

Here is an example of one thing people often think the client does NOT need to know... creatures that are behind a hill, or not in the players cone of view. However, what if the player whips around to look behind them? How disorienting and unplayable would it be if every time you turned it took half a second before you saw ANYTHING other than terrain? As for obstructed creatures, would you want to go around a corner in a dungeon and not see anything for half a second until the server caught up? Not to mention the exhaustively difficult math required to accurately determine whether you have line of sight to something or not.

Thin clients DON'T WORK. At least, not in MMORPG's. Works for MUDs though. If you don't mind, I'll be going back to my DikuMUD now. :-)

Re:Not Quite True

[Znork]

The game map is not sent. The game map is the zone files which are already local to the client (and which can be looked over using several means). The nearby creatures _and_ every other creature in the zone were sent last time I checked. To filter out which mobs position info to send to the client does not require communications with the client. It's trivial to fix that.

Creatures behind a hill or not in the players cone of view? Umm... maybe press the camera button to switch views could help you? This is already visible in the game.

LOS code is already in the game. As in 'you cannot see your target' - darn.

If a mob is around the corner you start sending info on that mob to the client when it gets into such a range that it is concievable that the player will see it in a second or five. Not as soon as they zone into the same area.

Thin clients ARE THE ONLY THING THAT WORKS or YOU WILL HAVE CHEATERS. Period. The client tells the player anything you send it. Period. The client tells your server any bullshit the player wants it to. Period.

The client is the enemys foothold in your fortress, not your foothold in the enemys fortress.

A moderately competent programming team and a reasonably smart architecture designer can code around the disadvantages of thin clients with things like path prediction and server-side components of player response time. But nobody, can _ever_ stop the players from reading anything you send them or sending you whatever they want.

Re:I guess

[Nkwe]

I wonder what would happen if Sony just shut off Everquest, the hardcore players would once again have to have a life.

Probably the same thing that would happen if the someone just shut off slashdot.

It's not as though it's a new problem

[Rogerborg]

Netrek figured this out about fifteen years ago. The source is open, so it was assumed from day 1 that clients couldn't be trusted. Attempts at client authentication were added later, but those were add ons (and could be and were subverted), they weren't the prime means of preventing cheating.

The strength of the Netrek model is that the game was designed from its infancy to send exactly and only the information that each client needs to display what it's supposed to be displaying. For example, cloaked units are supposed to be shown as unidentified contacts and on the galactic window only, with erratic position and irregular updates. One of the first things a hacked client developer will do is to display them on the tactical window as well, and there's nothing that the design can do to stop that. Also, it's not perfect; an ID is sent for the cloaked units, so the client can show what they really are. However, the server does only send irregular updates, and it flat out lies about the position, heading and speed of the unit, so the client can only show so much.

One of the most controversial design decisions involved torpedo weapons. The servers sends "start" and "end" packets, but instead of sending speed and heading and letting the client handle movement of the weapon, it sends regular "position" updates, with a jitter built in. This increases the bandwidth requirement significantly, but it means that the client doesn't know the exact speed and heading of the weapon, so can't make an easy calculation about how to dodge it.

The Netrek model is replete with decisions like this. There are a few snafus (like the cloaked ship ID), but in general there is very little that a client can display that it's not supposed to. And believe me, I tried.

The reason for this tight design is simple if you think about it. Netrek, like XPilot and Xfire, was originally an X-display game. The server handled both mechanics and display. When Netrek moved to a TCP(later UDP)/IP based model, that model was preserved and the server took on a lot of responsibility for culling information that each client shouldn't know.

It never fails to amaze me that commercial games developers never seem to learn the lessons that open source projects can teach them. I know (from bitter experience) that there's a huge rush to get results on screen, but hey, guys, do it right, don't do it twice.

Re:Sympathic view of cheating?

[Hrothgar The Great]

So what? Sony, so far hasn't gone after people for hacking or modifying software. They've simply made your mods not function correctly.

Let me ask you this: Do you believe that you have the right to make any modifications you want to software you own, but that Sony does NOT have the right to make modifications to software THEY own? Are you advocating rights for everyone, or only for you?