Continuing Security Concerns at DoubleClick

In 1999, DoubleClick bought the Abacus database, which got them a ton of data about our personal buying habits. They've promised not to correlate it with their banner-ad database, but that's not the concern this week. This week, the concern is their network security. Last week Thursday, the French site Kitetoa discovered three separate security issues on DoubleClick's network; the company deleted the evidence of one immediately, but left the servers up until Monday, when they mostly closed the other two. There are numerous other issues but the question on everyone's mind should be, how long and how far has DoubleClick been penetrated? And how long can we expect it to continue?

As I write, I'm aware of two security holes in their network, which is an improvement over last night, when the number was three. Unfortunately, this does not mean they are now 33% more secure. I don't have the background to be sure exactly how significant the remaining problems are, but I'll share what I know.

Unfortunately, DoubleClick's Chief Privacy Officer was not available by press time to respond to questions. We have offered the company a chance to respond, and we hope they'll take that opportunity to clear up some questions. Meanwhile, here's their official statement on the matter as of today:

"Over the last week there have been unsuccessful attempts made to hack into DoubleClick's servers. Those situations were immediately corrected," said Jules Polonetsky, Chief Privacy Officer, DoubleClick. "DoubleClick is now undergoing a comprehensive security audit, including the expertise of external security professionals and engineers, to fully ensure the continued integrity of our servers."

Now here's the history of DoubleClick security since last week, as far as I can tell.

Kitetoa ("KITE-a-toe-a") is a group of white-hat hackers who publish together under that pseudonym. They broke the news on their website simultaneously with transfert.net, and spoke with someone at DoubleClick to make sure the company knew to patch the holes. (I left several messages to this same contact, but he did not return my calls.) They continue to update their site with more DoubleClick security news.

The first IIS vulnerability is the commonly-known unicode bug, which lets you read and write files with the same permissions as the webserver, typically "IUSR."

Using this vulnerability, Kitetoa discovered the second security issue, which is that someone else had compromised the DoubleClick corporate webserver at some time in the past. The file eeyehack.exe was left on www.doubleclick.net. This is a backdoor written by the white-hat hackers at eEye, which opens port 6969 for attackers to telnet in.

DoubleClick assures us that eeyehack.exe could never have been executed, because that directory had script access disabled.

But I spoke with Marc Maiffret, Chief Hacking Officer of eEye (the people who brought you this port of nmap to Windows NT, by the way). He points out that the same backdoor could have been copied elsewhere, too, possibly into directories that allowed execution. I've asked DoubleClick whether they checked this; no answer at press time.

It's a separate question whether a cracker could have gotten SYSTEM level access through some other hole. With just IUSR level access, probably not much could have been done. There's no evidence that higher-level access was obtained ... but absence of evidence is not evidence of absence.

What concerns many people is that the eeyehack.exe file that was visible had a modification date of 1999. We know this date is not accurate, because the exploit that writes that file did not exist until last November. But that odd date does raise questions about how long DoubleClick's network has had these vulnerabilities.

The nightmare hypothetical is that a cracker has had access to DoubleClick's networks for the last couple of months or years, and has been reading the data they have been collecting about banner ad clicks. Or, worse, has had access to the Abacus database. Let me emphasize that I know of zero evidence that this has actually happened. But the potential for enormous privacy violations, with this company more than almost any other, is very serious.

DoubleClick assures me there is a good reason for the 1999 date on the backdoor program, but my question about it goes unanswered at press time.

The third hole is almost exactly one year old, and it allows ASP source code to be read. This alarms people because the server named AbacusOnline.DoubleClick.net was shown to be vulnerable. I verified this myself and learned the SQL passwords that go with two usernames, "gcolon" and "aowebuser."

Asked to estimate whether a determined cracker could have made use of those SQL passwords, Kitetoa guessed it was "85% certain" - for whatever that's worth.

Not that SQL should have been stored in the ASP code anyway. Marc Maiffret comments, "One of the better ways to secure SQL login information, within an ASP file, would be to store all of the login information and functionality within a COM object. This is not a silver bullet solution but it does provide you with much better security than storing things in plain text in a .asp file."

Was that database accessible from outside DoubleClick's network? Not that I could tell, but I didn't try very hard. What data was in that database? Considering the wealth of data that Abacus has collected on our purchasing habits, we might justifiably be concerned about a machine named "AbacusDirect."

Again, DoubleClick denies that this is a problem, saying it was a programmer's machine which "was not connected to consumer or production data in any way." We have to trust them on that.

That's what was known as of Monday, and you may have seen that in the MSNBC story, InternetNews story, or the ZDNet story.

But the problems continue. Kitetoa has continued scanning DoubleClick's networks, and continues to turn up vulnerable servers. A half-dozen servers of unknown significance are still Unicode-able. And - let me check - yes, I can still read ASP source code on travel.doubleclick.net.

As well as www.doubleclick.net. The company said Monday that the holes had been fixed on that, their main corporate website. But as of five minutes ago, I was still able to read ASP source code on that server using the year-old exploit. (I did not see any more SQL passwords, for whatever that's worth.)

DoubleClick's Chief Privacy Officer Jules Polonetsky claimed on Monday that "Even a partial breach of a noncritical server is unusual," a claim which looks more dubious every day.

Maybe none of these servers has any valuable data on them. But to a serious cracker, breaking into them could easily have provided the necessary opportunity to reach further into DoubleClick's systems. Being inside the firewall, setting up trojans, sniffing network traffic from other machines - these are all reasons why unauthorized access of any machine must be taken very seriously.

And there's more. A security writer from transfert.net - a sort of French "Wired" - was able to snoop around one of DoubleClick's internal mail systems, webmail.doubleclick.net. It's fixed today, but he sent me a partial list of employees whose mail files were in a directory. (He could not read the mail itself, and DoubleClick denies that mail could have been, at any point, read.)

Here's the big picture. Where long-term security is concerned, process is always more important than the individual problems. We can learn more about DoubleClick's security by observing their response to security reports.

I would have hoped to see servers taken down. As Kitetoa remarked to me, "I think they should have closed all the servers for an hour or two while they fixed the problem." Or, if necessary, longer. Tough decision, but better than possibly exposing data.

I would have hoped that DoubleClick would not announce to the media that everything was fine until they actually knew that this was true. At this point, we have to trust them when they say that attackers could not have seriously compromised their data.

And I would have hoped that they would start tackling problems sooner than they did. Vulnerabilities about which they were informed last Thursday were not fixed until Monday. (The vulnerabilities themselves, of course, are not new, and some of them are a year old.)

And more vulnerabilities continue to crop up. Paranos (based in Paris) found one just by using search engines. The funny thing they found was a database of DoubleClick employees in the U.K. who attended last year's Christmas party.

Less funny is the list of people Paranos found who apparently filled out a form in conjunction with FoodTV, the "Outfit Your Kitchen Sweepstakes."

It took about two hours after I notified their PR contact of the URL before it was removed. It was stored in what appears to be a directory of backup data which was never intended to be public: http://www2.doubleclick.net/live2/chefs/ foodtv-bak/form/chefs.txt.

This isn't a vulnerability, but it is indicative of the company's security process. DoubleClick should have policies in place to prohibit posting backup data on public websites; it should enforce those policies; and when it was done anyway, it should have found the leak with by internal audit.

This promotion ran in 1999. Remember, kids, data you give to corporations never goes away, and may pop back up at any time!

I picked one of the phone numbers for my state and gave it a call. Kathy Bankes, the wife of one of the people on the list, answered the phone. She said she was "very leery with the computer," and then proceeded to give me a common-sense understanding of the state of security on the internet.

"They say things are supposed to be secure," she said, "but I don't care how secure anything is - if somebody knows how to get in, they're going to get in if they have the technology."

"Should we worry about it or not?" she asks. I would say yes, especially when the company that owns an enormous customer purchasing database has problem after problem with its security. Maybe I'm naive to think that privacy promises mean anything in the real world, or that crackers can be fended off by a big corporation.

When I'd expressed my surprise at all this to Kitetoa, he'd just chuckled and said, "You'd be surprised what you can find on the internet. It's all like this."

Kathy, too, was pretty sure that there were people who have all this private information anyway. She very sensibly pointed out that credit card and other personal data can be stolen in the real world just as easily as the internet.

And she answered her own question for me: "There's really nothing you can do. I don't feel secure anyways."

Comments

Way of the Future

[zpengo]

Detailed consumer profiling is going to be a fundamental part of our future, and we ought to be developing ways to get around it, limit it, etc., instead of yammering out the usual boilerplate about how DoubleClick is the devil because they know if we like golf or not.

With the increasing sophisitication of profiling technology (and the databases that drive it), as well as the sketchiness of existing laws on the subject, it won't be long before every major company has a detailed consumer database. We complain and complain about privacy issues, but they don't know anything that we haven't made known to them. If they send us sailing magazines, it's because we clicked something somewhere to indicate that we were interested in it. We know what happens when we click those things. Everyone knows. My grandma knows.

You are being watched. Act accordingly.

Re:Way of the Future

[QuantumG]

I think it's hilarious that a company that lives off of software designed to help people pirate music is acting as if they are morally superior to the "evil" RIAA (a legal company trying to protect its own interests).

The RIAA isnt a company it's a trade group and the only reason copyright holders have their "interests" is because we the people choose to sacrifice our right to copy their work because we believe it encourages more artistic works (via economic incentive). If we the people decide we no longer want to honour this agreement then it will take a massive political uprising to sweep them away, specifically because of trade groups like the RIAA. That is what the "silly napster logos" are about.

security through obscurity

[perdida]

yaaagh. Bugtraq != slashdot not even en francais.

The notion that security through obscurity doesn't work only holds up to a point. If you focus enough distributed processing power on any security problem, like, say, through posting it to slashdot, you will overwhelm and outpace the sincere efforts to patch the problem that are being undertaken by the hapless victims.

Of course you can also claim that helpful slashdotters may lend advice to DoubleClick but er.. that is not going to happen. Slashdotters might help some open source site that was being lax and got exploited, but not DoubleClick who has copmmitted so many prior offenses against the mores of the /. community.

The unprotected consumers lose out, again.

Re:security through obscurity

[Erasmus Darwin]

through posting it to slashdot, you will overwhelm and outpace the sincere efforts to patch the problem that are being undertaken by the hapless victims.

However, to quote DoubleClick's Chief Privacy Officer (as listed in the story above), "Over the last week there have been unsuccessful attempts made to hack into DoubleClick's servers. Those situations were immediately corrected,". So everything's fine and good.

Besides, one could make the argument that leaving a known insecure system on the Net is at least mildly irresponsible. Leaving a known insecure system on the Net that contains all kinds of personal information about a lot of "customers" (which may or may not be the case; weeding through PR garbage is useless) is downright moronic and deserves to get them as much negative attention as necessary to convince them to correct the problem.

Re:security through obscurity

[jamiemccarthy]

"If you focus enough distributed processing power on any security problem, like, say, through posting it to slashdot, you will overwhelm and outpace the sincere efforts to patch the problem that are being undertaken by the hapless victims."

The company has now known about the problem for 5.5 days. I had debated how long to give the company to fix stuff before posting this, but since it was already picked up by MSNBC and other media two days ago, I don't really feel it's an issue anymore.

Jamie McCarthy

Re:security through obscurity

[atrowe]

jamie, your "They did it first, so it's okay" argument doesn't justify Slashdot's actions.

Other media ran the story because they have deemed that readers would be interested and would read the story, therefore increasing banner revenue. Their primary motivation in promoting the story was purely monetary.

I would like to think that Slashdot should be held in higher regards than mainstream commercial news services. Slashdot, by focusing on "geek" news and stories that focus on a more technical aspect, should hold a bit more journalistic integrity than others, and it's authors should hold some restraint in posting stories that could possible cause harm or privacy invasions to it's "geek" readership.

In short, I understand your motivations, but do not agree with them.

Re:security through obscurity

[HiThere]

On the other hand, maybe they'll be slashdotted, and achieve security that way. :-)

Sorry, but DoubleClick ... anyone who maintains that database doesn't deserve much in the way of "slack". The idea of maintaining a database that you promise not to use is just a bit unbelieveable. Now I will grant you that I am oversimplifying, but management changes all the time, company policies change, and they expect me to believe that they won't use this database for their maximum return? If they really felt that way about it they'd run the data through a one-way summarizing filter. (Of course, perhaps they did, and I just didn't believe the report, so I didn't remember it. That also happens.)


Caution: Now approaching the (technological) singularity.

Hacking DoubleClick

[zpengo]

I'm not entirely sure what's so terrible about having a band of scruffy Russian hackers knowing that my last ping-pong related purchase was over six months ago, and that my operating system is "Other."

Poof!

[adrox]

You know there's a pretty easy way to block doubleclick. In windows edit your c:\windows\hosts file to include:

127.0.0.1 doubleclick.net

There's also the hosts file in linux that can do the same thing. No more worries about doubleclick!

Re:Poof!

[BlowCat]

How is this "insightful"? It doesn't remove the data that they had collected about you before you edited "hosts". It doesn't protect you from the data that Doubleclick has bought. It doesn't protect people who cannot use editors because of disabilities, menthal or physical. It doesn't protect users without root access.

Re:Doubleclick may read /.

[jamiemccarthy]

The page was there for two hours after I emailed them about it, but was removed and redirected shortly before this story went up, so I added the link to where it used to be. You can see the "bak" in the URL, which is why I assume it was backup data never intended to stay on the corporate web server.

Jamie McCarthy

When are we going to get it right.

[mr_gerbik]

My lisp instructor recently gave a keynote speech at Los Alamos entitled When are we going to get it right. The speech does a great job covering many of the security issues regarding double click and just the overall state of security on the web. Its a good watch if you have the time. You can see the whole thing in realvideo from the link above.

-gerbik

For the love of god, opt out

[CaptainSuperBoy]

If you value not being tracked, you really should opt out of Doubleclick's tracking. For those not familiar, they use cookies attached to every ad in order to track which of their member sites you view. If this bothers you just a little, go to

http://www.doubleclick.net:80/us/corporate/privacy /opt-out.asp?asp_object_1=&

--

Re:For the love of god, opt out

[Tackhead]

>If you value not being tracked, you really should opt out of Doubleclick's tracking [using their cookie].

If you value not being tracked, you probably don't trust Doubleclick. Why, then, would you use an opt-out method that requires that you trust Doubleclick's word that they'll no longer track you?

Blocking Doubleclick and the other tracking firms at the router, on the firewall, in /etc/hosts or HOSTS, or with a proxy, are just as effective as a means of "opting out", and they don't require you to trust your adversary.

Regulations are necessary

[gazorg]

I think it is necessary to regulate the harvesting and use of data related to persons or equipment persons own. It isn't fair or reasonable that any company can collect personal information (adresses, shopping habits, general whereabouts etc.) and benefit from it.

This is due to privacy concerns.

First: information is dangerous.

Second: information gives power.

Third: noone want the ad buisiness to get power over all our lives.

Forth: The ad industry has a bad track record for computer security.

Example: Think what enormous amount of information is collected in various databases for one Swedish individual. Most people shop a lot using cards of different kinds. Almost everyone uses cellular phones (GSM, that is). This means that for many persons you can follow maybe 90% of the total spending and using the cellular network you can monitor the position at any time to within a hundred feet (next generation: five to ten feet).

I'm a europeean so my views are somewhat tilted in that direction. Some dislike some of the EU's newer regulations concernign personal information (the associated person must give his or her consent for the data to be legal and there are regulations for what information that may be collected by corporations (alas, states may do as they like)).

...and the big deal is?

[Mike1024]

Hey,

Is it just me, or is this just another company getting hacked? So it happened to be an advertising company. Big deal. This hardly seems slashdot-worthy; web servers are compromised all the time. Most of DoubleClick's data is just IPs and lists of websites.

It isn't automatically a big conspiracy, spying on you. Do you really think that, if hackers compromised doubleclick's servers they'd be looking for your information? Well, let me tell you this: They won't. To think that they are is paranoia taken to it's extremes.

So a website has a security bug or two.Why not just inform the site owners, and give them a chance to fix it, instead of proclaiming it loud and clear to the world? It seems helpful to no-one.

Just my $0.02

Michael

...another comment from Michael Tandy.

Re:...and the big deal is?

[Rupert]

Did you miss the bit where the site owners were informed five and a half days ago and have chosen not to fix it? That MSNBC reported this 2 days ago?


--

Re:...and the big deal is?

[MadAhab]

Hey, is it just me or is it really tired and trite to accuse all who value privacy of being paranoid?

Here's something fun you can do with someone's doubleclick profile; use it to assist trashing someone's credit. It's no secret that credit card fraud detection works largely by identifying purchases in unusual places or types. So if I'm running a credit card ring, and I know what type of purchases you make, I can probably multiply by a factor of 5 or more the amount I can extract from your stolen credit card before any fraud detection kicks in.

Not to mention corporate espionage; it's assumed that DoubleClick doesn't sell certain kinds of information in their database to all their customers. I wonder how much Bezos would pay to find out what B&N customers are up to? Bet he can't find out legally. Sure would be really tempting if some mysterious party offered a stolen report, wouldn't it?

There are lots of nasty things that can come out of this kind of hack, and not all of it is about finding your preferred vendor for butt plugs. That being said, it may still be easier to get this information by bribing DoubleClick employees than via hackery.

Oh, and RTFArticle; they did give doubleclick lead time before breaking the story.

Boss of nothin. Big deal.
Son, go get daddy's hard plastic eyes.

Re:Doubleclick may read /.

[timefactor]

I'd visit the link but Junkbuster won't let me!

Timebombs

[QuantumG]

There's one thing to say "all this information is available if you've got 5 hours to go searching for it" and it is a completely different thing to say "we've got a database full of this information on just about everyone". If it's all been compiled and stored somewhere that is worth stealing simply because it takes 5 hours to collect this information on one person. So the doubleclick database is worth a few gazillion man hours of searching and they know things that I couldn't find out even if I tried (like stuff you answered on a survey or your answers to the purity test on TheSpark). This database is a web stalker's dream. People would pay hard cash for a breakdown of potential dating partner's interests and the like. Imagine a searchable database keyed on dress size.

He acted responsibly

[Galvatron]

He didn't just say "oh, it's been published by others, I'll go ahead and break the story too." He gave them an additional two days to get back to him, and the fact that they didn't suggests that they probably never will. After 5.5 days, if a company hasn't responded to a message, they have probably chosen to ignore it.

I am actually quite impressed with the journalisitic merits of this article, and I'm happy that Slashdot has started putting more research into their stories.

Crossing the line

[ryanr]

I'm continually amazed at all the parties "investigating" this problem. There's a line between checking for evidence of a vulnerability, and actually exploiting it. I'm seeing any number of people in thes story going over the line, and actually compromising DoubleClick.

Re:Crossing the line

[QuantumG]

if you left your car unlocked on a busy street and someone opened the door, looked in the clove compartment, found your cell phone number and gave you a call to tell you it was left unlocked you would thank them, would you not? I can see the respond to my post already "but it's not like their server is left unlocked, it's more like someone uses a slim jim on your car and then calls you to tell you they could do it". Bah.

Err..

[Canonymous Howard]

Over the last week there have been unsuccessful attempts made to hack into DoubleClick's servers. Those situations were immediately corrected

What, so they called the crackers and showed them how to do it right?

Re: Menthol is not a disability!

[SpookComix]

"...people who cannot use editors because of disabilities, menthal or physical."

I know, some people don't prefer the cool, crisp taste of menthol cigarettes. That's their choice. But to say it's a disability? I've smoke menthols before, and I can use an editor just fine.

I'm really beginning to doubt the level of intelligence displayed here on Slashdot.

--SC

Re:Poof! stuff breaks!

[caffeineboy]

Yes, and once you do this, any number of sites stop working for you and all of your users, depending on how the site is coded.

I wish there a nice, free way to block ads that is transparent to end users and doesn't break everything. I used to use junkbuster, but it broke so many sites that people who use my computer (roommates) that I just stopped using it. Steve Gibson at GRC had a registry patch that added a bunck of web buggers to the "hostile" zone of internet explorer which worked pretty well, but then I don't use IE either.

Wishful thinking...

A director at DoubleClick....

[Crambone]

is on the Board of Directors at Internet Security Systems (ISS) .... You would think that they would have thought to at least run ISS Internet Scanner against their websites or had a third party PenTest of their site in the past 2 years. It would have surely found that backdoor.

Re:Poof! stuff breaks!

[ciscoeng]

Webwasher (www.webwasher.com) does a pretty decent job. And hey, it's free!!
That, along with a cookie filter to rid the doubleclick garbage, works pretty well...

Re:First Law

[leviramsey]

The reasoning behind the lifestyle profiles for security clearances is very simple. If you, say, have a large collection of kiddie porn on your system at home, that's a huge security risk, because somebody else could discover it and blackmail you into handing over secrets.

This was the reason that the CIA still has a prohibition on employing closeted homosexuals.

Of course, if you're fully out in the open about having the kiddie porn, then there's no reason to deny the clearance.

Why were they corrected?

[mgkimsal2]

Over the last week there have been unsuccessful attempts made to hack into DoubleClick's servers. Those situations were immediately corrected," said Jules Polonetsky, Chief Privacy Officer, DoubleClick.

If the attempts were unsuccessful, what needed to be corrected. If my firewall is blocking ports, people will be unsuccessful at hitting my site, and nothing needs to be corrected. I don't get it.

Re:Disclaimer and Conflict of Interest

[ichimunki]

I've never seen a disclosure of that type in any "responsible" print journal, nor in their online versions. Typically, it is assumed that advertising and editorial policy are not intertwined. In fact, including information that they are a primary advertiser on the site might imply that financial considerations had colored the presentation.

Lying as good netizenship

[clary]

You are being watched. Act accordingly.

Whenever possible, when I am asked for information on the net, I lie like a dog. Many sites that require filling out a form to download software now have in their database a 130-year-old woman who earns $300,000+ per year, lives in Alaska, likes rollerblading and tofu, and receives email at webmaster@microsoft.com.

Just my tiny contribution to the cause...

Re:First Law

[clary]

The first anti-profiling law we need is one that states that no company or govenment can make submitting a profile a condition of employment or contract.

I agree "profiling" by government agencies should be strictly limited.

However, why should a company not be able to impose what conditions of employment it sees fit? I know this is not a popular idea, but damn man, go get another job if you don't like the conditions at one company. Or start your own company with conditions you do like.

Re:First Law

[MadAhab]

They used to extend this prohibition to all homosexuals, not just closeted ones. That's why William Burroughs called being homosexual "the perfect cover." So, in fact, NAMBLA might just be the best cover for operating an open nest of spies that the CIA ever had.

Boss of nothin. Big deal.
Son, go get daddy's hard plastic eyes.

Re:First Law

[leviramsey]

I was only saying that in cases where you're applying for a security clearance that a profile is a good thing. If the gov't can turn up something on you, the odds are that anybody else will be able to, as well.