FBI Seizes Servers In Virginia

Axolotl_Rose writes "The FBI has seized servers belonging to several clients of a hosting company in Reston, VA, disrupting service for many other clients. 'In an e-mail to one of its clients on Tuesday afternoon, DigitalOne’s chief executive, Sergej Ostroumow, said: “This problem is caused by the FBI, not our company. In the night FBI has taken 3 enclosures with equipment plugged into them, possibly including your server — we cannot check it.” Mr. Ostroumow said that the FBI was only interested in one of the company’s clients but had taken servers used by “tens of clients.” He wrote: “After FBI’s unprofessional ‘work’ we can not restart our own servers, that’s why our Web site is offline and support doesn’t work.” The company’s staff had been working to solve the problem for the previous 15 hours, he said.'"

Comments

Restore from backup?

[gmhowell]

Couldn't they restore their customers' sites from backup?

Re:Restore from backup?

[mug funky]

not with half the datacentre gone, they can't.

the backup system was probably in one of the _racks_ the FBI seized.

Re:Restore from backup?

[scdeimos]

Restore to what? From what I've read DigitalOne's a co-lo customer and the FBI's taken all their physical hardware.

Re:Restore from backup?

[Michael Woodhams]

I've been around long enough to remember the Secret Service raid on Steve Jackson Games, which was the triggering event for founding the EFF.

Most companies don't have "The Feds turn up with search warrants and take all your stuff, including backup tapes" as a threat they plan for in their backup strategy. Off site backup doesn't protect against this.

I don't know what the problem is in this case - whether the backups were also seized, or that they simply lack the hardware to restore on to.

Re:Restore from backup?

[Dahamma]

If it's just a colo, the customers may own their own servers (and be responsible for the software on them as well as backups).

If the servers were important, it's even possible they had a few for redundancy - unfortunately, redundancy is usually designed to account for simple hardware (or software) failures, and doesn't do much good when someone takes ALL of them...

Re:Restore from backup?

[Black Parrot]

Of the data, yes. Of the hardware, which is currently missing, not really.

Really? I copy my hardware to my 3-D printer every night.

Re:Restore from backup?

[gmhowell]

Restore to what? From what I've read DigitalOne's a co-lo customer and the FBI's taken all their physical hardware.

That's where you went wrong: you read the article. I didn't bother.

Re:Restore from backup?

[gmhowell]

I've been around long enough to have had a UID on that system :p

This shouldn't be much different than "a hellmouth opened up under the datacentre and swallowed it" or "the tsunami washed it out to sea" or "a stray SCUD hit the building". While ridiculous, it would seem that a visit by the FBI is about as catastrophic as some naturally occurring events that one might want to plan for. I'm not in disaster recovery, so I dunno.

I'm also curious how dodgy the customer was and if the service provider knew. (IOW, did the Feds bust an online pharmacy that was known about?)

Re:Restore from backup?

[CAIMLAS]

If the FBI has taken a full rack or more of equipment (as the article suggests), and they're a small shop, it would seem to me that a day or more is not an unreasonable recovery time.

Also, a hosting company may not actually do backups for customers, they may just 'rack and manage' on an exigent basis, leaving day-to-day to the customer.

Look, it's more than possible for a single guy to manage a half dozen racks of equipment on his own w/o much issue. Two, three guys, done right with good infrastructure, could do a couple dozen. We're not talking about anything complex, just simple single servers running an application or three. In this situation we're talking about a web hosting company, where they're constantly doing piddly 'little' things but almost always running short staffed. Switching is done by one guy/group, and the server maint by others. There is no room for 'disaster recovery in an instant' here. It'll be all up-hill, in the snow, in January, on Mars.With a higher than expected gravity.

Those same three guys are going to be hard pressed to rebuild their own infrastructure in day, too, backups or no backups. Figure it's noon before they even get chassis from Dell/IBM/HP to replace the ones stolen by the FBI that had their infrastructure on it, and then they've got to rebuild the racks, too - cabling, racking, and hardware RAID (like that doesn't take forever to perform). Considering it takes, what, 10 minutes? on some of these newer IBM servers to boot, this is hardly surprising. Add to all that the fact that their tape backup system, their disk backup system, and/or infrastructure switches may have been taken, and you've got a huge, huge headache. It takes, what, a day for two guys to simply install, cable, and rack a single rack chassis (guessing here) to all 40+ Us? And realistically, you can't have many more than 2-3 guys doing the work.

I'd be surprised if they got back up to 'fully operational' within 2-3 days. I'll be impressed if they don't go out of business.

Re:Restore from backup?

[AmiMoJo]

Is there any penalty for the FBI grabbing the wrong servers or causing massive disruption to innocent people?

I have always found it troublesome that law enforcement seems to be able to smash your nice front door down, take all your stuff, sit on it for a year or two for "analysis", wipe the HDDs and eventually give it back to you, and meanwhile you lose your job*... Yet there is no come back for them. No matter how badly the bungle the investigation, how much collateral damage, how much it screws up your life. I can understand the need for law enforcement to operate without fear of being liable for large sums of money, but there should also be some kind of compensation fund for the wrongly accused and innocent bystanders.

* That actually happened to the admin of the Oink BitTorrent tracker, who was eventually found innocent of all charges.

Re:Restore from backup?

[tbird81]

Really? I copy my hardware to my 3-D printer every night.

According to the media companies, you've just stolen that hardware!

Re:Restore from backup?

[paulo.casanova]

From Professor Mark Stevens' page in California State University

Suing the government is the second most popular indoor sport in America, and police are often the targets of lawsuits, with over 30,000 civil actions filed against them every year, between 4-8% of them resulting in an unfavorable verdict, where the average jury award is $2 million. This isn't even counting the hundreds of cases settled thru out-of-court settlements, which probably runs in the hundreds of millions and involves about half of all cases filed. It may take up to five years to settle a police liability case.

The FBI should try that on cloud hosting

[initialE]

1. Take the servers
2. There is nothing on the servers - take the Storage
3. The storage is remotely replicated - pull the remote storage
4. You can't pull the remote storage, you don't have jurisdiction overseas

Re:The FBI should try that on cloud hosting

[Swampash]

It's ok, they backed up everything to S3 using Dropbox, and Dropbox has a new feature where you can log in to any account with any or no password.

Solution

[PPH]

Host offshore.

Re:Solution

[TooMuchToDo]

The hosting company I co-own with the rest of my employees is mid-sized (several million a year, but under 10 people), but we operate this way. Equipment is owned by corporations incorporated in the jurisdiction where it resides on a country-level basis. We own gear in the US, the EU, Japan, China, and Australia. No corporate entity is tied to another, and resources are redundant through the infrastructure. Come to me in the US with a subpoena for anything on any of our gear outside the US? Fark off. When the hell did people give up on their principles?

Re:Solution

[tomthepom]

DigitalOne is based in Switzerland, they did host offshore in the US. That might have been a mistake.

Civil and criminal liability

[dgatwood]

I think it's time to hold the FBI to the same standards that they would hold the rest of us. If I went in waving a gun around and demanding to walk away with somebody else's server, they'd throw my ass in jail.

If they want access to a particular client's content, they can go through the same process as a DMCA takedown request or a backup request would. They make a request, the company yanks that customer's access, then clones that customer's data onto a new drive, then hands them the drive.

As far as I'm concerned, every single client of this ISP ought to sue the FBI for the damage they caused—for the downtime, for the loss of data, for the time spent trying to reach the ISP to figure out what was going on, for the cost of any failover hardware or service that they had to pay for in lieu of that service, etc. If the FBI had to pay out a few million dollar settlements every time they pulled a stunt like this, they'd think twice about acting like a bunch of thugs, and they would go through proper channels and do their investigation in a way that doesn't cause collateral damage.

There's simply no excuse for such sloppy investigative work. If they screwed up so royally with the servers, you have to wonder how many grievous errors they made in other areas that would lead to the evidence being declared tainted, criminals going free, etc.

Re:Civil and criminal liability

[icebike]

You can try to file a suit, but you probably wouldn't get anywhere.

The Federal Tort Claims Act was enacted by Congress in 1946 to allow citizens to sue the federal government. Prior to that you had to get something
passed by congress in order to sue the government.

From http://www.finchmccranie.com/refresher.htm

While the passage of the FTCA constitutes a limited waiver of sovereign immunity, Congress specifically limited the government's amenability to suit in a variety of different circumstances. In 28 U.S.C. 2680, Congress specified that its limited waiver of immunity would not apply to the following claims:

(a) any claim based upon an act or omission of an employee of the government, exercising due care, in the execution of a statute or regulation, whether or not such statute or regulation be valid, or based upon the exercise of performance or the failure to exercise or perform a discretionary function or duty on the part of a federal agency or an employee of the government, whether or not the dis- cretion involved be abused; ...

So you see, you are effectively shut down before you get to the courthouse steps. All they need do is say "We had evidence that all servers we took were involved" and there is nothing more you can do. You will not be granted the ability to examine that evidence.

Re:Civil and criminal liability

[i.am.delf]

I wonder is the FBI could subpoena a critical control system say a Siemen's SCADA controller that had been hacked. If this control system were used to control a machine capable of causing grievous bodily harm or death, would the FBI not be negligent? If the FBI took a server legitimately housing an e-commerce site containing customer data, would that be considered a data breach under California law?(FTCA torts are determined under state law not under Federal)

The FTCA specifically allows claims based upon negligence to be brought against the Federal government. However, you are correct that this liability is limited by the exemption you have posted. I don't think that any reasonable person could suggest that if data were stored on a server at Google and distributed over an entire datacenter, that the entire datacenter could be seized. That exemption gives Federal employees the leeway to search a house they reasonably believe contains a fleeing suspect. It does not cover something like seizing all the cars on a block because one contained drugs. No reasonable person would suggest such a ludicrous argument in that case nor should anyone suggest that since these 100 or so servers are in close proximity in a data center they may all be seized.

Re:Civil and criminal liability

[sjames]

They rendered even the servers they DIDN'T take unbootable. That doesn't sound like due care. They had the opportunity to have employees of the colo (who were not under investigation) which machines belonged to the party named in the warrant, but they failed to do so. Again, no care at all, much less due care.

Re:Civil and criminal liability

[Moskit]

> They make a request, the company yanks that customer's access, then clones that customer's data onto a new drive, then hands them the drive.

Depends on legislation. In some countries only the original drive is considered evidence, therefore it cannot be returned until the whole process is over (think years). Copy cannot be made and returned either, for some other reasons (can't recall exactly).

Does the Constitution still mean anything?

[mykos]

Each of the clients who had their property seized without warrant should bring suit.

Re:Does the Constitution still mean anything?

[icebike]

Responding to your title, "Does the constitution still mean anything", the answer is NO.

Just about here is where I get jumped on by everybody who supports the Constitution and hold it dear. Who doesn't?

But the point is, nothing written in the constitution means anything any more, and hasn't for a long time.
Every sentence and every clause has been violated and circumvented by a web of laws and rulings such that any citizen who points to the constitution in his defense is laughed out of court. In the legal profession, an appeal to the constitution is a huge inside joke. The sign of a rube. A target to be fleeced.

Re:Something has to change

[n5vb]

This assumes that the FBI has some clue of what they're looking for, or that they know enough to be able to get a copy of just the directory tree containing that particular client's content. I don't think that's a safe assumption in most cases. :p

That being said, if it were any hosting service I were running, there'd be enough offsite hardware and data backups to be able to get my clients' sites back up at least to a recent and consistent state, if not the current state ..

Act of War

[sanzibar]

next time, use a drone.

FBI: Driving businesses out of the country

[mykos]

I think most of the smart IT people are beginning to view the U.S. as a threat to their business. If U.S. investigative agencies can disrupt dozens, or even thousands, of innocent individuals and businesses with impunity, why the hell would anyone take the risk hosting in the U.S.?

Re:FBI: Driving businesses out of the country

[MightyMartian]

Because, of course, other countries are so much less intrusive.

Re:FBI: Driving businesses out of the country

[Anzya]

Meh, this has already happened in Sweden when the police confiscated a lot of servers and disrupted service for other customers just to get at The Pirate Bay. Who cares if other gets hurt in the process for the greater good...?

Re:FBI: Driving businesses out of the country

[cold fjord]

I think most of the smart IT people are beginning to view the U.S. as a threat to their business.

Your link leads to an article complaining about shutting down "websites involved in copyright infringement, the sale of counterfeit goods or child pornography", among other things. I doubt most smart IT people are involved in criminal enterprises. If most of the "smart" people you know are, maybe you should think about moving to a different part of the industry. And when I say different, I mean legal.

   

Re:FBI: Driving businesses out of the country

[isorox]

I think most of the smart IT people are beginning to view the U.S. as a threat to their business.

Your link leads to an article complaining about shutting down "websites involved in copyright infringement, the sale of counterfeit goods or child pornography", among other things. I doubt most smart IT people are involved in criminal enterprises. If most of the "smart" people you know are, maybe you should think about moving to a different part of the industry. And when I say different, I mean legal.

 

Unless you run your own data center, and have multiple upstream links, you may be relying on a data centre that someone else is hosting those things -- either knowingly, or because a single box was compromised.

If you're not a beomouth fortune 500 company, chances are you've got a couple of physical machines in a colo, or even just a VM or two. You have no control over who Rackspace rent their servers and space too, so when the FBI come calling, you lose money.

Re:FBI: Driving businesses out of the country

[nedlohs]

And you know that the domain registrar you used didn't sell a domain to a single person/enterprise that might be suspected of a crime. And the DNS provider you use doesn't have such a customer. And the hosting provider you use doesn't have such a customer. And the data center the servers are in doesn't have such a customer. And you and any of your providers and any of their customers haven't annoyed someone enough to get setup for such a raid.

However, for a lot of the rest of us issues of size and finances mean we just lease space on a shared machine, or lease some machines, or lease some rackspace and hence have no control over the activities of nearby servers.

Also the rest of us love in a world where people make mistakes and say no-knock RAID the wrong house or seaze the wrong hardware...

Re:Not Surprised

[icebike]

Well I suspect walking in and taking every server in site is not going to go over well
in the long run. Group punishment is hardly constitutional, and as soon as some deep pockets
fight back this process will stop.

Still these lulzsec clowns need to be reined in and perp walked. If they had a point to
make they've already made it, now its time to pay the piper.

Ultimate DOS

[Anonymous Coward]

It's the ultimate Denial Of Service attack:
1) Co-locate stuff that the FBI doesn't like with the server that you want to DOS
2) Report your server to the FBI
3) Sit back and let the FBI do the rest.

Re:Not Surprised

[epyT-R]

I see it as one crime syndicate making a hit on another. The feds are no more principled...

Re:Not extreme

[hawguy]

I am a federal agent (non-FBI) who has seized large amounts of digital evidence. In criminal cases, you need entire hard drives so you can do forensic extraction. Can you ask the ISP to retrieve the data for you? Yes. However, it depends on 1.) Is this an email address or a large organization with colocated servers. 2.) How much do you trust the ISP? (based on past actions, size, clientele, etc.). BTW, if you search large companies who have their congressman on speed dial, you can be assured that the agents and judge have evaluated the impact to legitimate business vs illegal activity.

I'd think that the same thing applies when the FBI sees a suspect enter a parking garage - they know he entered the garage and are pretty sure that he hid his contraband in a car. The garage owner might be working with the suspect, so they can't trust him. The question is, can they seize all 200 cars in the garage and tow them back to be disassembled and searched to be eventually returned to the owners, perhaps no longer in working order? Would any judge allow that?

If the answer is no, why is it different with servers?

Re:Not Surprised

[sortius_nod]

To think that a law enforcement agency, and yes, that's all they are, can walk into a premises with a warrant for specific information and take most of your equipment goes against the whole idea of "freedom".

Unfortunately this is not the first time the FBI have done stuff like this, just watch Freedom Downtime (actually about Kevin Mitnick) and see what happened to Bernie. It's been happening for decades to people who haven anything to do with hackers, why not go after company equipment now rather than your dad's computer?

Re:The reason they took the whole rack....

[Wingman 5]

If I keep all of my data in a strongly encrypted container (that does not have a password that is brute force able in a reasonable amount of time), how do you expect to gain anything meaningful "dealing with it as mere data" without the decryption key which was stored in ram till you shut the machine off to clone the drive?

Re:Cloud

[billcopc]

(unless it's been bugged)

You just negated your own argument. Sorry, man, do not pass go. Do not collect 200 karma.

Law enforcement needs to decide on a firm, reliable way to identify those responsible for cybercrime, to punish them and ONLY them, not the people who happen to be providing service along the way.

Do they shut down the power company every time the crooked DEA finds a grow op ? No, because the power company is simply providing a service irrespective of usage. We need to start treating the internet like any other utility, since that's what it has become. Want a site shut down ? Track the IP, look up Whois, call the ISP, follow procedure. Randomly and illegally seizing property is NOT going to solve any problem. It will only incite more to rebel against the broken legal system.

Go ahead FBI, ruin someone's business and livelihood over fabricated evidence and feeble-minded assumptions, but don't act surprised when that ex-entrepreneur shows up at your doorstep with a bottle of jack and a loaded shotgun. Actions have consequences, and abuse of power merits the harshest consequences of all.

Hosting centre is at fault

[jamesh]

The hosting centre is at fault here. "Naughty Servers" should be clearly labelled as such so they can't be mistaken for "Benign Servers". If those fatcats in Washington had just listened when the 'Evil Bit' was first proposed we wouldn't be in this mess now!

Re:The reason they took the whole rack....

[fluffy99]

They don't need to keep the whole rack powered, just the one machine they are interested in, they could power down the rest of the rack and a off the shelf UPS could run it for plenty of enough time to get it to a truck with a inverter on it.

As for the "magic splicing" it is not hard to do, anyone with a basic understanding of electric circuits can splice two live cables together.

There is a product called HotPlug that is meant for seizing assets without powering them down. It works pretty slick. Basically, you plug it into the same power strip, flip the switch and unplug the powerstrip from the wall. You can also splice into the cord or outlet if needed.
http://www.wiebetech.com/products/HotPlug.php

Re:good point

[X0563511]

Not really. To work (the analogy) they would have to lift and tow away whole sections of traffic at a time, only to return the vehicles (maybe, if you're lucky) weeks or months later.

Re:Machines won't be coming back

[X0563511]

Which is bullshit.

The equipment needs to be kept until guilt or innocence is determined. At that point, any equipment belonging to an innocent needs to be fucking returned.

It's larceny otherwise. Can't understand how they get away with this...

It's not even like I'm saying compensation should be issued! Just an "our bad, here's your stuff!"

Re:you can splice cables to a single server

[X0563511]

Have you ever tried to move a server out of it's rack, out of the building, into a vehicle, and then wherever it needs to go... ... without the disk curb-stomping it's heads all over the platters?

Power is only part of the problem.

Iceland

[biodata]

Decent infrastructure, decent government, some coastguards but not really interested in starting wars with anyone unless it's about fish, and a legislative framework that is conducive to free speech.

A problem endemic with law enforcement

[Attila Dimedici]

I am pretty sure this happened as a result of a problem that is endemic with law enforcement. A large percentage of people in law enforcement have come to believe that all people that they interact with are criminals who are acting to keep law enforcement from discovering the evidence to convict that person and/or others. As a result, they did not trust the hosting company to work with them to obtain all of the data of the target of their investigation.
The proper way to have done this would have been to go in with someone from the FBI who was technically proficient who would then work with the hosting company to isolate and migrate all of the virtual machines containing the target's data to a single server (or several, if that was necessary) and seize that server(s).

Re:Not Surprised

[silas_moeckel]

Funny they have asked for just that.from hosting companies. They do not seize the phone companies computes when they have a warrant for info, they send the paperwork and the phone company sends the data. I've been at the receiving end of FBI warrants in hosting companies we package up what they need and even bill them for our time. Unless they had reason to believe that the hosting company or it's staff were part of the criminal activity there is no reason to do this. Sometimes they were even smart enough to ask us to leave it up and sniff it's traffic for weeks at a time.

As far as avoiding this sort of thing it's no different than any other major disaster you need backup servers with a different provider a good physical distance away.

Re:Not Surprised

[Zaiff Urgulbunger]

They (LulzSec) should've kept quiet about the US Senate hack and just used their web-servers. *THEN* it would've been more Lulzy when the CIA took down the US Senate.

Re:Not Surprised

[ldobehardcore]

The fact is:
The FBI has a whole suite of tools for copying hard disks and other digital media in 1:1 format very quickly A couple of them are EnCase and FTK (both of which I found on This Wikipedia page.) Just at a glance, there are over a dozen tools the FBI could have used to make a 1:1 copy of the hard disk they were searching for.
If it were a criminal investigation I would assume they would have to take at least some hardware anyway for original evidence.
If it were a civil deal I can't imagine a single instance in which the need to grab that equipment was so damn urgent that they'd be obligated to screw over a business.

Take my commentary with a grain of salt though....I've never been raided by the FBI, and I'm sure they can get approval to do anything in the name of protecting MPAA or RIAA's interests, since so much of the work that justifies the FBI's existence comes directly from the pockets of industry in greasing the wheels of government.

Re:Not Surprised

[Dunbal]

Er, the hosting company told them exactly where the data they were looking for was, but they still chose to take the entire racks. tl;dr - read the fucking article.

Re:Just trolling, ignore.

[malsbert]

Hi there Gonzo.

I'm not implying anything. I was simply Idling, when i noticed an AC had replied to one of my comments. As the comment in question is semi-flamebait
( O’er the land of the oppressed and the home of the cowards! ) and AC seemed to "accept the challenge" ( Italian fascists, German Nazis, etc ) i simply "shot" from the hip, to see if he was still online and would care to entertain me. to keep you and any others out of the crossfire i did set the comment subject to: "Just trolling, ignore", But it seems you chose not to :)

As for any "issues" the US my have, i can only say; you people need to get out more! your issues are by no means US issues, we have plenty of the same things right here in little old Europe and i'm pretty sure the rest of the world is in position to point fingers :)

So; sorry if spoiled you morning, next time do what subject line says and IGNORE ;)

Re:Not Surprised

[lthorne]

As the owner of a data center, I welcome this type of actions by the FBI. we filter all customers before giving them access to any server and monitor them on a weekly basis for spam, viruses and phishing scams. I for one am tired of the phishing scams that come into my networks from GoDaddy, bluehost and an array of US based discount hosting providers. You would not believe the number of brute force and Ddos attacks our firewalls log and block in any given hour and at least 60% originate from US based hosting provider ip addresses that often have nothing more than an apache/tomcat default setup page. In short: it's about f#%$ing time!

Re:Not Surprised

[HappyPsycho]

RTFA

DigitalOne provided all necessary information to pinpoint the servers for a specific I.P. address, Mr. Ostroumow said. However, the agents took entire server racks, perhaps because they mistakenly thought that “one enclosure is = to one server,” he said in an e-mail.