FBI Seizes Servers In Virginia

Axolotl_Rose writes "The FBI has seized servers belonging to several clients of a hosting company in Reston, VA, disrupting service for many other clients. 'In an e-mail to one of its clients on Tuesday afternoon, DigitalOne’s chief executive, Sergej Ostroumow, said: “This problem is caused by the FBI, not our company. In the night FBI has taken 3 enclosures with equipment plugged into them, possibly including your server — we cannot check it.” Mr. Ostroumow said that the FBI was only interested in one of the company’s clients but had taken servers used by “tens of clients.” He wrote: “After FBI’s unprofessional ‘work’ we can not restart our own servers, that’s why our Web site is offline and support doesn’t work.” The company’s staff had been working to solve the problem for the previous 15 hours, he said.'"

Comments

Not Surprised

[OverlordQ]

And so it begins . . . .

did lulzsec think they could get rid of it forever?

Re:Not Surprised

[icebike]

Well I suspect walking in and taking every server in site is not going to go over well
in the long run. Group punishment is hardly constitutional, and as soon as some deep pockets
fight back this process will stop.

Still these lulzsec clowns need to be reined in and perp walked. If they had a point to
make they've already made it, now its time to pay the piper.

Re:Not Surprised

[epyT-R]

I see it as one crime syndicate making a hit on another. The feds are no more principled...

Re:Not Surprised

[kakarote]

dissertation for access the seizes servers people need this http://ow.ly/5nrhM B)

Re:Not Surprised

[sortius_nod]

To think that a law enforcement agency, and yes, that's all they are, can walk into a premises with a warrant for specific information and take most of your equipment goes against the whole idea of "freedom".

Unfortunately this is not the first time the FBI have done stuff like this, just watch Freedom Downtime (actually about Kevin Mitnick) and see what happened to Bernie. It's been happening for decades to people who haven anything to do with hackers, why not go after company equipment now rather than your dad's computer?

Re:Not Surprised

[cold fjord]

I was just going to say.... "so we actually have no idea about any of this, right?"

I'm sure most people are clueless. I suppose they could be doing it for the "lulz".... but that really isn't their style.

FTA -

A government official who declined to be named said earlier in the day that the F.B.I. was actively investigating the Lulz Security group and any affiliated hackers. The official said the F.B.I. had teamed up with other agencies in this effort, including the Central Intelligence Agency and cybercrime bureaus in Europe.

Re:Not Surprised

[anagama]

Indeed -- FBI is great at DOS apparently.

Re:Not Surprised

[Glendale2x]

Well, they didn't walk in and take every server. They served the colo facility and they in turn pointed out that specific customer's equipment, likely leaving everyone else in the colo alone. FTA:

DigitalOne had no employees on-site when the raid took place. The data center operator, from which DigitalOne leases space, passed along the information about the raid three hours after it started with the name of the agent and a phone number to call.

I'm sure the colo operator had no idea which servers of that customer were specifically related to the warrant, so they took everything that fell under the customer that they had the warrant for.

Re:Not Surprised

[tinkerghost]

RTFA - The FBI took all of the racks that had a server related to the warrant. So, they took servers for "tens of customers", rather than just servers for a single company,

Re:Not Surprised

[Lord Byron II]

To think that a law enforcement agency, and yes, that's all they are, can walk into a premises with a warrant for specific information and take most of your equipment goes against the whole idea of "freedom".

What's the FBIs alternative? They have a warrant for the server containing a particular person's information. They can't realistically walk in and snip out the part of the hard drive that contains that, can they? They have to take the entire server and on a shared hosting setup, that means dozens or even hundreds of clients. They also need to maintain a good chain of custody on that server, so they can't really ask the hosting company to replicate the data off of server X - they need the real thing and they need it intact.

Possible solutions that would have avoided this for the innocent:
-dedicated server
-colocation
-a hosting provider who has the capability to quickly bring in a backup

Re:Not Surprised

[silas_moeckel]

Funny they have asked for just that.from hosting companies. They do not seize the phone companies computes when they have a warrant for info, they send the paperwork and the phone company sends the data. I've been at the receiving end of FBI warrants in hosting companies we package up what they need and even bill them for our time. Unless they had reason to believe that the hosting company or it's staff were part of the criminal activity there is no reason to do this. Sometimes they were even smart enough to ask us to leave it up and sniff it's traffic for weeks at a time.

As far as avoiding this sort of thing it's no different than any other major disaster you need backup servers with a different provider a good physical distance away.

Re:Not Surprised

[datapharmer]

Welllll I suspect there is some middle ground between taking the server(s) the data is on and taking an entire rack... Clearly this wasn't a colocated rack or it wouldn't have had other people's info on it too, so take the part that is relevant and leave the rest alone.

Re:Not Surprised

[dkleinsc]

What's the FBIs alternative?

One possible alternative would be to go to the hosting company and say: "We have a warrant for the contents of this server for criminal investigation. Our techs will help you quickly set up a backup server so none of your other customers will be affected, and flip over everything but the bad guy's traffic to that server. We also have a warrant to wiretap the bad guy's traffic, so we're going to send the traffic to his website to this special box of ours for a while to collect evidence."

That's more expensive then just charging in and grabbing servers off the rack, but if there's a specific crime they're after, that would probably be a better approach - among other things, it would mean the suspect doesn't suspect anything's wrong.

Re:Not Surprised

[Zaiff Urgulbunger]

They (LulzSec) should've kept quiet about the US Senate hack and just used their web-servers. *THEN* it would've been more Lulzy when the CIA took down the US Senate.

Re:Not Surprised

[ldobehardcore]

The fact is:
The FBI has a whole suite of tools for copying hard disks and other digital media in 1:1 format very quickly A couple of them are EnCase and FTK (both of which I found on This Wikipedia page.) Just at a glance, there are over a dozen tools the FBI could have used to make a 1:1 copy of the hard disk they were searching for.
If it were a criminal investigation I would assume they would have to take at least some hardware anyway for original evidence.
If it were a civil deal I can't imagine a single instance in which the need to grab that equipment was so damn urgent that they'd be obligated to screw over a business.

Take my commentary with a grain of salt though....I've never been raided by the FBI, and I'm sure they can get approval to do anything in the name of protecting MPAA or RIAA's interests, since so much of the work that justifies the FBI's existence comes directly from the pockets of industry in greasing the wheels of government.

Re:Not Surprised

[Dunbal]

Er, the hosting company told them exactly where the data they were looking for was, but they still chose to take the entire racks. tl;dr - read the fucking article.

Re:Not Surprised

[ultranova]

As far as avoiding this sort of thing it's no different than any other major disaster you need backup servers with a different provider a good physical distance away.

And by "a good physical distance away", you mean "in another country who's government doesn't like the US, and who's military can fight them off", right?

Re:Not Surprised

[sorak]

Since those servers are "evidence", this is not unprecedented. I'm not defending it, but I am not expecting the courts to care either.

Re:Not Surprised

[black soap]

but... your information could have been anywhere in that machine. we saw the wires connecting it to other machines. Just be glad they didn't seize the whole internet as evidence.

Seriously, though - I imagine that whole "off-site backups" plan won't be effective much with they are seized too, along with your IP phones for thoroughness.

Re:Not Surprised

[black soap]

I don't think you understand how reservations work.

Re:Not Surprised

[chill]

Newsflash. There is NOTHING that can 1:1 bit-copy large hard drives QUICKLY. Drives that are hundreds of megabytes in size, or even terabytes take FOREVER to clone. God help you if they're in a RAID. Cloning them can take DAYS, even with hardware copy systems.

Re:Not Surprised

[lthorne]

As the owner of a data center, I welcome this type of actions by the FBI. we filter all customers before giving them access to any server and monitor them on a weekly basis for spam, viruses and phishing scams. I for one am tired of the phishing scams that come into my networks from GoDaddy, bluehost and an array of US based discount hosting providers. You would not believe the number of brute force and Ddos attacks our firewalls log and block in any given hour and at least 60% originate from US based hosting provider ip addresses that often have nothing more than an apache/tomcat default setup page. In short: it's about f#%$ing time!

Re:Not Surprised

[HappyPsycho]

RTFA

DigitalOne provided all necessary information to pinpoint the servers for a specific I.P. address, Mr. Ostroumow said. However, the agents took entire server racks, perhaps because they mistakenly thought that “one enclosure is = to one server,” he said in an e-mail.

Re:Not Surprised

[Omestes]

What's the FBIs alternative? They have a warrant for the server containing a particular person's information. They can't realistically walk in and snip out the part of the hard drive that contains that, can they?

From what I've read about this instance, they could have taken the single server for the company they had a warrant for, instead they took the whole enclosure (several, non-related servers as well). From the sounds of it, the owner/maintainer of the data-center gave them the relevant information to grab just the single server, and the FBI disregarded it, either willfully or via ignorance.

That last bit is the most important. If its just ignorance, then its a mere mistake. No worries. If its willful, then it is a bona fide abuse, and we should get all uppity and indignant.

Not that I'm defending the FBI here, or our general data seizure policies. There's something a bit wonky about taking people's livelihoods (if you actually need computers for your job), and huge amounts of unrelated data. Sure the FBI can grab everything in my house that can be considered a computer (my phone, my gaming consoles, my laptop, my PC, all of my onsite backup CDs, DVDs, and HDDS, and probably my MP3 player and cameras), but this gets absurd when all they need to know is if offending file x is present or actually illegal. So they get my full finances, years of communications with my family, business, coworkers, and such. They get 30 years of photography, they get my full music collection, my browsing history, my choice of wallpaper, etc... Most of which would fall out of the domain of information that is actually pertinent to their case, and rational for the warrant. There isn't much of a bar to keep them from mining your data for other offences if the original charges aren't backed up by the seized evidence.

  Worse, as far as I can tell, they don't have to return them in a timely manner, or even return them in the original, or even functional, condition. If you actually depend on your equipment for a living, this can be a very, very, bad thing. I find this a bit out of sync with how actual, physical, warrants function.

Re:Not Surprised

[stabiesoft]

Your assuming the server company will not do a pakistan and tip off the clients. I suspect the FBI did not trust the server company, hence they did a brute force.

Re:Not Surprised

[HappyPsycho]

Now lets tackle your countermeasures:

Co-location / Dedicated server - These guys took racks of equipment, and judging from what I can get off of google's cache of DigitalOne's site these guys don't offer shared hosting "DigitalOne offers Quad and Six core Xeon CPU's Blade Servers, equipped up to 128GB of memory, RAID controllers and SAS/SATA and Flash (SLC) hard drives.". Neither of those offer a solution to this problem, especially when you take into account the fact that the company actually told the FBI what server to take / which one they were after. Co-locating somewhere else? They'll just hit both at once, come on you haven't seen those cop shows on TV. While I'll concede that you aren't their target, I find it hard to believe that a bunch of hackers only have 1 server to co-ordinate their activities.

A hosting provider who has the capability to quickly bring in a backup - Lets think of this a different way, if this was the command and control for a botnet do you really think the FBI will let them bring it back online? Also if backups exist, the FBI would most likely take them to analyze what the servers have actually been doing.

Best defense is actually get a different hosting provider in a country that is not friendly to the US which will therefore say F U when the FBI comes knocking. No amazon hosting over in Europe probably won't do because the FBI has jurisdiction over at least part of the company. Lets just hope the hackers don't pick the same places.

Oh yeah, this is what the innocent have to do to protect themselves from their government, not the hackers. Terrorism - 1, US - 0.

Re:Not Surprised

[aix tom]

"Hundreds of Megabytes"? Like, the size of a CD? Takes a few seconds on my end here.

As for the "Terrabytes", here is the end of the log from one of or tape backups, to a 6 year old tape library:

Total number of bytes transferred: 247.79 GB
Data transfer time: 3,595.75 sec
Network data transfer rate: 72,261.73 KB/sec
Aggregate data transfer rate: 53,638.98 KB/sec
Objects compressed by: 0%
Elapsed processing time: 01:20:44

A quarter of a TB in 1:20 h, so a copying an entire Terabyte to tape over a 1Gbps Network could probably be done in about 5 hours or so.

Re:Not Surprised

[chill]

Typo. Hundreds of Gigabytes.

Yup, your already configured tape drive on the network is snappy. Now try it with drives physically removed from a non-connected system and do it while preserving the chain of evidence. It takes a lot of time.

Re:Not Surprised

[aix tom]

That's even faster. Connect disc to controller, do a copy of the raw device.

One that *will* take time is to come up with a system of checks and procedures that the chain of evidence is proven. But that has to be done BEFORE the process is started. So it has no bearing on the time the process itself takes.

The other thing that *will* take time is to figure out what the data mean, if and how it was RAIDed (pun intended) or encrypted. But that also has no effect on how long it takes to make a 1:1 copy of the bits.

Re:Not Surprised

[maxwell demon]

What if some illegal activity was done from a cloud process? Will they seize the complete cloud infrastructure?

Re:Not Surprised

[houstonbofh]

As far as avoiding this sort of thing it's no different than any other major disaster you need backup servers with a different provider a good physical distance away.

Under a different company, so the FBI doesn't seize them as well.

Re:Not Surprised

[Cramer]

Except they didn't take "one server", they walked in and took 3 entire RACKS of servers. Many of the devices taken have nothing at all to do with what they're after, and as such falls outside the scope of their warrant, but being the FBI they can go unchallanged. The really bad thing is just how horribly LEO's handle the stuff they've taken... if you ever get if back, it will be throughly trashed.

Re:Not Surprised

[silas_moeckel]

At this point if the US Government wants to kill an internet based company they can and will. If nothing else they will seize any domain names associated with the company. So you better be .ir or .cu and those governments better like you.

Re:Not Surprised

[Cramer]

They cannot seize backups they don't know about.

Cloud

[seepho]

Need to suffer the same repercussions that anyone fitting the loose modern definition of 'cyber criminal'?

To the cloud!

Re:Cloud

[epyT-R]

Not a fair comparison.. An axe is quite simple to verify because it has no hidden function. it's not a black box. (unless it's been bugged). computer equipment is the ultimate trojan horse because they are so difficult to completely audit.

Re:Cloud

[billcopc]

(unless it's been bugged)

You just negated your own argument. Sorry, man, do not pass go. Do not collect 200 karma.

Law enforcement needs to decide on a firm, reliable way to identify those responsible for cybercrime, to punish them and ONLY them, not the people who happen to be providing service along the way.

Do they shut down the power company every time the crooked DEA finds a grow op ? No, because the power company is simply providing a service irrespective of usage. We need to start treating the internet like any other utility, since that's what it has become. Want a site shut down ? Track the IP, look up Whois, call the ISP, follow procedure. Randomly and illegally seizing property is NOT going to solve any problem. It will only incite more to rebel against the broken legal system.

Go ahead FBI, ruin someone's business and livelihood over fabricated evidence and feeble-minded assumptions, but don't act surprised when that ex-entrepreneur shows up at your doorstep with a bottle of jack and a loaded shotgun. Actions have consequences, and abuse of power merits the harshest consequences of all.

Re:Cloud

[icebraining]

How was it random? They didn't shutdown the datacenter, just a couple of racks. What exactly do you expect them to do?

Re:Cloud

[GameboyRMH]

Actually this seems like the best reason to have clustered hosting, something like Amazon EC2. If TEH CYBER POLICE seize a bunch of servers for no reason, your site will stay online! Backhoes are yesterday's biggest threat, it's the cyber police admins need to worry about today!

Re:Cloud

[tinkerghost]

How was it random? They didn't shutdown the datacenter, just a couple of racks. What exactly do you expect them to do?

Hmmm, how about they take the servers listed in the warrant and leave the ones that aren't. You know, follow that pesky 4th amendment. It probably would have been easier to remove a couple of servers than an entire rack - except the FBI probably 'forgot' to have anyone with any computer experience accompany the seizure team.

Re:Cloud

[Culture20]

How was it random? They didn't shutdown the datacenter, just a couple of racks. What exactly do you expect them to do?

there can easily be 20-40 servers in a rack. Not all colos keep a "one user per rack" policy. Maybe they should take just the servers in question?

Re:Cloud

[__aagbwg300]

That's kind of the problem, right? The warrant, which is sealed, probably lists the servers by IP or MAC address and not by model name and number. They might be able to make an educated guess as to which rack is housing the machine, but since there were no employees present to actually identify the servers by IP, it's hard to see how they could do any different.

Re:Cloud

[_Shad0w_]

Properly co-ordinate with the hosting provider - not the facility owner - and remove only the servers directly related to their suspect? The co-lo provider will know exactly which of the machines needs to be removed, given the relevant information, e.g. the name of the customer or the IP address of the server(s).

Re:Cloud

[_Shad0w_]

It should be noted, they were actually told exactly which server they wanted based on the IP address they gave DigitalOne. They just chose to take the entire enclosure instead - I'm going to go out on a limb and say this is because they don't actually have anyone with domain knowledge and thus have no idea what they're actually doing.

Re:Cloud

[icebraining]

Assuming they, or even the colo employee, knew, and it's very possible that they didn't (and they couldn't ask in advance, since the suspect could be tipped and start deleting stuff).

Re:Cloud

[ultranova]

It should be noted, they were actually told exactly which server they wanted based on the IP address they gave DigitalOne. They just chose to take the entire enclosure instead - I'm going to go out on a limb and say this is because they don't actually have anyone with domain knowledge and thus have no idea what they're actually doing.

"Nice servers you have here. Wouldn't want anything to happen to them, right? Better not do business with anyone we don't approve of, then!"

Never attribute anything to incompetence that can be adequately explained by malice.

Re:Cloud

[_Shad0w_]

Cock-up before conspiracy.

Re:Cloud

[_0xd0ad]

since there were no employees present to actually identify the servers by IP, it's hard to see how they could do any different

TFA's been updated a few times, so maybe this quotation wasn't included when you read it (yes, that was a joke).

DigitalOne provided all necessary information to pinpoint the servers for a specific I.P. address, Mr. Ostroumow said. However, the agents took entire server racks, perhaps because they mistakenly thought that “one enclosure is = to one server,” he said in an e-mail.

Re:Cloud

[billcopc]

No one is prepared to change how they live. The problem is that these abuses of power are forcing us to change our ways, to better suit the corporate interests that now dominate every government.

Push hard enough, and even the Xbox Live folks will get pissed off and fight back. It's going to take a hell of a disaster to get the average citizen motivated, but it will happen, that much is certain.

Re:Cloud

[houstonbofh]

I see a new business opportunity here. "The Pirate Bay Hosting Company!" The harder they push, the more reasonable the "criminals" look.

Something has to change

[Anonymous Coward]

It is easy to acknowledge the FBI and other police force's need to obtain servers belonging to a client, but with the reality being multi-client servers that most that should be allowed is a copy that doesn't violate any other customer's right of privacy.

Re:Something has to change

[n5vb]

This assumes that the FBI has some clue of what they're looking for, or that they know enough to be able to get a copy of just the directory tree containing that particular client's content. I don't think that's a safe assumption in most cases. :p

That being said, if it were any hosting service I were running, there'd be enough offsite hardware and data backups to be able to get my clients' sites back up at least to a recent and consistent state, if not the current state ..

Probably a proxy box

[assemblerex]

They'd have to be pretty stupid to use a server located in the USA.

Re:Probably a proxy box

[iamhassi]

So just don't have your site hosted in the US and you're instantly safe?

FBI isn't the only problem. There are outages and disasters to worry about, so could this be avoided? As someone with a website, how would I go about having my site mirrored on different hosts so if one goes down another takes over automatically? I know I could "get a good host" but every host claims they have redundant backups and everything, I would feel more comfortable if I used several entirely different hosts and had the content mirrored nightly. Is that possible?

Re:Probably a proxy box

[malsbert]

As someone with a website, how would I go about having my site mirrored on different hosts so if one goes down another takes over automatically? I know I could "get a good host" but every host claims they have redundant backups and everything, I would feel more comfortable if I used several entirely different hosts and had the content mirrored nightly. Is that possible?

That might be a little tricky.

Normally you would have no problems, you are simply looking for a "Failover/Heartbeat" setup.

However in this case i am not sure a "normal" setup would work, as such a setup would involve putting both the main and the backup behind the same router.

that is; if the main fail, the backup will detect it and request the router to change it's routes so that, from the outside, the backup becomes the main.

But you are looking to change from one host to another, that is; between to servers that are not behind the same router.

So you may need to use DNS switching. meaning; if you main server is not responding, you update your DNS entry to point to the backup.

This might work if updating your DNS entry is fast.

NO what am i saying! just google "Failover/Heartbeat" i am sure there are many out there better qualified to answer this then me :)

Re:Probably a proxy box

[realityimpaired]

Could do like Yahoo and Google do...

tara@TweedleDee:~$ host google.ca
google.ca has address 74.125.226.51
google.ca has address 74.125.226.49
google.ca has address 74.125.226.48
google.ca has address 74.125.226.50
google.ca has address 74.125.226.52
google.ca mail is handled by 10 google.com.s9a1.psmtp.com.
google.ca mail is handled by 10 google.com.s9b2.psmtp.com.
google.ca mail is handled by 10 google.com.s9b1.psmtp.com.
google.ca mail is handled by 10 google.com.s9a2.psmtp.com.

Multiple servers returned by the DNS request. In Google's case, that's probably multiple load balancers, but the principle is basically the same. Set up a cron job on your main server to synchronise with the secondary servers, and have all of the secondary servers listed in the DNS request. Won't do any sort of load balancing, but it will serve as an automatic failover, because any standards-compliant system will automatically jump to another IP address in the DNS entry if the first one it tried fails.

Re:Probably a proxy box

[oreaq]

Won't do any sort of load balancing

It actually will provide load balancing also. The DNS-server changes the order of the addresses supplied in the response, choosing the order randomly or in a sequential “round robin” fashion.

Restore from backup?

[gmhowell]

Couldn't they restore their customers' sites from backup?

Re:Restore from backup?

[poity]

That's what I was wondering. What professional operation can't get customer data back from onsite or offsite backup withing the day?

Re:Restore from backup?

[mug funky]

not with half the datacentre gone, they can't.

the backup system was probably in one of the _racks_ the FBI seized.

Re:Restore from backup?

[scdeimos]

Restore to what? From what I've read DigitalOne's a co-lo customer and the FBI's taken all their physical hardware.

Re:Restore from backup?

[Michael Woodhams]

I've been around long enough to remember the Secret Service raid on Steve Jackson Games, which was the triggering event for founding the EFF.

Most companies don't have "The Feds turn up with search warrants and take all your stuff, including backup tapes" as a threat they plan for in their backup strategy. Off site backup doesn't protect against this.

I don't know what the problem is in this case - whether the backups were also seized, or that they simply lack the hardware to restore on to.

Re:Restore from backup?

[Rudolf]

the backup system was probably in one of the _racks_ the FBI seized.

No offsite backups?

Re:Restore from backup?

[alanthenerd]

This would seem to be the case as DigitalOne can't even get their own website back online.
It's pretty good going of the FBI to completely disable a companies ability to operate just to get the data of one of their clients.

Re:Restore from backup?

[SharpFang]

Of the data, yes. Of the hardware, which is currently missing, not really.

Re:Restore from backup?

[brainboyz]

Data is easy, hardware not so much.

Re:Restore from backup?

[Dahamma]

If it's just a colo, the customers may own their own servers (and be responsible for the software on them as well as backups).

If the servers were important, it's even possible they had a few for redundancy - unfortunately, redundancy is usually designed to account for simple hardware (or software) failures, and doesn't do much good when someone takes ALL of them...

Re:Restore from backup?

[Black Parrot]

Of the data, yes. Of the hardware, which is currently missing, not really.

Really? I copy my hardware to my 3-D printer every night.

Re:Restore from backup?

[gmhowell]

Restore to what? From what I've read DigitalOne's a co-lo customer and the FBI's taken all their physical hardware.

That's where you went wrong: you read the article. I didn't bother.

Re:Restore from backup?

[jd]

Unless the clients were running specialized hardware, the backup images can be thrown onto virtual machines in the interim. A dead site gathers no hits.

Re:Restore from backup?

[gmhowell]

I've been around long enough to have had a UID on that system :p

This shouldn't be much different than "a hellmouth opened up under the datacentre and swallowed it" or "the tsunami washed it out to sea" or "a stray SCUD hit the building". While ridiculous, it would seem that a visit by the FBI is about as catastrophic as some naturally occurring events that one might want to plan for. I'm not in disaster recovery, so I dunno.

I'm also curious how dodgy the customer was and if the service provider knew. (IOW, did the Feds bust an online pharmacy that was known about?)

Re:Restore from backup?

[sjames]

I'm sure the jackbooted thugs were ever so careful not to damage anything while they ripped the data center to shreds....

I'm guessing by the comment that they cannot restart their own servers that said thugs trashed whatever they didn't take. So the question is, restore the backups to what?

Re:Restore from backup?

[sjames]

Virtual machines running on what? It sounds like the FBI was ever so careful.

Let's say I bust into your house and fireaxe all of your computers at 1 A.M.. What are the odds you'll be up and running by noon?

Re:Restore from backup?

[CAIMLAS]

If the FBI has taken a full rack or more of equipment (as the article suggests), and they're a small shop, it would seem to me that a day or more is not an unreasonable recovery time.

Also, a hosting company may not actually do backups for customers, they may just 'rack and manage' on an exigent basis, leaving day-to-day to the customer.

Look, it's more than possible for a single guy to manage a half dozen racks of equipment on his own w/o much issue. Two, three guys, done right with good infrastructure, could do a couple dozen. We're not talking about anything complex, just simple single servers running an application or three. In this situation we're talking about a web hosting company, where they're constantly doing piddly 'little' things but almost always running short staffed. Switching is done by one guy/group, and the server maint by others. There is no room for 'disaster recovery in an instant' here. It'll be all up-hill, in the snow, in January, on Mars.With a higher than expected gravity.

Those same three guys are going to be hard pressed to rebuild their own infrastructure in day, too, backups or no backups. Figure it's noon before they even get chassis from Dell/IBM/HP to replace the ones stolen by the FBI that had their infrastructure on it, and then they've got to rebuild the racks, too - cabling, racking, and hardware RAID (like that doesn't take forever to perform). Considering it takes, what, 10 minutes? on some of these newer IBM servers to boot, this is hardly surprising. Add to all that the fact that their tape backup system, their disk backup system, and/or infrastructure switches may have been taken, and you've got a huge, huge headache. It takes, what, a day for two guys to simply install, cable, and rack a single rack chassis (guessing here) to all 40+ Us? And realistically, you can't have many more than 2-3 guys doing the work.

I'd be surprised if they got back up to 'fully operational' within 2-3 days. I'll be impressed if they don't go out of business.

Re:Restore from backup?

[mjwx]

not with half the datacentre gone, they can't.

the backup system was probably in one of the _racks_ the FBI seized.

That's OK, I'll just grab the latest backup we shipped off site.

More like the insurance wont cover FBI seizure. Bloody heavy handed of the FBI though, this is the sort of thing expect of a third world police force looking for bribe money, not a respected first world law enforcement agency.

Couldn't the FBI have worked with the DC's owners? Wouldn't that be more effective in rooting out LulzSec members/leaders? Oh but that wouldn't show the world that Something(TM) was being done, damn media circus.

Re:Restore from backup?

[dbIII]

Couldn't they restore their customers' sites from backup?

Onto what? They have lost three racks worth of gear. It's a bit much to expect that degree of spare capacity.

The way these raids go the backup media gets seized as well. Yet another argument for offsite backups.

Re:Restore from backup?

[AmiMoJo]

Is there any penalty for the FBI grabbing the wrong servers or causing massive disruption to innocent people?

I have always found it troublesome that law enforcement seems to be able to smash your nice front door down, take all your stuff, sit on it for a year or two for "analysis", wipe the HDDs and eventually give it back to you, and meanwhile you lose your job*... Yet there is no come back for them. No matter how badly the bungle the investigation, how much collateral damage, how much it screws up your life. I can understand the need for law enforcement to operate without fear of being liable for large sums of money, but there should also be some kind of compensation fund for the wrongly accused and innocent bystanders.

* That actually happened to the admin of the Oink BitTorrent tracker, who was eventually found innocent of all charges.

Re:Restore from backup?

[nosferatu1001]

Commodity hardware that ships in a day, or less if you pay enough to dell / hp / whoever.

Or, you know, actually have good DR with someone like Sunguard or HP, who turn up with a lorry of servers just for you....

Re:Restore from backup?

[evilviper]

A tidal wave won't simultaneously take out your backup datacenter, and off-site backups in another state. The FBI most likely will, if that information is known in advance. You may actually have to keep your off-site backups a secret to avoid them being seized in the same unnatural disaster.

Re:Restore from backup?

[icebraining]

They didn't fireaxe all of the computers, just a couple of racks. Yes, I expect an hosting company to have a few failover machines for unexpected situations, whether it's an FBI search or any other problem.

By the way, I would be perfectly able to be up and running by noon. Rent a laptop, install Debian, install packages from list, restore /etc and $HOME from online backups.

Re:Restore from backup?

[L4t3r4lu5]

So you're telling me that the principles which apply to RAID scale up well?

Shocked, I am!

Re:Restore from backup?

[L4t3r4lu5]

You want to keep stuff secret from the FBI during an investigation?

They used to call that "accessary-after-the-fact", but now it's just "accessary to X". If the FBI want to look at my gear because someone's doing something naughty on it, my response would be "Would you like a receipt with that?" Federal prison isn't my idea of a holiday resort.

Re:Restore from backup?

[tbird81]

Really? I copy my hardware to my 3-D printer every night.

According to the media companies, you've just stolen that hardware!

Re:Restore from backup?

[paulo.casanova]

From Professor Mark Stevens' page in California State University

Suing the government is the second most popular indoor sport in America, and police are often the targets of lawsuits, with over 30,000 civil actions filed against them every year, between 4-8% of them resulting in an unfavorable verdict, where the average jury award is $2 million. This isn't even counting the hundreds of cases settled thru out-of-court settlements, which probably runs in the hundreds of millions and involves about half of all cases filed. It may take up to five years to settle a police liability case.

Re:Restore from backup?

[petermgreen]

I'd think you could set up a rack chassis very quickly if you just stacked the servers in and didn't bother to bolt most of them into the rack (just enough that it didn't collapse on you) and didn't bother with fancy cabling (just throw in a powe and network cord to each machine). Have someone in an office cloning hard drives to drop into the rigs.

No it wouldn't be pretty, no you wouldn't have hardware raid but i'd think you could get a lot of capacity up pretty quickly this way a bigger problem is likely to be actually getting and paying for the hardware.

Re:Restore from backup?

[petermgreen]

Most companies don't have "The Feds turn up with search warrants and take all your stuff, including backup tapes" as a threat they plan for in their backup strategy. Off site backup doesn't protect against this.

Short of backups accross multiple continenets and a plan for fleeing the country even if the cops don't want you to leave there isn't much you can do against a determined seziure action directed at you.

However this story does indicate some things you can do to reduce the affect on you of the collatoral damage of an action like the one in TFA. Many of those actions are things that also help to mitigate other risks

For customers:

1: Rent directly from the datacenter. Renting from an intermediary (especially a small one)increases the chance of the agents being unable to accurately identify what they want and therefore taking a lot more and also increases the risk of the provider having trouble providing relief for it's customers.
2: If your requirements justify it rent whole racks so you don't have other peoples stuff sharing a rack with you.
3: Assume that loss of all stuff in company X's datacenters is a reasonable risk. Mitigate by using a different companies datacenters for each

For providers:

1: have a presense in multiple datacenters.
2: spread your critical infrastructure in a DC accross multiple racks, making sure those racks have no common customers. Obviously this is only possible when you have more than a couple of racks.

Re:Restore from backup?

[Qzukk]

Yet another argument for offsite backups.

Yet another argument for coordinated raids. Why steal just three racks of servers when you can have your guys steal three more from the other side of the country?

Re:Restore from backup?

[Techie_79]

Unfortunately, I agree. They'll be lucky to stay in business. And so, an overzealous FBI drags down small-business America (yet again) in their pursuit of internet security.

Re:Restore from backup?

[Bardwick]

That's the bummer. What do you restore it to? If they took entire UCS/HP/IBM chassis, might be a bit before they can get new hardware shipped in. Running 30-40 VM's per blade isn't unheard of. Even if they wanted 1 virtual server and took that hardware it ran on, they could have potentially downed hundreds of VM's.

Re:Restore from backup?

[jedidiah]

"The FBI steals 2 racks of machines" was probably not something they considered in their disaster recovery plan.

Re:Restore from backup?

[SecurityGuy]

I can understand the need for law enforcement to operate without fear of being liable for large sums of money

I find that really disturbing. If you or I cause significant harm to someone, we face civil liability or jail time. A group of people who are given greater power than the common citizen to cause significant harm to someone should face GREATER scrutiny and penalties to prevent abuse of that power.

Re:Restore from backup?

[green1]

If you are the true target of the FBI raid, you probably have bigger problems than the lost data, and in fact restoring the data may in fact cause you to be in even more legal trouble as they likely seized the servers specifically to stop you from continuing what you were doing. If you really do want to be immune to this sort of raid I would recommend your off site backups be stored in a different country than your primary server, and ideally in a jurisdiction with little history of cooperation with American law enforcement agencies (you probably want backup copies of your staff in those foreign countries too, as the primary staff members were likely "seized" as well)

If however you are simply part of the collateral damage of an FBI raid (you are one of 10 clients on a rack, not the one being investigated, but the FBI took the whole rack anyway) then the odds of your off site backups being seized in the same action are rather slim (unless you somehow managed to choose the same off site storage location for your backups as the other company did, in which case you really are insanely unlucky!) So in respect to the innocent companies that were taken down through no fault of their own, any disaster recovery plan that would cover the data centre burning to the ground/washing out to sea/being hit by a hijacked jetliner/etc should be quite sufficient to restore their services.

The bigger problem is that if a fire had destroyed their server, their insurance company would probably help with costs, an FBI raid (even if not directed at you) is likely not covered by insurance, leaving these people in worse shape financially, even if not technically.

Re:Restore from backup?

[ultranova]

Bloody heavy handed of the FBI though, this is the sort of thing expect of a third world police force looking for bribe money, not a respected first world law enforcement agency.

Draw from that what conclusions you will.

Re:Restore from backup?

[couchslug]

What is this "backup" of which you speak?

Re:Restore from backup?

[alta]

And you assume a level of service that includes the company doing their backups. Many hosting providers offer this as an option. Particularly when it's not shared hosting. When I was using offsite dedicated hosting, I didn't use their backups. I backed it up myself to my office.

Re:Restore from backup?

[icebraining]

Does your house insurance (assuming you have one) include protection against
* Fire, started by neighbor on the east of the house
* Fire, started by neighbor on the west of the house
* Fire, started by random person crashing a car into the house
* Fire, started by a leak of gas left open by inhabitant A
(etc)
?

Now it was the FBI, tomorrow it could be any other problem. The specifics how the two racks failed are irrelevant - they need to account for a failure, regardless of how it happens.

Re:Restore from backup?

[AmiMoJo]

I should explain exactly what I mean. Law enforcement should not be put off arresting someone with a high paying job for fear of having to cover their lost wages when they are found innocent. There should be compensation, it should just not be a factor in the police's decision to take action against someone.

Re:Restore from backup?

[sjames]

According to TFA, they not only took 2 racks away, they apparently damaged the machines they left behind such that they wouldn't boot.

So they took 2 racks and fireaxed the rest.

Re:Restore from backup?

[marcosdumay]

It may even have protection against a thermonuclear bomb exploding in the middle of the city, but I doubt the insurer will be able to pay if that happens.

Re:Restore from backup?

[Cramer]

If their "proper records" are within the colo (i.e. in the blade center management app(s)), then it's entirely possible to not know jack with all the systems in disarray. All you'd have to do is take my workstation to make me lose track of what's where. (even that's not current :-))

Re:Restore from backup?

[jwhitener]

Assuming they only had the hardware on hand to run the exact amount of servers they were hosting, then yeah, setting up the hardware alone is going to take time.

But with modern virtual machines (I use ZFS and VMware) and a some extra processing power and disk space around, there isn't any excuse for downtime longer than half a day. You can take hourly or nightly snapshots of running virtual machines, and send those snaps to other servers or offsite even. That whole rack of virtual machines blows up, you pull the snaps back onto spare room on another rack, and presto, everything is working again.

I've done this multiple times with pretty complex architecture. Not as disaster recovery, but to install newer, more powerful hardware. Set up the new hardware, "zfs send snapshot1 > new server X", "zoneadm -z snapshot1 boot", done. I work for a non-profit with a pretty tight budget, and even we have enough extra computing power around to double up virtual zones for most of our applications. Not to mention, an entirely offsite data center (rented space) that our servers sync to in case our main data center is literally taken out (earthquake or something).

Re:Restore from backup?

[jwhitener]

A rented offsite data center would protect against them physically taking your main data center though. My non-profit is doing that now, and we aren't exactly rich. Your main data center, everything running virtual servers, periodically sends vm copies to the offsite data center. It isn't very complex.

Now, if the FBI for some reason wanted to confiscate the rented data center copies as well.....not sure why they would though.

Re:Restore from backup?

[evilviper]

If you are the true target of the FBI raid, you probably have bigger problems than the lost data, and in fact restoring the data may in fact cause you to be in even more legal trouble as they likely seized the servers specifically to stop you from continuing what you were doing.

On the contrary. You assume the FBI WAS RIGHT to target you. In fact with their near immunity from prosecution, the FBI screws-up plenty, and doesn't care.

Try the founding EFF case as the canonical exception to everything you just said: http://www.eff.org/about/history

False accusations of copyright infringement, DRM circumvention, leaked top-secret documents, etc. All things that could make you a target, yet perfectly legal, irreplaceable, and important to get restored as quickly as possible.

Re:Restore from backup?

[Kalriath]

Unmanaged hosting companies don't keep backups (technically, they can't - they have no access to your server beyond the physical). That's your problem. According to their SLA (the Google cache of it anyway) they are unmanaged, and do not keep backups.

Re:Restore from backup?

[nagnamer]

Really? I copy my hardware to my 3-D printer every night.

According to the media companies, you've just stolen that hardware!

You are permitted to make one (1) copy of the said hardware without express written permission, for backup purposes only, and provided that it is not used at the same time as the original.

Re:Restore from backup?

[green1]

I never said they were right. Only that you will have bigger problems than restoring backups, and that doing so will likely cause them to get you in further trouble specifically because, as you say, they don't care, and they seldom admit they are wrong.

I also stated how to get around that particular problem further down in my post.

Re:Restore from backup?

[evilviper]

I never said they were right. Only that you will have bigger problems than restoring backups,

Steve Jackson's biggest problem was indeed that he didn't have a backup of his manuscript, and had to lay off half his staff and nearly went out of business because of it (missed deadlines).

His mix-up with the FBI was not a "bigger problem" than his lack of backups, it was indeed a far smaller problem.

and that doing so will likely cause them to get you in further trouble

His manuscript was never the object of the seizure, and of course not at all illegal, or even on shaky ground. He did not risk any further trouble, he just didn't have a "backup", and could not get the FBI to return it.

The FBI should try that on cloud hosting

[initialE]

1. Take the servers
2. There is nothing on the servers - take the Storage
3. The storage is remotely replicated - pull the remote storage
4. You can't pull the remote storage, you don't have jurisdiction overseas

Re:The FBI should try that on cloud hosting

[TooMuchToDo]

THIS! Although, you have to be careful. If your storage is outside of US jurisdiction (Amazon S3 Asia/EU AZs), but the company is still a US company for the most part (for this example, Amazon), it's very likely LEO will get the data they're looking for. Take into account the people who run whatever equipment/storage systems you're using outside of jurisdictions you're working against.

Re:The FBI should try that on cloud hosting

[jd]

The FBI can't seize it, but due to crap security, apparently everyone else can. Hmmm. Not a great swap.

Re:The FBI should try that on cloud hosting

[Swampash]

It's ok, they backed up everything to S3 using Dropbox, and Dropbox has a new feature where you can log in to any account with any or no password.

Re:The FBI should try that on cloud hosting

[TaoPhoenix]

"Hi Amazon! Here's a 42 wheel truck. All your servers are belong to us in one click. You have no chance to survive. Ha Ha Ha. Ha Ha. (Duet between Zero Wing and Nelson.)

Re:The FBI should try that on cloud hosting

[whiteboy86]

This would be devastating for normal cloud hosting, because customers have data spread over lots of hardware, that means, if thay grab an entire rack, they break business for pretty much everybody. The 'remote storage' might be applicable for Amazon but hardly any other local hoster. Another confirmation that dedicated (not shared) PC with your own IP and your very own solo server is your best bet. At least you can be sure that you are not collocated with some blackhats.

Re:The FBI should try that on cloud hosting

[Junta]

3. The storage is remotely replicated - pull the remote storage
4. You can't pull the remote storage, you don't have jurisdiction overseas

I don't see why they'd care about seizing all copies of the data. They're looking for evidence, not making sure all copies of the data are destroyed. Even one full copy is sufficient for evidence.

I'd wager that for business continuity, large cloud hosting providers all have a process to export data to comply with a warrant without having to give up a bit of production hardware. There is always the chance they think the provider is complicit or incompetent and take hardware anyway, but I'd be surprised if they tried to take the hardware hosting data from a large provider.

Re:The FBI should try that on cloud hosting

[Chris Mattern]

5. Have the manager of the data center--who is right here, surrounded by our agents--pull the remote storage. Yes, he has assured me he'll be cooperative.

Re:The FBI should try that on cloud hosting

[black soap]

Those copies/backups are what the FBI likes to call "more evidence." They like to get all the evidence they know about the existence of. Anything less weakens the case.

Re:The FBI should try that on cloud hosting

[mounthood]

5. Senator makes a phone call and Amazon does whatever the FBI wants

Centralizing service providers and making them bigger only serves those in power better. Monopolies play the game willingly.

Solution

[PPH]

Host offshore.

Re:Solution

[countertrolling]

Name one place that won't do exactly the same thing...

Re:Solution

[Anonymous Coward]

Keep random chunks of your data in several countries offshore so that it would be almost impossible to get together the hundred subpoenas needed to gather what you want before the data owner gets wind of it and pulls the data offline.

Re:Solution

[TooMuchToDo]

The hosting company I co-own with the rest of my employees is mid-sized (several million a year, but under 10 people), but we operate this way. Equipment is owned by corporations incorporated in the jurisdiction where it resides on a country-level basis. We own gear in the US, the EU, Japan, China, and Australia. No corporate entity is tied to another, and resources are redundant through the infrastructure. Come to me in the US with a subpoena for anything on any of our gear outside the US? Fark off. When the hell did people give up on their principles?

Re:Solution

[tomthepom]

DigitalOne is based in Switzerland, they did host offshore in the US. That might have been a mistake.

Re:Solution

[hawkinspeter]

The Principality of Sealand?

Re:Solution

[biodata]

I heard Iceland are putting in a legal framework to support free speech online that would exactly stop them from doing any such thing. They are wily vikings and can see that every business is going to need reliable hosting in the future, free from this kind of interference. America is starting to look like a worse and worse place to keep data.

Re:Solution

[Threni]

I'm in the UK - the US is offshore, although I'm not sure I'd consider them now if the goverment bodies can flash a badge/gun and start ripping out boxes. If you're happy that this sort of behaviour is the 'cost of doing business' you might as well go even cheaper and host in Iran or something.

Re:Solution

[rgviza]

or don't use a vhost,  rent a whole rack or colo one. If this disruption ends up costing you as much as renting a whole rack, it's worth it, if for nothing else, so your customers don't lose faith in you and your reputation for reliability doesn't take a hit. Don't use the cloud; here's a perfect example of one reason why the cloud is a bad idea for anything that's mission critical (proprietary lock-in being another damned good reason)

Sounds like someone in a vhost/cloud was doing something illegal so they took the whole rack of equipment.

Re:Solution

[Fnord666]

So how can I contact you to get hosting with your company?

Re:Solution

[TooMuchToDo]

For those that request and pay for it, yes. Note: None of our clients (to my knowledge based on the due diligence we do when on-boarding them) perform businesses that are illegal in the United States. I'm not for breaking the law. I *am* for enforcing your own rights.

Re:Solution

[countertrolling]

Of course all that depends on the other jurisdictions also telling the US to fark off.. which doesn't happen very often.. Usually they are more than happy to cooperate.. They might not knock you offline, but they will get what they are looking for.. The authorities can interpret the law any way they wish and have the weaponry to make you an offer you can't refuse. Networking needs to be much more 'ad-hoc' and less dependent on corporate service providers if we want serious protection against this. A hosting company is a single point of failure... oops! So sorry.. we thought you were someone else..... Of course they could always blame that on the 'terrorists'

Civil and criminal liability

[dgatwood]

I think it's time to hold the FBI to the same standards that they would hold the rest of us. If I went in waving a gun around and demanding to walk away with somebody else's server, they'd throw my ass in jail.

If they want access to a particular client's content, they can go through the same process as a DMCA takedown request or a backup request would. They make a request, the company yanks that customer's access, then clones that customer's data onto a new drive, then hands them the drive.

As far as I'm concerned, every single client of this ISP ought to sue the FBI for the damage they caused—for the downtime, for the loss of data, for the time spent trying to reach the ISP to figure out what was going on, for the cost of any failover hardware or service that they had to pay for in lieu of that service, etc. If the FBI had to pay out a few million dollar settlements every time they pulled a stunt like this, they'd think twice about acting like a bunch of thugs, and they would go through proper channels and do their investigation in a way that doesn't cause collateral damage.

There's simply no excuse for such sloppy investigative work. If they screwed up so royally with the servers, you have to wonder how many grievous errors they made in other areas that would lead to the evidence being declared tainted, criminals going free, etc.

Re:Civil and criminal liability

[phantomfive]

They make a request, the company yanks that customer's access, then clones that customer's data onto a new drive, then hands them the drive.

Oh yeah, that'll be real great. Then the mafia guys the FBI is chasing get a tip off because they are the ones who own the datacenter. Not saying I like how it is, but your plan has serious holes.

Re:Civil and criminal liability

[icebike]

You can try to file a suit, but you probably wouldn't get anywhere.

The Federal Tort Claims Act was enacted by Congress in 1946 to allow citizens to sue the federal government. Prior to that you had to get something
passed by congress in order to sue the government.

From http://www.finchmccranie.com/refresher.htm

While the passage of the FTCA constitutes a limited waiver of sovereign immunity, Congress specifically limited the government's amenability to suit in a variety of different circumstances. In 28 U.S.C. 2680, Congress specified that its limited waiver of immunity would not apply to the following claims:

(a) any claim based upon an act or omission of an employee of the government, exercising due care, in the execution of a statute or regulation, whether or not such statute or regulation be valid, or based upon the exercise of performance or the failure to exercise or perform a discretionary function or duty on the part of a federal agency or an employee of the government, whether or not the dis- cretion involved be abused; ...

So you see, you are effectively shut down before you get to the courthouse steps. All they need do is say "We had evidence that all servers we took were involved" and there is nothing more you can do. You will not be granted the ability to examine that evidence.

Re:Civil and criminal liability

[corbettw]

The work of cloning the data could either be supervised or actually performed by trained FBI agents (from a chain-of-evidence point-of-view, the latter would be preferable). But before going in, they should have at least as much information as is needed to know which servers and which clients on those servers they need. Fishing expeditions like this one need to stop.

Re:Civil and criminal liability

[denbesten]

If you were an impacted client, your next course of action is to file a claim against your business interruption insurance and to locate your off-site backup tapes.

Given the fact that defense attorneys exist, my bet is that the FBI went through the proper channels (getting a warrant) and followed well-established procedures that are geared towards preventing the destruction of evidence and towards maintaining the chain of evidence.

Attempting to sue the FBI would likely result in a lecture on sovereign immunity. Even if a miracle happened and you were to prevail, the settlement money would really just comes from the taxpayers, which would have the effect of punishing you and me, not the government employees (other than the fact that they too are taxpayers :-).

Re:Civil and criminal liability

[hairyfeet]

And you MR AC just hit the nail on the head as to what is wrong with the current laws regarding computer seizure. these laws were written when a server was weak enough it held ONE client and one client only, so if you seized the machine you were seizing only what you were supposed to. The problem is now servers are the multicore monsters with more than enough horse to host dozens of virtual servers all of which belong to someone else and the FBI has NO right to mess with AT ALL.

So the feds really do need to get slapped down over this. this is like knowing someone on a city block is dealing dope so you kick down the doors of every house and the tear through their stuff looking for proof. they have the right to seize that ONE server that was named on the warrant and NOT every server in the colo.

I hope all those that have downtime because of this grab a lawyer and hand the feds with a really fat lawsuit, as that is the only way these clowns will learn it isn't 1986 anymore and servers don't work that way.

Re:Civil and criminal liability

[phantomfive]

It's not easy. You have to be able to figure it out on-site, because the owner cannot be considered trustworthy (although in most cases he/she probably is). How do you ensure you only get the correct computers, and also avoid giving notice to the criminals in time to hide the evidence? Obviously the FBI is mainly interested in the latter, but there needs to be balance.

Re:Civil and criminal liability

[i.am.delf]

I wonder is the FBI could subpoena a critical control system say a Siemen's SCADA controller that had been hacked. If this control system were used to control a machine capable of causing grievous bodily harm or death, would the FBI not be negligent? If the FBI took a server legitimately housing an e-commerce site containing customer data, would that be considered a data breach under California law?(FTCA torts are determined under state law not under Federal)

The FTCA specifically allows claims based upon negligence to be brought against the Federal government. However, you are correct that this liability is limited by the exemption you have posted. I don't think that any reasonable person could suggest that if data were stored on a server at Google and distributed over an entire datacenter, that the entire datacenter could be seized. That exemption gives Federal employees the leeway to search a house they reasonably believe contains a fleeing suspect. It does not cover something like seizing all the cars on a block because one contained drugs. No reasonable person would suggest such a ludicrous argument in that case nor should anyone suggest that since these 100 or so servers are in close proximity in a data center they may all be seized.

Re:Civil and criminal liability

[X0563511]

I suppose you could compare this to the FBI "seizing" a whole office park in one grab. It is kind of extreme.

Re:Civil and criminal liability

[arkenian]

IANAL. I agree with your first sentence, the laws were written back when server models were different. That said:

Basically the issue here is that a warrant is issued for the FBI to be able to take a piece of hardware which was probably used for criminal activity back to their lab. This makes a lot of sense, in some ways. If the server isn't offline and in the FBI's possession, how do they establish chain-of-custody?? Several techniques for restoring deleted data more or less require physical access, (and certainly require that other servers not write to them) etc. etc. Just cloning a VM may not establish evidentiary custody, and also may not allow full forensic analysis.

On the other hand, as you point out, modern data centers don't work that way. As we move to the cloud, load-balanced data centers, multiple-client hosting services etc. this can cause unreasonable impact. Not sure what the solution would be. My gut feel is that in some cases seizing hardware merely isn't practical . . . their warrant sweeps up more that isn't relevant than is etc. OR they need to furnish replacement hardware, OR . . . but really, an entirely new area of law needs to be developed. This is one case where "on a computer" or "on the internet" actually DOES change the rules. The FBI doesn't have any CHOICE but to seize hardware, because otherwise the evidence is not, currently, usable in court... this isn't just about seizure rules, but about evidentiary standards for digital data, etc. etc. etc.

Re:Civil and criminal liability

[sjames]

They can still grab the relevant systems, they would just think twice about causing unnecessary damage in the process and about indiscriminately taking other people's stuff.

Re:Civil and criminal liability

[phantomfive]

That is a lot better idea than the GP's

Re:Civil and criminal liability

[sjames]

They rendered even the servers they DIDN'T take unbootable. That doesn't sound like due care. They had the opportunity to have employees of the colo (who were not under investigation) which machines belonged to the party named in the warrant, but they failed to do so. Again, no care at all, much less due care.

Re:Civil and criminal liability

[Alain Williams]

If the FBI had to pay out a few million dollar settlements every time they pulled a stunt like this, they'd think twice about acting like a bunch of thugs,

No, it would just come out of government budget, the individuals in the FBI suffer no penalty for causing avoidable mayhem. The change their behaviour you need to make the individuals pay -- by the time a few of them have lost their homes, they may start to act repsonsibly.

Re:Civil and criminal liability

[Moskit]

> They make a request, the company yanks that customer's access, then clones that customer's data onto a new drive, then hands them the drive.

Depends on legislation. In some countries only the original drive is considered evidence, therefore it cannot be returned until the whole process is over (think years). Copy cannot be made and returned either, for some other reasons (can't recall exactly).

Re:Civil and criminal liability

[Qzukk]

they probably avoided a number of grievous errors

Like what? "We took the wrong server and had to turn around and go back" so they just take everyone's server just to be sure? It's bad enough that they shoot granny when they do a drug bust at the wrong address, now they should shoot every granny on the block to make sure they got the right one?

Their warrant was for specific servers, they requested the information on which servers to take, they were given the information on which servers to take, they ignored it and took everything.

On the other hand you probably build a heck of a server and perform backups with the best of them.

You take backups? Uhoh, that might be evidence. Better seize all the backups too.

Re:Civil and criminal liability

[Wrath0fb0b]

I think it's time to hold the FBI to the same standards that they would hold the rest of us. If I went in waving a gun around and demanding to walk away with somebody else's server, they'd throw my ass in jail.

Yeah, because you didn't go to a neutral magistrate and get a search warrant based on a sworn affidavit. There's a difference between saying the FBI should follow the law (and, if there's any indication they didn't get a warrant or the application was defective, it's possible but not mentioned in TFA) and saying that there should effectively be no law enforcement on private property at all.

Heck, the local police here barge into private homes with guns fairly often. About 50% of calls to a home (I worked a bit of IT there a while back and would peruse the logs) were because some guy decided to whale on his wife a little bit or (rarely) vice versa. You wouldn't seriously say that they should face the same standards that would hold me if I barged into someone's house with a gun right?

If they want access to a particular client's content, they can go through the same process as a DMCA takedown request or a backup request would. They make a request, the company yanks that customer's access, then clones that customer's data onto a new drive, then hands them the drive.

Evidence in criminal investigations does not need to be seized voluntarily, that's the entire point of a warrant. What's more, your proposed scheme puts them at the mercy of a hosting company that might be conspiring with the suspects. There's no reason to trust the hosting company not to wipe the actual customer disks and give you some random junk instead.

What's more, if there is incriminating evidence on there and it comes to trial and some FBI agent has to testify about the chain of evidence for some particular files found on a hard drive, defense counsel is going to rip him to shreds if anyone other than a sworn officer or a federal employee so much as touched that drive. "The hosting company cloned it for me" isn't remotely going to fly -- heck it probably won't even get to the jury because it's so lacking in the indicia of reliability.

Re:Civil and criminal liability

[jaypifer]

Don't ever sue someone who has unlimited access to free attorneys.

Re:Civil and criminal liability

[Karl Cocknozzle]

Instead of suing for liability on the business loss, (which their own insurance should be covering anyway) there's nothing stopping the owners of the companies affected from suing the FBI and agents involved on 4th Amendment grounds, since the FBI seized their servers without a warrant. To be clear: A warrant existed, but it wasn't for the most of the items taken: It listed equipment belonging to a specific company. But they indiscriminately seized whole racks, so some of the servers in the rack (not to mention the rack itself, PDUs, switches, storage) were taken without a warrant.

Re:Civil and criminal liability

[Khyber]

"Surprisingly enough, you don't get to decide what they can take."

You say "That isn't listed in the warrant, any attempt to take that will result in me arresting you for theft."

And you have every right to do it.

Re:Civil and criminal liability

[black soap]

Or maybe shutting down a phone company every time they go in to get some records they have a warrant for.

Re:Civil and criminal liability

[b4dc0d3r]

I agree, but you're horribly misguided. FBI does not know who is doing what, or which servers host which data. They are not going to send a letter looking for the hosted clients' data, since the host could be the one with the website they are after.

In other words, the host is a suspect, and until they figure all that out, everyone hosted there is a suspect as well. Except that they don't care who is a suspect, they just know they want the data, so everyone's data goes out the door.

You have a warrant to sieze information, then you don't know what information to request specifically (racks 1-2 out of 4 is not something you can know in advance).

Yes it is disruptive, so do yourself a favor when getting a hosting company. Look for one that specifically can restore within a reasonable timeframe from offsite backup, or hosts in multiple companies. That's the only way out of this. Paying the lowest price possible means you're wide open to this, and filing a lawsuit against the government when your small business is dead is not going to help.

Re:Civil and criminal liability

[dgatwood]

No, they can't. That's the whole point. As soon as you "grab" a system, you're taking down tens or even hundreds of different websites, each of which is a separate commercial company, each of which has the potential to lose significant business as a result. It is the equivalent of closing and seizing a shopping mall because some drug dealer is selling drugs in the parking lot. Property seizure is utterly ridiculous in this case, and whether they return it intact after a week or not is really not relevant.

And if we're talking about a colossal setup with distributed virtual data volumes spread across multiple boxes, physically seizing the data might be closer to bombing an entire city because of one drug dealer who sold drugs in the parking lot last week.

There is simply no way to "grab" a server that does not cause huge amounts of collateral damage to large numbers of other people/companies. It is simply not possible to do so. Therefore, locking out that customer and cloning the system in place (including that customer's data and any unallocated blocks that might have previously belonged to the customer, but not including anyone else's data files) is the only legitimate solution to the problem. Period. Anything more than that cannot be done without collateral damage, and the FBI should be liable for any collateral damage that they cause.

Re:Civil and criminal liability

[rahvin112]

You don't SUE the government. You sue the jackass working for the government for negligence. The Government will then be forced to step in and defend the employee and can't use the sovereign immunity defense.

Re:Civil and criminal liability

[dgatwood]

Yeah, because you didn't go to a neutral magistrate and get a search warrant based on a sworn affidavit.

Even if I have appropriate legal authority to repossess a car, I can still be held criminally liable if I do so with someone's child in the back seat. This is basically the equivalent, only with several thousand people's children in the back seat.

Evidence in criminal investigations does not need to be seized voluntarily, that's the entire point of a warrant. What's more, your proposed scheme puts them at the mercy of a hosting company that might be conspiring with the suspects. There's no reason to trust the hosting company not to wipe the actual customer disks and give you some random junk instead.

This is why the FBI agents involved should be proper computer specialists who are trained in how to clone the data, and should perform the cloning themselves. What you're giving isn't an argument. It's an excuse for lack of training.

Re:Civil and criminal liability

[sjames]

In those cases, they'll need to just make a copy of the relevant data and call it good. They shouldn't take those servers anyway because they are not owned by the suspects.

Re:Civil and criminal liability

[hedwards]

Not that it makes it OK, but where's the facilities responsibility in all this? It seems to me that somebody should have been keeping backups, granted I'm not sure that the service was responsible, but it's just amateurish to have to go and say that you can't get your own server back up because you haven't got backups.

If I were hosting my site with them, I'd be looking for a new place to house my site pronto.

Re:Civil and criminal liability

[dgatwood]

FBI does not know who is doing what, or which servers host which data. They are not going to send a letter looking for the hosted clients' data, since the host could be the one with the website they are after.

A modicum of common sense comes into play here. 99.999% of the time, a hosting provider is just that. We really shouldn't have millions of dollars in collateral damage every few weeks just for an edge case of an edge case. We should have a different warrant process for cases where the police suspect that the hosting provider isn't a legitimate business, and there should be liability involved if you abuse that process for a provider that obviously is legitimate.

Look for one that specifically can restore within a reasonable timeframe from offsite backup, or hosts in multiple companies.

Another poster made a comment that pretty much covers this. It went something like this: "Oh, look. The hosting provider keeps backups offsite. We'd better simultaneously raid the data warehouse because that might be evidence, too."

You can't win. The only winning move is not to play. This means either A. host everything yourself, B. massively overpay to design a replicated hosting system from two different hosting providers, or C. force the FBI to use some common sense. We're not talking about Fortune 500 companies here. These are small firms that can't afford the sort of replicated hosting that would be required to protect against this sort of event.

Does that mean that equal protection under the law and the right to be secure against unwarranted searches and seizures should apply only to people and companies with enough money to secure themselves? Because that's really the only way to interpret your suggestion, and it's not consistent with the Constitution that I grew up believing in.

You have a warrant to sieze information, then you don't know what information to request specifically (racks 1-2 out of 4 is not something you can know in advance).

Again, you know who the client is. You already had to determine which server(s) the user's data was on (and trust that the hosting provider gave you the true information) because no judge is going to give you a warrant to seize the entire data center, and if you try it anyway, you're committing grand larceny. How much of a stretch is it, then, to require that they take that one extra step to seize an account instead of an entire box? This isn't about rules of evidence or any such bullshit. This is about not having the proper training, not understanding the collateral damage that they can cause, and gross criminal negligence in data acquisition. Nothing more, nothing less.

Re:Civil and criminal liability

[sjames]

We don't know if they have backups or not. We do know that they have no working hardware to load them on.

They just said unbootable. If I kick your server around the datacenter enough, no amount of backups will make it bootable.

Re:Civil and criminal liability

[icebike]

Actually the terms of the warrant are not in evidence anywhere I can find.
We don't even know upon WHOM the warrant was served. If it was served on the colo site,
then chances are ALL the servers physically belong to them, and they lease them to customers.
Its also possible many of these customers were actually hosted on virtual hardware.

We've only seen one side of this issue so far.

I would fully expect it to say things like "any or all" equipment used in furtherance of....
It then becomes up to you prove that your web server was NOT hacked and used...

Re:Civil and criminal liability

[icebike]

The government won't step in.
They will simply point to FTCA exemption (a), and your suit is tossed with prejudice. One letter from one junior assistant attorney general.
You can't sue a federal employee doing his job. He had a warrant. He had orders. He acted in good faith, even if misguided.
Did you not pay any attention in JR. High civics?

Re:Civil and criminal liability

[corbettw]

I completely agree that the situation is delicate and there is almost certainly not an easy answer. No one wants to see criminals get away with breaking the law, but neither should innocent third parties be subject to having their entire disrupted so completely.

Maybe the only real effective solution is for businesses to pay the extra money for dedicated servers, and not just have a shared hosting arrangement with hundreds or thousands of sites on one server. Yes, that increases their costs, but at least then it would be feasible for law enforcement to seize only those servers directly associated with the domains under investigation.

Re:Civil and criminal liability

[icebike]

The single company named was the Hosting company, which RENTED computers to a lot of other people.
As such, you can see they exercised a lot of restraint, by not taking ALL the computers.

They did not destroy the entire data center. You made that up.

Re:Civil and criminal liability

[hairyfeet]

Then they should just yank the drive of the offending server (while giving the company hosting time to migrate the other VMs while taking the offending one offline) so that the company can restore service to those that are not subject to the warrant and the feds can go on their way.

But even that I believe could raise issues, such as "fruit of the poisoned tree'. like what if they find out while going through a hard drive looking for evidence against company B that company C has been doing something illegal? Legally there should be nothing they can do but we know that isn't the way it works IRL.

In the end what I think is gonna have to happen is to have a clear standard for making a bit for bit copy of VMs and other electronic data, so that it can be verified and work for the chain of evidence without taking down entire colos just to serve a warrant. It would have to be FOSS because even though I see nothing wrong with proprietary per se, when someone's life is on the line we are gonna want to be able to see the source and ensure it is doing what they claim.

Sadly IRL what is gonna happen is more and more are simply gonna avoid hosting in the USA, and that giant sucking sound of jobs leaving to never return is gonna slam dead bang right into the colos, and who can blame them?

Re:Civil and criminal liability

[arkenian]

In the end what I think is gonna have to happen is to have a clear standard for making a bit for bit copy of VMs and other electronic data, so that it can be verified and work for the chain of evidence without taking down entire colos just to serve a warrant. It would have to be FOSS because even though I see nothing wrong with proprietary per se, when someone's life is on the line we are gonna want to be able to see the source and ensure it is doing what they claim.

That's more or less what I was trying (badly) to advocate in my post, yes.

Re:Civil and criminal liability

[Wrath0fb0b]

Even if I have appropriate legal authority to repossess a car, I can still be held criminally liable if I do so with someone's child in the back seat. This is basically the equivalent, only with several thousand people's children in the back seat.

In the sense that a child is equivalent to an inanimate object, sure.

This is why the FBI agents involved should be proper computer specialists who are trained in how to clone the data, and should perform the cloning themselves. What you're giving isn't an argument. It's an excuse for lack of training.

Which is exactly what they do -- seized computers are cloned and read-only copies made in a controlled facility in a well-documented way that preserves the chain of evidence for trial.

Re:Civil and criminal liability

[Kalriath]

It would have to be FOSS because even though I see nothing wrong with proprietary per se, when someone's life is on the line we are gonna want to be able to see the source and ensure it is doing what they claim.

Why? None of the existing forensic tools are FOSS. Why does it suddenly matter with this one?

Machines won't be coming back

[Anonymous Coward]

Every time I hear this story, and in the one situation where I witnessed federal agents confiscating equipment, the equipment never returns.

Re:Machines won't be coming back

[countertrolling]

They can buy it back at the auction... Probably at a pretty good price

Re:Machines won't be coming back

[X0563511]

Which is bullshit.

The equipment needs to be kept until guilt or innocence is determined. At that point, any equipment belonging to an innocent needs to be fucking returned.

It's larceny otherwise. Can't understand how they get away with this...

It's not even like I'm saying compensation should be issued! Just an "our bad, here's your stuff!"

Re:Machines won't be coming back

[countertrolling]

Of course it's bullshit, but when nobody fights back, it doesn't matter... The authorities get away with this because people accept it.. Nobody will challenge authority when it means risking what little power they have over their own little corner of the their world.. It's what you call a monkey trap

Does the Constitution still mean anything?

[mykos]

Each of the clients who had their property seized without warrant should bring suit.

Re:Does the Constitution still mean anything?

[DavidRawling]

Note being a USAnian, I am guessing here - but ISTR there's a law preventing you from suing the government? Basically - immunity from prosecution unless the government (dept) agrees to be sued, or something like that. And I always think, hearing something like that, the argument would be something like, "It's not in the national/public interest for you to be sue us, so no. Neener neener neener."

Re:Does the Constitution still mean anything?

[phantomfive]

You can sue the government but the rules are different than suing private parties, because the government is different. There is a different set of laws that apply to the government (for example, a private party would not be able to request a warrant to seize someone else's equipment, no matter how much kiddie porn it has). Uh, YMMV if someone manages to steal your your computer because you have kiddie porn on it don't blame me

Re:Does the Constitution still mean anything?

[icebike]

Responding to your title, "Does the constitution still mean anything", the answer is NO.

Just about here is where I get jumped on by everybody who supports the Constitution and hold it dear. Who doesn't?

But the point is, nothing written in the constitution means anything any more, and hasn't for a long time.
Every sentence and every clause has been violated and circumvented by a web of laws and rulings such that any citizen who points to the constitution in his defense is laughed out of court. In the legal profession, an appeal to the constitution is a huge inside joke. The sign of a rube. A target to be fleeced.

Re:Does the Constitution still mean anything?

[SharpFang]

One of the caveats is that government has to consent to be sued. Yes, they can say "we do not agree for this lawsuit" and the result is "case dismissed."

Re:Does the Constitution still mean anything?

[CodeBuster]

The sign of a rube. A target to be fleeced.

Even lawyers must choose their targets with some care. Filing a lawsuit against the "wrong" people can result in an "out of court settlement". You can use your own imagination as to what constitutes an "out of court settlement" in that context...

Re:Does the Constitution still mean anything?

[DarkTempes]

You can definitely sue the US government. Separation of powers and such say, in theory, that the FBI/executive branch people can't just make it disappear. From my understanding you are sort of correct in that the executive branch can ask the judicial to not hear the case in the best interest of the nation.

Remember when ICE took down all those websites via domain seizures? Some of those companies are suing over it. I dunno if it'll actually go anywhere but I believe they weren't thrown right out. Lots of important historical changes have happened in the courts in party vs state/government/whatever...

Re:Does the Constitution still mean anything?

[X0563511]

At which you talk to your judicial (appeal) - and if that can't/won't work, the legislature. Congress can pretty much do what they want, if you can convince them to do it! Unfortunately these days that's less about presenting a problem vs presenting a check.

Re:Does the Constitution still mean anything?

[davek]

But the point is, nothing written in the constitution means anything any more, and hasn't for a long time.
Every sentence and every clause has been violated and circumvented by a web of laws and rulings such that any citizen who points to the constitution in his defense is laughed out of court. In the legal profession, an appeal to the constitution is a huge inside joke. The sign of a rube. A target to be fleeced.

If the "personal mandate" of the health care law doesn't get declared unconstitutional, then I will agree with you. Until then, I still have hope.

Re:Does the Constitution still mean anything?

[KiahZero]

Reread the Fourth Amendment; it doesn't say that searches have to be accompanied by a warrant, only that searches and seizures must be reasonable, and that, when warrants are issued, they must be supported by probable cause:

“The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.”

That "web of laws and rulings" you so despise is what requires, as a general principle, a warrant in order for a search to be reasonable.

As for the rest of your post, it's clear you haven't thought through the implications of having each case turn on a blank-slate interpretation of the Constitution as it applies to a particular set of facts. Instead of having laws and rulings to support one's case, a person would instead only be able to appeal to a particular judge's reading of broad language. Forget uniformity throughout the legal system: you'd be lucky to get uniformity from a single judge.

Re:Does the Constitution still mean anything?

[KiahZero]

It's a tax. Congress can levy taxes.
It's also regulating interstate commerce. Congress can regulate interstate commerce.

If the individual health insurance mandate were overturned as unconstitutional, that would be evidence of the manipulation of the Constitution, not the lack thereof.

Re:Does the Constitution still mean anything?

[KiahZero]

In the abstract, yes, a sovereign entity has to consent to being sued in its courts. However, there are many statutes which provide broad consent for most claims against the government, including the Federal Tort Claims Act and 42 USC 1983 (which covers constitutional violations).

Re:Does the Constitution still mean anything?

[SharpFang]

Theoretically, Judicial independence means that this is still fine. While the government is paying the judge's salary, they are not legally allowed to affect the verdict or pressure the judge in any way. In reality, the theory is quite near practice and yes, it seems the judges usually don't seem to be really biased in citizen vs government lawsuits that occur at all. Still, it isn't like the government really has to worry. "We do not consent to this lawsuit" is the ultimate defense and no amount of judge or jury bias matters any more with that. Why use illegal pressure for dodgy favors if you can shamelessly declare "I win because I say so".

Re:Does the Constitution still mean anything?

[icebike]

Seems to me I predicted your arrival in my original post.

Suggesting we have any semblance of uniformity is laughable.
You've simply used that ridiculous example to justify the very mess I was lamenting, and in doing so, you once
again proved my main point.

Re:Does the Constitution still mean anything?

[c 0xFF]

Indeed, Congress can levy taxes ... but only certain kinds. The argument is that the penalty for not having health insurance constitutes an unconstitutional taxation. Of course, I'm oversimplifying things ... let's let SCOTUS figure that one out. I think a ruling either way would reaffirm the fact that this country follows the rules laid out in our constitution (sometimes, at least).

Re:Does the Constitution still mean anything?

[KiahZero]

Congratulations; you said something stupid and accurately predicted that people would point it out.

Re:Does the Constitution still mean anything?

[davek]

It's a tax. Congress can levy taxes.

You do realize that is exactly the opposite of what it was billed as, right? No new taxes! Keep your current plan if you like it! All our costs will go down!

Bald-faced lies.

If it's a tax, get congress to rewrite it as such and pass a new giant social welfare bill. Without that truth, I'll lean on the US constitution (hey, it got us out of slavery, didn't it?)

Re:Does the Constitution still mean anything?

[KiahZero]

Yes, having read the law, I'm aware that the individual mandate was referred to as a "penalty" rather than a tax for PR purposes. I'm also aware that there is no "clear statement" rule in the Constitution requiring Congress to spell out the source of its power for adopting any given proposal into law.

Further, I'm even aware that the Constitution did not "get us out of slavery"; as originally written, it preserved slavery.

good point

[decora]

it's like if they stopped a bunch of trucks on the highway, and scanned every single one of them for nuclear weapons, drugs, and bombs, even though they had no probable cause whatsoever.

oh wait. they already do that.

( google VIPR )

Re:good point

[X0563511]

Not really. To work (the analogy) they would have to lift and tow away whole sections of traffic at a time, only to return the vehicles (maybe, if you're lucky) weeks or months later.

Re:good point

[hawkinspeter]

You're comparing apples to oranges there.

There are reasons to scan trucks on a PUBLIC highway, but they'd need a warrant to do it on PRIVATE property.

The servers seized were most definitely private and thus they should have had a warrant to seize them.

Re:good point

[Dunbal]

return? hahahahahahahahahaha thanks for the laugh. Those servers are GONE.

Re:good point

[_0xd0ad]

Not only that, but they'd have to do it from private property - say, every vehicle in a parking garage because they suspected that one of them had a drug stash in its trunk.

Re:good point

[Cramer]

You mean *YEARS* later. Partially disassembled, missing parts, with all the windows smashed.

Re:good point

[nagnamer]

Not only that, but they'd have to do it from private property - say, every vehicle in a parking garage because they suspected that one of them had a drug stash in its trunk.

So because it's too much trouble, they just lift the whole garage.

Re:A war?

[Anonymous Coward]

one must admit, one begins to warm up to the goal, if that's exposing the real shenanigans and making real evildoers sweat it...

Act of War

[sanzibar]

next time, use a drone.

FBI: Driving businesses out of the country

[mykos]

I think most of the smart IT people are beginning to view the U.S. as a threat to their business. If U.S. investigative agencies can disrupt dozens, or even thousands, of innocent individuals and businesses with impunity, why the hell would anyone take the risk hosting in the U.S.?

Re:FBI: Driving businesses out of the country

[MightyMartian]

Because, of course, other countries are so much less intrusive.

Re:FBI: Driving businesses out of the country

[corbettw]

While no one's going to suggest setting up a co-lo in Zimbabwe or Venezuela anytime soon, there are other countries that are safer from the risk of government seizure than the US is now. Ireland, Switzerland, any of the Nordic countries, and New Zealand all spring to mind. Any one of those places would be a much better bet for setting up a new co-lo, were one inclined to do so, than the good ol', freedom lovin', US of A.

Re:FBI: Driving businesses out of the country

[X0563511]

It's bad enough driving downtown to punch a server in the face... I'd rather not have a 10+ hour flight as well.

Re:FBI: Driving businesses out of the country

[Anzya]

Meh, this has already happened in Sweden when the police confiscated a lot of servers and disrupted service for other customers just to get at The Pirate Bay. Who cares if other gets hurt in the process for the greater good...?

Re:FBI: Driving businesses out of the country

[cold fjord]

I think most of the smart IT people are beginning to view the U.S. as a threat to their business.

Your link leads to an article complaining about shutting down "websites involved in copyright infringement, the sale of counterfeit goods or child pornography", among other things. I doubt most smart IT people are involved in criminal enterprises. If most of the "smart" people you know are, maybe you should think about moving to a different part of the industry. And when I say different, I mean legal.

   

Re:FBI: Driving businesses out of the country

[Anonymous Coward]

there are other countries that are safer from the risk of government seizure than the US is now. ... any of the Nordic countries... spring to mind. Any one of those places would be a much better bet for setting up a new co-lo, were one inclined to do so, than the good ol', freedom lovin', US of A.

Riiight, no other companies were affected by the raid on PRQ to take The Pirate Bay servers in Sweden. It can happen anywhere, though I give you that some countries [cough-US-cough] are more gung ho about it...

Re:FBI: Driving businesses out of the country

[isorox]

I think most of the smart IT people are beginning to view the U.S. as a threat to their business.

Your link leads to an article complaining about shutting down "websites involved in copyright infringement, the sale of counterfeit goods or child pornography", among other things. I doubt most smart IT people are involved in criminal enterprises. If most of the "smart" people you know are, maybe you should think about moving to a different part of the industry. And when I say different, I mean legal.

 

Unless you run your own data center, and have multiple upstream links, you may be relying on a data centre that someone else is hosting those things -- either knowingly, or because a single box was compromised.

If you're not a beomouth fortune 500 company, chances are you've got a couple of physical machines in a colo, or even just a VM or two. You have no control over who Rackspace rent their servers and space too, so when the FBI come calling, you lose money.

Re:FBI: Driving businesses out of the country

[nedlohs]

And you know that the domain registrar you used didn't sell a domain to a single person/enterprise that might be suspected of a crime. And the DNS provider you use doesn't have such a customer. And the hosting provider you use doesn't have such a customer. And the data center the servers are in doesn't have such a customer. And you and any of your providers and any of their customers haven't annoyed someone enough to get setup for such a raid.

However, for a lot of the rest of us issues of size and finances mean we just lease space on a shared machine, or lease some machines, or lease some rackspace and hence have no control over the activities of nearby servers.

Also the rest of us love in a world where people make mistakes and say no-knock RAID the wrong house or seaze the wrong hardware...

Re:FBI: Driving businesses out of the country

[Asic Eng]

Other jurisdictions may not be less intrusive, but it might be interesting to have a second server in another jurisdiction. As a hosting company, you could perhaps team up with a hosting company in another country and have a mutual backup agreement.

Re:FBI: Driving businesses out of the country

[PPH]

Because, of course, other countries are so much less intrusive.

They make it their business to be hosting friendly. Like Switzerland is banking friendly, Denmark, Hong Kong and Ireland are business friendly. Someone will see the opportunity to attract hosting companies (and all the related IT business) by establishing a respectful attitude toward people's data. And people will move their operations there.

Re:FBI: Driving businesses out of the country

[Travelsonic]

In Soviet Russia, joke age YOU!

Re:Digital Forensics

[geminidomino]

perhaps because they mistakenly thought that âoeone enclosure is = to one server"

More likely, they realized that one enclosure would bring in more at auction than one server...

Ultimate DOS

[Anonymous Coward]

It's the ultimate Denial Of Service attack:
1) Co-locate stuff that the FBI doesn't like with the server that you want to DOS
2) Report your server to the FBI
3) Sit back and let the FBI do the rest.

Re:Ultimate DOS

[geekprime]

it's like a new age swatting!

Re:Ultimate DOS

[petermgreen]

Frankly it seems like if you could get colocated in the same rack as your target there would be less risky ways of taking them out than setting the FBI on yourself.

not Group punishment more like hiting the main to

[Joe_Dragon]

not Group punishment more like hitting the main to the building trun off one office.

The reason they took the whole rack....

[Wingman 5]

... is they did not want to power down the server.

Law enforcement is trained that if you are seizing a computer, if possible, do not let it be shut down/locked. Forensics can snapshot the RAM and possibly get encryption keys that would be lost if the server was powered down. Worst case there could be a whole drive encryption that needs a password every boot, if you let the computer shut down you lose everything and all you will have is a worthless box without the password.

It is likely there was no way to remove the server from the enclosure while keeping it supplied with power. So what they likely did is they spliced in their own UPS to the cabinet and rolled the whole thing out. to their truck where they could keep it powered till a expert could get in and get a dump of the system state.

Re:The reason they took the whole rack....

[icebike]

Sounds like wild speculation to me. And a great deal of fantasizing.
If you physically have the server, you simply power it down, even by yanking the cord, (not nearly as harmful to a modern server as you've been lead to believe) then pull the hard drives and clone those, and deal with their content as mere data. Taking the entire cabinet is the sign of fools and novices.

Re:The reason they took the whole rack....

[hawguy]

It is likely there was no way to remove the server from the enclosure while keeping it supplied with power. So what they likely did is they spliced in their own UPS to the cabinet and rolled the whole thing out. to their truck where they could keep it powered till a expert could get in and get a dump of the system state

And if they have this magic splicing capability (as opposed to relying on redundant power supplies to let them transparently hook up their UPS), are you saying that it was easier to supply 10KW of power to an entire cabinet than it would have been to supply 400W of power to a single server?

They'd need 1000 pounds of batteries to keep the cabinet powered for any appreciable amount of time.

Re:The reason they took the whole rack....

[Wingman 5]

If I keep all of my data in a strongly encrypted container (that does not have a password that is brute force able in a reasonable amount of time), how do you expect to gain anything meaningful "dealing with it as mere data" without the decryption key which was stored in ram till you shut the machine off to clone the drive?

Re:The reason they took the whole rack....

[Anonymous Coward]

Nope,

From my forensics class - The drives may have an encryption with a bit width that would make decryption a serious task.
While the machine is on the decrytion key is in memory somewhere (or more probably in the TPM) As long as the machine
remains on you have access to the hard drive. Shut the machine down and you loose the key. Try getting it out of the owner
of the machine (well there *is* always water-boarding).

Re:The reason they took the whole rack....

[Wingman 5]

They don't need to keep the whole rack powered, just the one machine they are interested in, they could power down the rest of the rack and a off the shelf UPS could run it for plenty of enough time to get it to a truck with a inverter on it.

As for the "magic splicing" it is not hard to do, anyone with a basic understanding of electric circuits can splice two live cables together.

Re:The reason they took the whole rack....

[cold fjord]

Taking the entire cabinet is the sign of fools and novices.

Or someone concerned about the chain of custody for evidence.

Re:The reason they took the whole rack....

[jamesh]

As for the "magic splicing" it is not hard to do, anyone with a basic understanding of electric circuits can splice two live cables together.

But it's a lot more entertaining when someone without a basic understanding of electric circuits does it :)

In any case, if I was writing malware i'd be detecting when network connectivity changed (eg my server was being loaded into a truck and no longer connected to the data centre) and initiating an erase of all the disks and RAM... keeping the server hot to run forensics would only make this easier.

Re:The reason they took the whole rack....

[fluffy99]

They don't need to keep the whole rack powered, just the one machine they are interested in, they could power down the rest of the rack and a off the shelf UPS could run it for plenty of enough time to get it to a truck with a inverter on it.

As for the "magic splicing" it is not hard to do, anyone with a basic understanding of electric circuits can splice two live cables together.

There is a product called HotPlug that is meant for seizing assets without powering them down. It works pretty slick. Basically, you plug it into the same power strip, flip the switch and unplug the powerstrip from the wall. You can also splice into the cord or outlet if needed.
http://www.wiebetech.com/products/HotPlug.php

Re:The reason they took the whole rack....

[Wingman 5]

Thanks, i was looking for that, I wanted to include that in with my OP but I could not find it.

Re:The reason they took the whole rack....

[hawguy]

In any case, if I was writing malware i'd be detecting when network connectivity changed (eg my server was being loaded into a truck and no longer connected to the data centre) and initiating an erase of all the disks and RAM... keeping the server hot to run forensics would only make this easier.

Or better, an inertial motion sensor located inside the server case:

      http://www.motionnode.com/

As soon as it detects motion, have the server erase everything.

Re:The reason they took the whole rack....

[X0563511]

It's not so easy if you need to keep the power in phase, or not backfeed something. It's kind of hard to just magick a transfer switch into a live circuit...

Not to mention that even getting the thing moved 10 feet without disks shitting their pants is a whole other difficulty...

Re:The reason they took the whole rack....

[X0563511]

In a completely wrong power phase, resulting in things shutting down, crashing, or exploding when you try to splice it into a live circuit without a transfer switch...

Lets not forget this would also cause all sorts of hell in the PMM or UPS systems, probably shutting half the damn facility down in the process.

Re:The reason they took the whole rack....

[hawguy]

There is a product called HotPlug that is meant for seizing assets without powering them down. It works pretty slick. Basically, you plug it into the same power strip, flip the switch and unplug the powerstrip from the wall. You can also splice into the cord or outlet if needed.
http://www.wiebetech.com/products/HotPlug.php

Interesting device -- I see on the specs page that it only goes up to 5 amps.

So it looks like I might (barely) be safe on my fully loaded Sun E450 (500W power supplies) as long as I step down the input voltage to 100V.

Of course, my original point still stands that it makes no sense to power an entire rack when all you want is one server. If you can figure out how to cut over to your own UPS, I think you can figure out how to keep power to the server while you unrack it.

Re:The reason they took the whole rack....

[hawguy]

http://www.google.com/products/catalog?hl=en&q=10000+watt+generator&safe=off&sqi=2&cid=11842679777176480618&os=contents

Weighs about 200 pounds and will run however long you need it.

It turns out that computers don't run so well off of small generators - I had small facility that I tried to run off off a generator part-time. We tried a variety of small generators, everything from a small Honda 2000W Inverter model up to a 15KW gasoline powered generator. The APC SmartUPS UPS's didn't fare any better - they switched to battery as soon as we cut over to generator. One computer managed to run for about an hour on the Honda inverter generator, but then the power supply literally began to smoke, I guess the generator doesn't do a good job of approximating a sine wave.

The magic fix was a 20KW diesel generator, but only when we had it about 50% loaded by turning on most of the lights in the facility. When it was lightly loaded, I'm guessing that the frequency wasn't stable enough.

Re:The reason they took the whole rack....

[uncqual]

Ah, but only if the FBI had noticed the accelerometer nestled among the components that should have been on the mainboard before moving the rack.

Re:The reason they took the whole rack....

[sjames]

Most datacenters have a central UPS. If you take the rack, it will surely power down when you reach the limit of the power cords.

Besides that, an instant power down is less likely to cause damage than moving spinning hard drives around.

Re:The reason they took the whole rack....

[sjames]

So, why then didn't they just splice the relevant servers and leave the rest?

Re:The reason they took the whole rack....

[fnj]

I'd like to see you try to splice into a live AC cable. I think that would be lots of fun. And I'd like to see you try to synchronize the two AC phases so you can shut down the original source (the mains or the UPS) and replace it with the new offline source without interrupting the flow of power. That ought to be a whole lot of fun. Maybe you could do it, but my money's on you making a spectacular mess when you try it.

Here's what I think. I don't think the reason is what has been speculated. I think they took the whole rack just to make their life easier, so they wouldn't have to dink around identifying the particular server(s), and so they wouldn't have to dink around with tools inside the victim's premises removing out the particular server(s). Maube there were too many servers they wanted. Or maybe their victim actually did take up a whole rack or a major part of it.

Re:The reason they took the whole rack....

[fnj]

You do realize that during the time the two UPSes are both hooked up to the equipment (to allow uninterrupted flow of power during the switch), you would have to precisely phase lock your new UPS to the original one. Two UPSes free running would just make a big short circuit when you connect them together.

Re:The reason they took the whole rack....

[dbIII]

Hey I saw that Seinfeld episode too!
What is more likely is a shotgun approach of grabbing everything that may potentially be evidence.

Re:The reason they took the whole rack....

[dbIII]

Criminals are typically so because their crime is simpler than real work. We could have malware more advanced than most applications but nobody bothers - they go for the low hanging fruit.

Re:The reason they took the whole rack....

[L4t3r4lu5]

If I keep all of my data in a strongly encrypted container (that does not have a password that is brute force able in a reasonable amount of time), how do you expect to gain anything meaningful "dealing with it as mere data" without the decryption key which was stored in ram till you shut the machine off to clone the drive?

Oblig. xkcd

Re:The reason they took the whole rack....

[psiclops]

great for earthquakes

Re:The reason they took the whole rack....

[petermgreen]

And if they have this magic splicing capability

A basic UPS would probablly work fine (you don't want a UPS that tries anything too clever) if you connected it's input to it's mains, then it's output to the server then disconnected mains from the server and finally disconnected mains from the UPS. A purpose designed device could probablly do it even more easilly.

are you saying that it was easier to supply 10KW of power to an entire cabinet than it would have been to supply 400W of power to a single server?

Could well be

Consider what would be involved in de-racking a server while keeping it powered. If it doesn't have a pair of redundant PSUs (and afaict a lot of low end servers of don't) then bascially you only option would be to splice into the power cord close to the server (before it disapears into the maze of wiring in the rack). Splicing mains voltages in the restrictive conductive envrionment of a rack sounds dangerous to me and the spliced connection would have to be done very carefully to make sure it didn't fall apart as the server was de-racked. Even if the does have redundant PSUs then the non-locking power connectors mean it would be very easy to accidently lose power.

If you take the whole rack you can either splice into the power cables before they enter the rack (where you have much more room to work) or just plug your kit into the racks existing power distribution. You can then have experts deal with how best to get inside the individual servers later.

Re:The reason they took the whole rack....

[petermgreen]

Afaict you could do a whole rack pretty easilly using a standard UPS bypass switch.

1: Connect bypass switch and UPS to mains
2: Connect bypass switch to rack (which is still connected to mains)
3: Disconnect rack from mains
4: connect UPS to bypass switch (which should then switch out of bypass mode)
5: disconnect UPS and bypass switch from mains.

Re:The reason they took the whole rack....

[Asic Eng]

Well, it's also easier to just take everything. If you grab one server and that happens to be the right one - great. However if you grab one server and it turns out later that the data was actually on another machine ... you don't want to explain that to the boss. And there might be something possibly related on one of the other servers - if the people you are after have rented one server from that ISP, they might have rented another one, too. It's a possibility.

I'm not saying it's legitimate, just that they might find it more convenient.

Re:The reason they took the whole rack....

[Doctor Faustus]

Why not just do it right there?

Re:The reason they took the whole rack....

[hedwards]

So, instead they steal property which doesn't belong to them and for which they don't have a warrant? Yeah, that's so much better, I suppose if they aren't going to file through those other machines looking for incriminating evidence either.

As for the operator of the facility, what's wrong with them, why is it that they don't know whose equipment is now missing? They should know whose equipment is where if for no other reasons than inventory and security.

Re:The reason they took the whole rack....

[koollman]

You're so right! I will modify my customer creation form so that they have to check
"I am not a criminal" before they can create the account.
That way I can inform my customers that they aren't hosted with any criminals.

Although I have a doubt, maybe I need also a "I cannot be the target of an FBI investigation" checkbox, you know, just in case.

And I would have to make sure no other potential criminal or potential suspect will be
hosted anywhere near my own servers or racks, too.

Yes,it's simple. That way I will be certain that it cannot happen to me.

(now that I think of it ... I would probably be safe if I only do hosting for some of the state organisations. or the fbi)

Re:The reason they took the whole rack....

[icebike]

They can also freeze the circuitry on the motherboard (with a liquid-nitrogen-like spray)

Oh, Please!!!

Stop, Just stop. Ok?

Re:The reason they took the whole rack....

[icebike]

Really?
Hardware encryption modules on a rented colo box hosting blogs and web traffic?

Who are you trying to kid. This was your typical rent-a-blade operation, where you as the user
have no idea exactly what you are renting, real hardware, virtual hardware, or cloud service.
This was not your in-house datacenter. Nobody bothers to encrypt storage on non-mission-critical
public facing rented web servers that they don't even have physical access to.

Just stop, OK?

Good hosting providers outside the US?

[Lord Juan]

Call me paranoid but I am starting to look around for hosting options outside the US. The stories of the massive collateral damage when they take away shared servers and seize domain names is getting me nervous.

Re:Good hosting providers outside the US?

[Skuto]

Other countries, specifically developed ones who are "allies" of the US, probably do exactly the same.

Government is "fire first, respect the law later" pretty much everywhere.

Tempted to start a demolition company.

[SharpFang]

I'm tempted to start a building demolition company. Using tactical nukes. You point out the town your building you want to demolish is in, and we guarantee it's razed to the ground, no other details needed.

Re:Not extreme

[hawguy]

I am a federal agent (non-FBI) who has seized large amounts of digital evidence. In criminal cases, you need entire hard drives so you can do forensic extraction. Can you ask the ISP to retrieve the data for you? Yes. However, it depends on 1.) Is this an email address or a large organization with colocated servers. 2.) How much do you trust the ISP? (based on past actions, size, clientele, etc.). BTW, if you search large companies who have their congressman on speed dial, you can be assured that the agents and judge have evaluated the impact to legitimate business vs illegal activity.

I'd think that the same thing applies when the FBI sees a suspect enter a parking garage - they know he entered the garage and are pretty sure that he hid his contraband in a car. The garage owner might be working with the suspect, so they can't trust him. The question is, can they seize all 200 cars in the garage and tow them back to be disassembled and searched to be eventually returned to the owners, perhaps no longer in working order? Would any judge allow that?

If the answer is no, why is it different with servers?

Re:Not extreme

[icebike]

As a federal agent (non-FBI) you should have been trained that the "entire hard drive" does not extend to the entire RACK of servers.

you can splice cables to a single server

[dutchwhizzman]

so I don't see the reason to take several racks. The risk that the server powers down that way is roughly the same as for an entire rack. Also, the reason why things were taken is not given. For all we know, there may be an illegal mp3 hosted on one machine and the MAFIAA had it seized for "economic terrorism". The feds better come up with a pretty good explanation, or there will be a lot of damages to be paid by the USA tax payer.

Re:you can splice cables to a single server

[Wingman 5]

Have you tried to remove a server from a rack without accidentally detaching the power cable? They went with the option that had the lest chance of failure.

Re:you can splice cables to a single server

[X0563511]

Have you ever tried to move a server out of it's rack, out of the building, into a vehicle, and then wherever it needs to go... ... without the disk curb-stomping it's heads all over the platters?

Power is only part of the problem.

Hosting centre is at fault

[jamesh]

The hosting centre is at fault here. "Naughty Servers" should be clearly labelled as such so they can't be mistaken for "Benign Servers". If those fatcats in Washington had just listened when the 'Evil Bit' was first proposed we wouldn't be in this mess now!

Re:Hosting centre is at fault

[Dynedain]

It was a colo. And the hosting company (the owner of the machines) gave the FBI the info needed to pinpoint the one single server they were after. The FBI took several racks of equipment the hosting company had in that colo instead of just the single machine.

Re:Hosting centre is at fault

[xenobyte]

The usual procedure is to seize the suspect servers and everything connected to it... Guess the FBI needs a bigger warehouse...

Seriously, all the unrelated companies affected by these over-raids need to join forces and sue the FBI for some serious cash, including personal liability for the agent in charge for the raid. If they win it will make any future agent in charge of raids very, very, very careful not to seize as much as an unrelated patch-cable... and that's the way it should have been from the beginning.

Re:Hosting centre is at fault

[jamesh]

Hmm, you mean hosting centers should, by default, include a (unused) server in each rack and label it 'naughty server'? that would fix the downtime for customers for a while and give an early warning to make extra backups when this server gets confiscated

A box with a few flashing lights on it and a few fans humming away would probably do. In fact if you put more flashing lights on it than any other server in there the FBI would just take it by default, whatever label you put on it. Add a skull and crossbones and a swastika or two just to be sure.

Re:Hosting centre is at fault

[black soap]

The usual procedure is to seize the suspect servers and everything connected to it...

"Good God, Agent Smith, this thing is connected to the Internet"

"I'll call in a second truck."

Re:Not extreme

[cold fjord]

Mod up.

SPAM!

[malsbert]

nt

Re:catastrophic

[TaoPhoenix]

This is worse.

Conventional disasters don't give you jail sentences for owning a copyrighted pic of a terrorist engaging in Pr0n.

Come to Canada, we don't do this... yet

[Wrexs0ul]

I've seen folks comply with DMCA out of courtesy, but for the most part your Canadian-based providers would have a lot more tape before the RCMP knocked on their door for an American agency.

So far it seems the RCMP are mainly concerned with counterfeit goods and pot. Assuming we don't run out of these I'd imagine your colo is pretty safe :)

-Matt

so let me get this straight...

[bmo]

If I want to perform the ultimate denial of service - get the servers ripped out - all I have to do is create a stir in the press from the same hosting company that I want to target for my denial of service attack.

Good to know.

--
BMO

FBI to Lulzsec:

[Cyberllama]

"That's not a denial of service attack! This is a denial of service attack."

Re:FBI to Lulzsec:

[xenobyte]

The obvious conclusion: In order to perform the perfect long-term denial of service you gain access to servers at the target company, then hack or launch a 'normal' denial of service attack on some high profile third party from there, leaving plenty of traces for the FBI to follow, then wait for the FBI to seize the servers at the real victim, causing an effective denial of service there that can't be easily alleviated or prosecuted. It's a perfect crime with the FBI as a key player.

Re:FBI to Lulzsec:

[biodata]

I wonder if that's what happened here. It does seem unlikely that anyone would use their own servers to launch a giant hack attack. Does anyone know which of the victim companies was the real target of the DoS?

DoS

[xenobyte]

OMG! They were hacked from the Internet! Seize the Internet! All of it!

Just trolling, ignore.

[malsbert]

Yes, you have none of those Nazi types

And surly none of those Fascist types either

If you Americans are not oppressed, then your smoking something good ( and you really should learn to share! ).

As for being brave? well you do have a few 100.000 soldiers that have shown some balls, so that leaves what? 309.800.000 yellow-bellys?

As your benji-boy wrote; They who can give up essential liberty to obtain a little temporary safety, deserve neither liberty nor safety.

Re:Just trolling, ignore.

[gonzo67]

You give two examples of websites or groups that have ZERO power in the US. We also have Al Qaida supporters in the US, but they are not in any position to influence the country.

While the US has issues, they are NOT as you would imply.

Re:Just trolling, ignore.

[Gandalf_the_Beardy]

"We also have Al Qaida supporters in the US, but they are not in any position to influence the country." - yes because they so spectacularly failed to influence anything before in the US didn't they? I presume you have heard that there is a big hole in Lower Manhattan?

Re:Just trolling, ignore.

[malsbert]

Hi there Gonzo.

I'm not implying anything. I was simply Idling, when i noticed an AC had replied to one of my comments. As the comment in question is semi-flamebait
( O’er the land of the oppressed and the home of the cowards! ) and AC seemed to "accept the challenge" ( Italian fascists, German Nazis, etc ) i simply "shot" from the hip, to see if he was still online and would care to entertain me. to keep you and any others out of the crossfire i did set the comment subject to: "Just trolling, ignore", But it seems you chose not to :)

As for any "issues" the US my have, i can only say; you people need to get out more! your issues are by no means US issues, we have plenty of the same things right here in little old Europe and i'm pretty sure the rest of the world is in position to point fingers :)

So; sorry if spoiled you morning, next time do what subject line says and IGNORE ;)

Re:Just trolling, ignore.

[Dunbal]

If you Americans are not oppressed, then your smoking something good ( and you really should learn to share! ).

I'll say it again: I've never heard the words "you can't" more often than in the US. Oh and remember to STAND BEHIND THE YELLOW LINE, SIR. I SAID BEHIND THE YELLOW LINE...

Iceland

[biodata]

Decent infrastructure, decent government, some coastguards but not really interested in starting wars with anyone unless it's about fish, and a legislative framework that is conducive to free speech.

Re:not Group punishment more like hiting the main

[Sardaukar86]

not Group punishment more like hitting the main to the building trun off one office.

You are one weird robot, dude. Why don't you try that Preview button and actually read your drivel before you post it?

Hint: if you can't understand what you've written, we sure as hell aren't going to either.

Re:not Group punishment more like hiting the main

[Cederic]

hmm. 'trun' isn't an acceptable abbreviation for 'to turn' in your lexicon?

I thought his analogy was very apt, irrespective of his dyslexia.

The elder Bush

[ThatsNotPudding]

had a phrase for such behaviour, but I can't quite remember it...

A problem endemic with law enforcement

[Attila Dimedici]

I am pretty sure this happened as a result of a problem that is endemic with law enforcement. A large percentage of people in law enforcement have come to believe that all people that they interact with are criminals who are acting to keep law enforcement from discovering the evidence to convict that person and/or others. As a result, they did not trust the hosting company to work with them to obtain all of the data of the target of their investigation.
The proper way to have done this would have been to go in with someone from the FBI who was technically proficient who would then work with the hosting company to isolate and migrate all of the virtual machines containing the target's data to a single server (or several, if that was necessary) and seize that server(s).

Re:A problem endemic with law enforcement

[Toze]

I have some bad news for you; there are enough laws in America (and Canada, and England, etc.) that you break roughly 8 laws a day. You *are* a criminal acting to keep law enforcement from discovering the evidence to convict you.

Re:A problem endemic with law enforcement

[_0xd0ad]

I realize you're joking, but Halon doesn't actually work like that.

If something is actually burning, it's possible for the Halon to be converted into phosgene, which is highly toxic, and other toxic or irritating substances. However, Halon itself is not toxic although it does cause temporary giddiness and impairment.

http://en.wikipedia.org/wiki/Halomethane#Safety

Halon 1301 is more usually employed in total flooding systems. In these systems, banks of halon cylinders are kept pressurised to about 4 MPa (600 psi) with compressed nitrogen, and a fixed piping network leads to the protected enclosure. On triggering, the entire measured contents of one or more cylinders are discharged into the enclosure in a few seconds, through nozzles designed to ensure uniform mixing throughout the room. The quantity dumped is pre-calculated to achieve the desired concentration, typically 3–7% v/v. This level is maintained for some time, typically with a minimum of ten minutes and sometimes up to a twenty minute "soak" time, to ensure all items have cooled so reignition is unlikely to occur, then the air in the enclosure is purged, generally via a fixed purge system that is activated by the proper authorities. During this time the enclosure may be entered by persons wearing SCBA. (There exists a common myth that this is because halon is highly toxic; in fact, it is because it can cause giddiness and mildly impaired perception, and also due to the risk of combustion byproducts.)

Non-compliance on the host's part?

[78spb89]

I used to work for a hosting company, and the FBI was interested in plenty of our customers. They would show up with a warrant for information and explain that we could either provide the data they needed, or they could seize the equipment the data was on. I wonder if the host failed to comply with a warrant requesting data?

Re:Non-compliance on the host's part?

[PPH]

That's a nice little data center ya got there buddy. It would sure be a shame if something happened to it.

Re:Non-compliance on the host's part?

[blueg3]

Probably. Or the FBI had reason to believe the hosting company would protect the customer under investigation by supplying false information. (Or the investigation is so serious that it's dangerous to trust the hosting company at all.)

In order to not miss evidence, you either need the assistance of a trustworthy sysadmin who can accurately identify which resources might have been used by the suspect, or you need to seize everything.

Law enforcement can and has had warrants rejected because they are too broad and would cause unnecessary injury to the business. It's not really in the FBI's best interests to seize everything unless they have a reason to do so.

Plus, it annoys their forensics guys, since now there's a lot more machines to look through. If the hosting company is smart, there'll be a suit demanding the return of the hardware not needed for the investigation, which adds substantial time pressure. Heck, a number of forensics guys have had to work under the restriction that, as soon as possible, they: image all of the case-related machines, redact illegal data from the original machines, and then return the machines.

Re:Non-compliance on the host's part?

[Skapare]

No one (at least who knew the technical stuff) was physically present at the site at 3AM when the raid was conducted. So effectively, no one assisted the FBI. And, of course, FBI agents are not knowledgeable about how to figure out which servers have which sites, and would have to do their forensics to figure that out. Apparently an even bigger problem is that the way things were taken, servers that were not taken were left inoperable or unconnected (e.g. they unplugged stuff not exactly knowing what was where or tracing things down).

IMHO, law enforcement should be required to conduct these raids during business hours (to obtain identifying assistance from staff) unless there is a specific reason to believe that doing so would compromise the investigation (e.g. a very urgent need to move now, or that the ISP staff itself is involved and would hit a kill switch somewhere). And, maybe they thought this was just such a case.

The big question in your hosting company case is: how would they even know which equipment the data was on if no one assisted them?

Re:not Group punishment more like hiting the main

[Sardaukar86]

hmm. 'trun' isn't an acceptable abbreviation for 'to turn' in your lexicon?

I thought his analogy was very apt, irrespective of his dyslexia.

You're quite right, the fault is mostly mine in restrospect - my parser barfed completely on his post, it looked like pure Engrish.

Of course I see now that it's actually quite understandable so I guess I must be tired.

Oh, and to be fair, probably prejudiced too I guess; Joe_Dragon looks much like Joe_The_Dragon who posts in the same style and is similarly immune to the Preview function. Maybe I'm just a grumpy old fart but I tend to feel that if one thinks a post is worth others' time to read, expending the effort to read it over oneself is hardly that much to ask.

Re:Virtualization

[Qzukk]

If the cops are too stupid to take one server from a rack, what makes you think they'll be able to figure out this signed VM state image thingy?

Giganews

[Subratik]

Giganews has servers hosted in ashburn which isn't too far from where they mentioned in the article. More specifically, they host the VyperVPN service they have in ashburn.... wouldn't be surpised if they confiscated their hardware because behind all those proxies they saw a Giganews IP address. I have no idea if vypervpn is down though... so I'm just speculating X_x

Law moves slowly

[tekrat]

By the time guilt or innocence *is* proven, the equipment seized becomes useless.

I've seen cases where it can take a decade for things to resolve to the point where you can try and re-obtain your equipment, but by then, who bothers? The hard-drives have seized, the pentium II has since been replaced by the pentium 4, and the OS is 5 generations behind.

Once the law takes your equipment, it's gone. Unless you have some emotional link to a particular computer, it's not worth the effort, paperwork, expense, and headache required to retrieve your now-useless, and very likely terminally damaged, equipment.

Updated disaster recovery plan....

[justsayin]

So, should we all start adding FBI seizures to our disaster recovery plans?

Strange

[jms1]

DigitalOne's chief executive, Sergej Ostroumow, said: "This problem is caused by the FBI, not our company. In the night FBI has taken 3 enclosures with equipment plugged into them, possibly including your server - we cannot check it."

Am I the only one who finds it odd that the management of a datacenter "cannot check" whether or not a particular machine was taken by the FBI? Every datacenter I've ever worked in, had an inventory of what equipment was where, and KNEW where every machine was, down to the specific "U" for shared racks, or at least which rack or cage (in cases where a single client had rented an entire rack or cage.) Presumably they know which racks were emptied, they should be able to check their inventory for those three racks to see what was taken...

Or is he saying that the FBI is preventing his personnel from entering the building to check on what was taken and what's still there?

Re:Strange

[Kalriath]

DigitalOne doesn't own the datacentre. It's either CoreSite or Equinix who own it. DigitalOne is a company headquartered in Switzerland, so the "cannot check" is likely due to "because we're on the other side of the planet".

Re:Not extreme

[Overzeetop]

No, but they can probably shut down the garage, barring all entry and exit, until such a search is completed. It has the same effect - preventing access to your property/data, even though it is almost certainly not involved in the crime.

Fryazino transliteration

[Alex Belits]

Though completely offtopic, "Sergej Ostroumow" is a now-unusual Cyrillic romanization of what would be "Sergey Ostroumov" -- it is based on matching Cyrillic and Roman characters in KOI-7 and KOI-8 charsets.

A terminal made in Fryazino in 80's would show one in place of another if program omitted SO or SI control character (to be precise, also switching around uppercase and lowercase), so it became associated with this kind of transliteration. It is currently alive in form of "Phonetic" keyboard layouts that allow Cyrillic input on keyboards with no Cyrillic labels, though "Phonetic" is kind of a misnomer, considering that it includes such mappings as "v" to "zhe" and "q" to "ya".

The moral of this story....

[SwedishChef]

Don't put all your servers in one co-location site. Spread them around. If one goes down the others will still provide service (but probably slower service). I know this is a more expensive option but if you're getting substantial income from your site then you need to make sure than no single site that gets seized will disrupt that income.

Jack-booted thugs with wire cutters

[mschaffer]

Welcome to the USA.
Former home of the US Constitution.

Could be Worse

[SaroDarksbane]

Did they get shot? Did any pets get shot? If not, I'd say they are one-up on most people who interact with what passes for "law enforcement" these days.

"Your honor, the data center employee was was wielding that server in a hostile manner. They even called it a 'blade'! I feared for my life."

Re:What company?

[gknoy]

I think he wanted to post useful and interesting strategic information ("Here's how we do it") without risking being blasted as a karma whore.

I'm sure that a ton of people would like to be his customer, now, though, and would upmod a link to his company. :D

Re:What company?

[TooMuchToDo]

Somewhat this. If you search online, you can find us. We do enterprise hosting for the majority of our clients (none of our clients pay less than $7K-10K/month), and a fair number of them are household names (if you use laundry detergent, purchase car tires, or drink soda, you probably know our clients).

I by no means posted on Slashdot to brag (we are quite the humble type). I posted to say "Business can do this the right way, they just have to commit to doing it".

Repressive measures

[nagnamer]

If this happened in China, I bet someone'd start the repressive measures discussion right away...

Been there myself.

[Enos Shenk]

I've been one of the collateral victims of one of the FBI's "grab everything and go" raids. Some of you might recall the FooNet raid back in 2004.

http://www.thewhir.com/web-hosting-news/foo021604

My shell host had their boxes colocated with Foonet, so when the FBI went in and just grabbed every single machine in the building my host was screwed. And by extension, I was screwed. My host ended up losing almost all their customers (Including me) when they neglected to get a backup online in a reasonable amount of time. As far as I know nobody else on my host ever got their data back from the feds.