Slashdot Mirror
Too slow! FBI Shuts Down Hosting Service
Chope writes "If FBI agents showed up at your data center bearing a warrant, would you be able to provide them prompt access to customer data?
BZZZZT! I'm sorry, but you've taken too long to answer. We'll be confiscating all the hardware you use, er, used to use, to run your business. But we'll get it back to you 'real soon now.' Thank you for playing. CarrierHotels.com is carrying the story of a FBI raid on a web hosting company. When the hosting company didn't and/or couldn't provide the information the FBI was looking from its several terabytes of data within "several hours", the FBI decided it was more "efficient" to seize all the web servers and customer data as part of the FBI's investigation of a hacking incident."
Last year I found the a controller of the proxy that was installed on a NT workstation happened to be controlled out of the same data center that was shut down. That machine was telling the NT box to send out massive amounts of spam.
This is about the last data center on earth where script-kiddies can get free shell accounts.
This is a case were many servers got caught in the crossfire aginst the script kiddies and spamers.
FBI Shutters Web Host
By Rich Miller
Carrier Hotels Editor
Posted Feb 19, 2004
If FBI agents showed up at your data center bearing a warrant, would you be able to provide them prompt access to customer data? How long would it take?
That's an important question in the wake of an FBI raid of Columbus, Ohio hosting company CIT Hosting last Saturday. Federal agents wound up shutting down the entire operation, seizing all the company's web servers and all customer data as part of its investigation of a hacking incident.
CIT Hosting, also known as FooNet, markets itself as "the leader in the IRC and DDoS protection business for the last 5 years." The company posted a web page informing customers that its data center was shut down, and instructing customers to contact the FBI if they needed access to their files.
"The FBI executed a search warrant issued by the United States District Court for the Southern District of Ohio regarding the IRC network that we host," the company said in its statement.
IRC (Internet Relay Chat) is a live chat system that allows users to create private discussion rooms. While IRC has a lengthy history of legitimate use, it is also a medium for discreet communication between hackers. CIT said the FBI was "investigating whether someone hosted on our network hacked and attacked someone else."
"After several hours of attempting to track down, inspect and audit the terabytes of data that we host, the FBI determined that it was more efficient (from their point of view) to remove all of our servers and transport them to the FBI local laboratories for inspection," the statement continued. "The FBI has assured us that as soon as the data has been safely copied and inspected, the equipment will be promptly returned. Unfortunately, the FBI has not been able to tell us when they will be completed with their inspection."
The seizure isn't standard procedure, and there's no way to know exactly what prompted it. CIT's account suggests the FBI may have lost patience with the process. The IRC-focused nature of CIT's business may also have been a factor.
But if you're a data center operator, you want to avoid any scenario in which the FBI gets impatient and starts hauling away your servers. Just one more item on the contingency planning checklist for the times in which we live.
From their site - don't forget to let the FBI know what you think! rwhite3@leo.gov
02/23/2004 CIT re-establishes service.
We have restored service at Equinix's Chicago Data Centers. We are in the same facilities as MSN and many fortune 500 companies. The facility has multi OC192 connections to the backbone.
The FBI has begun retuning equipment to CIT which is being shipped to our new facilities in Chicago.
At this time CIT will continue to provide dedicated DDOS Protected web hosting only.
CIT provides reliable and scalable solutions for customers of all sizes and services. Located in Equinix's Chicago Data Centers , CIT has access to all the major carriers without the need for local loop circuits.
Our Chicago staff is focused first and foremost on customer satisfaction, and will take every action necessary to accommodate each customer. Unlike many large ISPs, CIT prides itself in its ability to provide personalized service to each customer - if a customer calls twice for assistance, they can usually speak to the same representative. Our sales and support teams are allowed a great deal of flexibility to work together to resolve each customer's needs on an individual basis. Our success and rapid growth can be attributed to the satisfaction of our customers - word-of-mouth referrals account for a large portion of the new business we receive each month.
The IRC Network will remain down until further notice.
02/14/2004 FBI Confiscates all servers
Dear Customers of FOONET/CIT:
We regret to inform you that on Saturday February 14, 2004 at approximately 8:35 am EST, FOONET/CIT's data center in Columbus, Ohio temporarily ceased operations.
Here are the facts of what occurred:
The FBI executed a search warrant issued by the United States District Court for the Southern District of Ohio regarding the IRC network that we host. According to the warrant, it appears that the Bureau is investigating whether someone hosted on our network hacked and attacked someone else.
After several hours of attempting to track down, inspect and audit the terabytes of data that we host, the FBI determined that it was more efficient (from their point of view) to remove all of our servers and transport them to the FBI local laboratories for inspection. This was completed at 7:00 pm EST same day.
The FBI has assured us that as soon as the data has been safely copied and inspected, the equipment will be promptly returned. Unfortunately, the FBI has not been able to tell us when they will be completed with their inspection.
We have been told by the Special Agent in charge of the investigation that If you need access to your data you are asked to please contact the Bureau via email to rwhite3@leo.gov. Make sure to include in your email your name, mailing address, and telephone number with area code.
Since we wish to focus 100% of our efforts on restoring services, we would appreciate it very much if you do not attempt to contact us directly. Please rest assured that we are doing everything possible to restore service to you as quickly as possible.
To the many who have inquired, Paul and family are OK, although shaken by these events. They are at home and awaiting the blessed event of their new child's birth. We thank you for your good wishes and prayers.
Please check back here often. Through this site, we will keep you informed of ongoing developments as we know them.
Thanks again for your understanding.
It's not like I agree with this, if indeed things happened as the article state... but a quick google on FooNet (AKA / DBA CIT ) turns up some VERY interesting results.
I google'd quickly on a hunch, and sure enough I got some rather interesting hits.
I claim to know nothing about SPEWS and how they go about adding to the blacklists, but they apparently are no stranger to it.
Furthermore, it seems that this IS NOT the first run-in with the FBI that FooNet/CIT has had: from here, if you scroll down a bit, you'll see the following text: The FBI executed a search warrant issued by the United States District Court for the Southern District of Ohio regarding the IRC network that we host # We regret to inform you that on Saturday February 14, 2004 at approximately 8:35 am EST, FOONET/CIT's data center in Columbus, Ohio temporarily ceased operations. And this was from Feb. 14 ...
Another incident was reported out here on 07/12/03 (search the page for "foonet") ... seems that 84898 spams swamped a box, and follow-up by FooNet sucked - e.g. they turned a blind eye.
There are far too many hits to return ... if you're interested in more, you can always head here. For now, I'll close with this: I do not agree with the methods used, if they were as described ... however, FooNet/CIT is no stranger to the FBI, and perhaps this is all rolled in to the Feb. 14th notice ... maybe the FBI actually gave them 10 days to comply... I'd really like to see how this ends.
I do wonder how cooperative CIT was. After several hours of requests for the info (with a warrent) the FBI must have been riled to say "F-this-S, haul it away!". Think about how much extra work that must have been. There's more to this story, pity no news service has looked into it yet.
One line blog. I hear that they're called Twitters now.
Rumors have ben flying for quite awhile that Paul (the owner) was either involved or turned a blind eye to DDoS drones on his network. Some rumors stated that he's DDoS competitors to prove the superiority of CITHosting's DDoS hardened servers.
Seeing as this "data center" seems to have been his basement, I'd bet his (lack of) logs, records, and monitoring left the FBI little choice but to seize the whole thing. And, we can assume he was uncooperative as he may have been involved or at least knoweledgeable.
The general reputation of Foonet also seemed to be a bit on the black hat side. No doubt there may have been some legitimate customers as well, but they seem to be known more for their spammers and script kiddies (and cheap shell accounts) than for their legitimate webhosting.
All in all, it looks to me like the FBI did what it had to do to effectively process the warrant. They were evidently going after a network, not a specific machine. Unfortunately, some legitimate customers got caught up in it.
It looks like CTIHosting was recently sold, and is being moved to a new data center in Chicago. Let's hope that it comes back as a legitimate business this time. They've already stated that IRC will be down indefinitely, so that's a good sign.
It is routine, however, that the FBI or police seize computer equipment and never return it. So it was reasonable to assume that this was the case here (they still haven't returned 100% of the equipment anyway). It's not obviously stated under the law one's rights when this happens, nor are there limits to how long your equipment can be held (so far as I know). This is a huge problem.
I haven't seen this story picked up on any other news outlet yet :-)
Anyway, if you are interested in knowing more, have a look at the records at SPEWS .
ciao, .mau.
Maybe you looked at the wrong sources
02/23/2004 CIT re-establishes service.
Hey, look, I tried my best, by submitting this three days ago:
2004-02-21 09:18:16 FBI confisticates (sic) ISP's servers: "more efficie (articles,usa) (rejected)
and it was rejected in about thirty minutes.
Maybe I should write more sensationalistic submissions?
But seriously folks, yeah, the FBI is returning the equipment now, but how much damage was done to an innocent ISP just because the FBI couldn't figure out how to do on-site data mining?
And if searching for evidence on a computer requires the FBI to physically cart the equipment to some distant lab, I guess we just write off any expectation that they'll be able to find data quickly in an emergency -- like, just off the top of my head here, for instance, wholly unlikely I'm sure, an imminent terrorist act?
Well, maybe a business got ruined, maybe the FBI can't scan data quickly enough to stop a terrorist crime in progress, but at least we all feel safer now that arch-criminal Tommy Chong is in jail.
Opinions on the Twiddler2 hand-held keyboard?
Of course, that was a long time ago, these days they would probably just have sent anyone suspected of having a copy of Illuminati to Guantanamo.
_O_
.|< The named which can be named is not the true named
If you consider 2600 a news outlet, then you'll be glad to know that Off the Hook spent quite some time last week talking about the incident.
--Leo
I live in Columbus, and have had the misfortune of working with foonet/Creative Internet Technologies/Creative Internet Techniques - they have called themselves all three. The small ISP which I used for my website unexpectedly moved our web site to a server at foonet. All of our mail forwarding was getting blocked by about every blacklist on the planet, and the uptime was horrendous. Needless to say, despite the 3 month prepay, we immediatly moved to another ISP. While we were being hosted at foonet, located about 10 minutes from us, I called them (local, no 800 # - ) multiple times, telling them that they were on blacklists. I never could talk to anyone, just leave messages that would go unanswered. If you are doing anything remotely important, avoid foonet/CIT like the plague. Their phone numbers are/used to be Sales - 614 353 8243 and General Inquires - 740 881 0323
Never. Hard drives are forensically examined by being removed from their machines and duplicated (usually using dd). No investigator would ever boot a machine which is the subject of an investigation - auto-deletion scripts are just too easy to write.
The FBI cart equipment away to their premises in order to duplicate the systems and environments. If ever you get into information systems forensics, they would at least perform 2 copies. One is kept as an exact duplicate (to keep for their investigation records) and at least another to actually run analysis against (since searching on an active system can change the data stored in it).
It also makes it easier to catalog what they are working with, and prevents any interference from the outside.
Doing some simple math, with a decentish disk controller, it will take 3 hours just to stream 1TB from disk to /dev/null. That assumes that the data is perfectly sequential and that no 'analysis' (such as accessing in a filewise manner, looking for a particular name of other data within the stream, etc).
Touching the data at all will easily double that to 6 hours. Add in more time because the volume is probably archival (read slower) rather than being set up as an enterprise DB system. Add even more since the server has other things to do running the business.
Most likely, what they were after was logs. Logs tend to be optimized to be stored quickly rather than for fast access. After all, logs are being stored constantly, but unless something unexplained is going wrong, they aren't analyzed at all. When they are analyzed, it's usually one of a handful of standard reports (such as logins, changes to suid, etc) and is only done over a reletivly short span of time.
Given the above, and that there were multiple TB of data to sift, it is not even vaguely reasonable to expect a complete result in less than several days.
If this report is even vaguely factual, I sincerely hope the person who made the decision to sieze is forced to spend the remaining years of his career in the basement sifting through endless lines of:
1337 d00d> D000dZ! I R s0 1337!
To the best of my knowledge, there is no posibility of an all encompassing regular expression that can translate 1337 to english.
Delete your logs. Delete them early, and delete them often. Searching through 24 hours worth of data is a lot easier then searching through 2 years worth...
"Freedom means freedom for everybody" -- Dick Cheney
But seriously folks, yeah, the FBI is returning the equipment now, but how much damage was done to an innocent ISP just because the FBI couldn't figure out how to do on-site data mining?
I'm sorry to break this to you all, but this hosting provider is far from innocent. This particular provider has been a PITA for the major IRC networks for a long time due to the amount of DoS drone nets being held on private ircds hosted by foonet. Good riddance, and applause to the feds for finally dealing with this.
Ah. That explains a lot. The anti-spam folks (including SPEWS) have been trying to bring this ISP's child-porn-spammer problem to their attention for months. It hadn't worked; the child porn stayed up on their servers and the spammers kept blasting ads for it to all and sundry -- including a very worried biologist at my site, who wanted to know why he seemed to be on some spammer's list of paedophiles?
By the time the FBI got around to investigating, the ISP had probably (as "bulletproof bulker hosting" ISPs usually do) told their spammer customer that they were taking fire. Under those circumstances, the FBI's move was probably a good one -- to keep the child-porn spammers from deleting all their files and hiding their traces.
LVD SCSI: 3.5 hours
U160: 1.75 hours
U320: 45 min
This is assuming maximum transmission speeds across a single bus. I would hope that TB of data would be on properly organized RAID arrays, and thus would span across multiple SCSI buses, and thus, creating a mirror of said data, while not cheap, should not take on the order of more than a couple of hours, provided hardware is available.
Place the cost of that against the cost of shutting you down, and it's pretty obvious which one you want. Then again, I'm astounded that the FBI would shut down a business.The cesspool just got a check and balance.
I know the Ashcroft-obsessed crowd will drown out this message, but I will say it anyway.
foo.net has, for the longest time, been protecting carders. They've been told so, repeatedly, by the anti-spam community and weaseled. My suspicion at this point is that either they are actively involved and/or some of their members are involved. FBI methods aside, foo.net isn't the innocent-victim they would have you believe.
As someone who has had multiple run-ins with Foonet and their customers over the years, I'm personally glad to see this happen, even if it's only temporary. The FBI doesn't just decide to dismantle an entire datacenter on a whim, there obviously has to be just cause. I feel that in this case, there's probably more than enough cause. If you are a (wannabe) "hacker" or "packet kiddie", Foonet is the place for you, and most people know it.
I run a large text based chat server (IRC), and as such we see frequent (D)DoS attacks. Far too many of these attacks in some way lead back to Foonet. It's even rumored that some of their employees harvest and sell Denial of Service drone networks... how's that for service! Since Foonet was raided a week and a half ago, we've seen maybe 25% of the DDoS attacks that we reguarly receive.
Bottom line... don't target "kiddies" as your primary customer base, and don't tolerate their abuse and things like this will not happen. But hey, what do I know.
I think his point was that the warrant didn't cover the other few dozen customers who also had data on these hard drives/arrays.
If the cops come busting into my local gym because somebody told them that Locker #514 has dope in it and they have a warrant to search said locker can they seize the entire bank of lockers because the owner couldn't find the key in time? Could they then charge me (the user of locker #515) if they found something incriminating in my locker when they never had permission to search it in the first place?
Think about it along those lines. What if they found pirated software (or god forbid the MS Source Code) or kiddie porn on an account that they weren't interested in and didn't have a warrant for? Can they then charge that guy or open an investigation?
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.