Some Companies Don't Care about Web Defacement

An anonymous reader sent in an interesting link to a story that talks about companies that just Don't care about Defacement. The story is just a light think piece worth a glance. And hell, its the holidays so its not like anything else interesting is gonna turn up to read for a few days :)

Comments

Sounds familiar...

[Chuck Milam]

Gee, this sounds just like a certian company I work(ed) for. They were getting all proud when they bought a package that detected defacements and automatically copied a "known good" version of the web page back in place. Of course, I'm kind of a low man on the totem pole, so my idea of plugging the security holes, so there's no defacement in the first place has yet to make it past my next-level management.

Some take it too far though.

[rmadmin]

I knew a kid in high school that stumbled onto a permissions mistake or something along that lines, he backed up the html, threw up a defacement, and went 'Hahahaha'. A week later the FBI was trying to put the smackdown on him saying that 'By defacing the (Small, 200 customer) ISP's webpage he caused them $17,000 in business and damages'. So a small ISP like that loses $17,000 in business in 4 hours? Unlikely... So does that mean when someone DoS's my workstation and I can't access apache from home for more than 15 minutes I've lost $1062.50?

Re:Some take it too far though.

[dillon_rinker]

My ISP business website has been defaced.

(1) Obviously, there's a security breach. How widespread is it? We need to audit the network and see how severe the breach is and what hole was unpatched. I've got to put either employees or consultants onto it.

(2) We can't trust any code on our network, so the other copy of the web site on this other server may be bad, too. We'll have to check that against a known good copy, which means looking at our backups. Really, we need a known-good historical copy, too, just to be sure, so we've got to pull our off-site backups of the web site from records management vendor.

(3) One of our business clients saw the defaced web page and decided that they didn't trust us to protect their data. They will no longer do business with us. We have lost all of the income they would have provided forever.

(4) As part of our immediate security response, we had to shut down briefly. If someone had hacked our server, they might be trying to punch through to our client machines. Not a huge deal, but we had to issue a month's credit to everyone who complained about being unable to connect.

Add together 1-4, and I think you could easily come up with $17,000. Think about 2-3 net admins + 1 security consultant doing security cleanup for a week.

So does that mean when someone DoS's my workstation and I can't access apache from home for more than 15 minutes I've lost $1062.50?
No, because you are not a business concern. Note that the four hour downtime doesn't mean that all the costs were incurred in that four-hour timeframe. The ongoing security audit that becomes necessary in the event of a hacked server could have gone on for a week.

Are the figures inflated? Possibly. Did the idiot cost the business money? Certainly. Is the FBI playing hardball with the idiot who did it? Undoubtedly. You seem to be missing the point that your friend shouldn't have done it; instead, you are whining that the FBI talked mean to your friend.

Re:Some take it too far though.

[gnovos]

My ISP business website has been defaced.

(1) Obviously, there's a security breach. How widespread is it? We need to audit the network and see how severe the breach is and what hole was unpatched. I've got to put either employees or consultants onto it.


No, this is just a shifted cost. Since you DIDN'T pay to secure your network at the beginning (either through poor-quality admins or by not paying for intrusion detection tools, whatever), you are paying now. This is not a cost that you are suddenly having to pay, this is a cost that you didn't pay in the past when you should have. If a resturant decides that they don't want to pay for a new oven, does that mean that the fire inspector "costs thier business" when he says it's out of code and needs to be replaced?

All of your arguments stem from the same problem. If you are going to base things on your security and up-time (such as your policy to pay back a month's salary to those who couldn't connect), then you had better damn well make sure that you *won't be going down*. If you have a business model *based* on security then you can't *skimp* on security, it's common sense.

Dead On...

[Bonker]

Sayeth the article:

What I am speaking of is investigating and prosecuting the criminal element involved in the act of defacement, root compromise or infection by "worms". In otherwords, companies tend to "fix & forget".

Actually, this is probably the stance that every serious IT department out to take. If your website was cracked, then it's almost certainly *your* fault your server was compromised. There just aren't any rootkits out there that don't exploit known buffer-overflows or other bugs. There are a few situations when this is not the case, but it's usually still someone sitting around testing a web application (like Slashcode) for buffer overflows or back doors.

Even if you do prosecute, it's like stomping cockroaches. There will just be more, and if you hadn't left the food out on the counter to rot, they wouldn't have come to your apartment in the first place.

Finally, there's the human element to contemplate. We all did stupid stuff when we were kids, which most website vandals are. I don't know any kid who didn't tresspass or vandalize property at least once during their youth. For many, it was the old junkyard or the cemetary. For these kids, its websites. Are you really going to put them in prison for decades because they're young and stupid? You might as well ruin their lives for experimenting with drugs or sex....

Oh wait. We do that too. Nevermind.

Re:Dead On...

[Snowfox]

Finally, there's the human element to contemplate. We all did stupid stuff when we were kids, which most website vandals are. I don't know any kid who didn't tresspass or vandalize property at least once during their youth. For many, it was the old junkyard or the cemetary. For these kids, its websites.

Maybe my experience was different from others', but - as a kid - I stopped experimenting with stupid things once I was caught. I kept doing bigger and more risky things until I finally got in trouble, and I realized that I wasn't the smartest guy in the world, and that rules weren't just for other people.

Nailing a kid for defacement now might mean that he doesn't need to be nailed for something much more serious later on.

Re:Dead On...

[Bonker]

Just because I forgot to lock my door doesn't mean I've invited you into my house. Unauthorized access is just that, unauthorized. Once little shit 'kids' recognize that every computer connected to the Internet isn't put there for you to hack into or DOS, the world will be a much better place.

Hmmm.... No, but you're pretty stupid if you don't lock your door... or replace your locks if they're recalled.

You're not considering the relative seriousness of the crimes here. If someone breaks into your house and steals your stuff or kills your pets, then yeah, you wanna press charges. If they spraypaint or break your windows... maybe.

How about if they stomp your flowerbeds? Or rearrange your rock garden to spell out dirty words? How about if they egg your door or toliet-paper your trees?

You need to think about that, because that's the mental level that most kids who vandalize websites are working on. (Show me a person who's never done at least one of these things, and I'll show you someone who was very sheltered as a child.) They're not hurting anyone, at least in their own minds. They're doing the equivalent of dropping a big nasty stink-bomb on your front porch.

You don't put kids in prison because they're being mischevious, regardless of what John Ashcroft tells you. You tell them that what they were doing is wrong, give them incentive not to do it again, and then let them get on with life.

Unfortuneately, police don't have the option of giving script kiddies a 'firm talking to', since any kind of computer crime has been labeled 'terrorism' by both our corporate oligarchy and our reactionary government.

And this is surprising why?

[dirk]

This stuff doesn't surprise me at all. Companies are in the business of making money. If they report every intrusion that happens, that means other people find out about them (potentially). If people find out, they may be less likely to use that company (or their website or whatever) than if they believe there was never a compromise. I think companies should be forced to report it when there is a compromise that includes user information or something like that, but if it is just a web-site defacement (with no possibility of anything else) I would probably not let it get out either. Add onto that fact that some PHB automatically will assume it is the admins fault, even if they were told not to patch it/didn't have enough money to do it right/were ignored on their suggestions, that measn the less people who know about the exploit, the better off you are. I don't agree with the policy, but it is certainly understandable.

Yep, this isn't unusual at all.

[Anonymous Coward]

For professional reasons, I'm posting this anonymously.

I've worked at one or two places where boxes have been cracked and once the initial panic settled down the word that came down from On High(tm) was to quietly pull the system, disinfect it (but not reformat/reinstall), and return it to service. "This system needs to be available for the developers, we don't have time for you to find whomever did it."

Needless to say, I wasn't real happy at the prospect of putting a questionable system back into active duty. Just because you found the /usr/lib/.../31337^k17 directory and copied back the files replaced by the rootkit does not mean that you've found every last trojan horse or old config file. I'm surprised that the more intelligent kiddies havn't started doubling up their rootkits yet - one which acts as your basic rootkit, replacing system binaries et al, and a second in an entirely different location that they leave in place for situations just like this: If the primary rootkit is removed but the system isn't reinstalled, they've still got a way back into the system and a backup toybox to get revenge with. It wouldn't take much at all.

Not to rip on Redhat exclusively, but with all the RH servers popping up these days I'm surprised that the newer rootkits aren't being passed around as .rpm files. No muss, no fuss, but the sysadmin would still notice if (s)he did a verification from the install CD-ROM.

At the end of all of it, I did what they asked me to and put the box back into service. I'm reasonably sure that I swept the system clean but you can't prove a negative, you can only state a negative to within a certain tolerance. For all I know, the backed up system binaries I'd found and put back into place were trojans as well and the originals had long since been overwritten.

But that's in the past now.

Statistics *are* collected

[tshoppa]

CERT/CC has been gathering statistics on incidents, vulnerabilities, security alerts, and hotline calls for over a decade now. They also analyze the statistics for trends, present courses on security issues, and publish reports for general consumption.

To me, the real problem is that every couple of months folks come along like internet security is something new, when in fact the exploits and vulnerabilities of today are very much like the same problems from a decade ago.

And maybe not

[r_j_prahad]

I think a lot of companies would care if they could afford to, they've just made a business decision not to go after this sort of thing. Investigations can take months, and prosecution can take years. What responsible CEO would be willing to commit those resources to a process that won't yield a cash return? How much money do you think Intel got back from Randall Schwartz?

I, for one, cannot afford to have my servers collecting dust in an evidence locker while I rearrange my business schedule around interviews, depositions, and testimony. Sorry folks, but yes, I'd bury it and forget it.

Re:Simple solution

[Hiro Antagonist]

How about running web servers booted off cd-rom getting all of their content dynamically by calling java servlets against a remote machine using the secure xfer methods covered in yesterday's secure credit card transfer discussion?

There are a couple of good reasons why this unlikely to be a workable solution. First, this requires almost double the equipment (a two-tier minimum), and it requires the front-end servers to have some type of read-only storage, which most server appliances (like the Netra X1) don't have.

Second, keeping the systems patched and up-to-date (which will still be imporant) is even more of a chore, as you can't just install patch foo -- you need to install the patch on a clean system, make a bootable CD, and then go physically insert the CD and reboot the machine to install the patch. In terms of administrator time, this is completely unacceptable.

Third, it requires that you use JSP (and possibly EJB); things like PHP and Perl won't work with this kind of set-up. As nice as JSP+EJB can be for building complex and stateful web applications, it's really lousy for doing simple things like customer-feedback forms and the like.

Fourth, the applications on the second-tier server are still open to exploit, as is the OS on the external server -- it's possible to crack and root a machine even if it has a read-only root filesystem.

Fifth and finally, it completely violates the KISS principle (Keep It Simple, S*). More machines means more overhead for the admins, higher operating costs; and, most importantly, a more complex system. One of those little rules-of-thumb is that the more complex a system becomes, the more easily it will fail.

Something like a serial cable into the "servlet server" with a non-TCP/IP listener on the serial port. At max speed 115KB serial is like a 1Mbit connection. The web servers won't have IP access to the content server, and can't be defaced. Don't have to care about snort logs, tripwire -- all that happy hoo ha.
Want to run a bunch of web servers for load balancing? put an 8-port digiboard in the servlet server.


I fail to see where a 115Kb/s serial connection is equal to a 1Mb/s link; I would suggest checking the numbers again, as I'm pretty sure that the latter is about ten times as fast as the former, and requires less processor overhead -- serial connections consume much more CPU time than ethernet ones.

Snort and tripwire are very useful tools, and whether or not you have a "secure" setup, it's a good idea to run them. Snort is an extremely capable IDS (Intrusion Detection System), and if your uebersecure system is cracked, can provide valuable logs to find the attacker (and the original security hole). Furthermore, it's always a fun thing to watch the IIS exploit attacks pile up against your smug little Apache server...

HTH. HAND.

Happened here too...

[tsmit]

Surfing around my intranet at my last job, found an internal test webserver 0wn3d by poisonbox. Nobody in the company gave a shit.

That is, until, i sent a message to the CEO, COO, and CFO with their credit card information. Apparently there were credit cards and user information stored on this machine.

They started to care then. Just a bit though. Of course, two months later, we were one of the companies that had to shut down EVERYTHING due to Nimda.

They're out of business now. Take that for what it's worth.

Too hard to prosecute

[greensquare]

The problem is that it is too hard to prosecute.

The FBI is way too busy with the real bad guys, like Bin Laden. You should go check out Gibson's story about the DOS attack that he was subjected to, and the results of his attempt to get the law involved. Basically, if your damages are less the $20,000 they don't care, and if the alleged hacker is less the 18, they probably don't care. It may be very hard to put a value on a webpage defacement that will hold up in court. Courts don't like to do much to kids either.

To make a long story short, it only makes sense to not throw good money after bad by trying to apprehend and prosecute someone. The effort on behalf of the corporation will be better spent shoring things up to prevent it from happening again.

Cheers!

gs

Re:Simple solution

[LinuxHam]

Hiro, nice shredding!

this requires almost double the equipment (a two-tier minimum)

you normally have 3 tiers in professional ebusiness configurations. web servers, business logic, and database servers.

patch a clean system, make a bootable CD, and insert the CD and reboot the machine. this is completely unacceptable

I think we're looking at it from two different angles. You appear to be approaching it from a datacenter admin point of view, like a Qwest rack monkey watching 1,000 servers. My approach imagines an admin with about 20 servers for one e-business/e-commerce solution. If it's one guy's job to keep maybe 8 web servers, three or four servlet engines, and four database backends running, then occasionally publishing a new CD for the web servers is not "completely unacceptable". Plus, with multiple servers, you design one clean layout, burn 8 CD's, and reboot the web servers one at a time so the site never goes down.

the second-tier server are still open to exploit

if there is no IP connectivity from the web servers to the 2nd and 3rd tier, how are you going to get there? the web server would submit an ascii url to the servlet engine, and the servlet engine would reply with the content, also over serial. the web clients won't even have access to sending url requests over the serial line. even if they crack the box, LIDS will let you specify precisely which apps/binaries can use the serial port.

it's possible to crack and root a machine even if it has a read-only root filesystem.

www.lids.org - can't get root if root isn't even root

I fail to see where a 115Kb/s serial connection is equal to a 1Mb/s link

you're right. I'm an idiot. Need more coffee. that makes the whole thing too slow for anything over 128k upstream.

One of those little rules-of-thumb is that the more complex a system becomes, the more easily it will fail.

that of course depends on how well you plan and implement.

Management education of the legal consequences

[satch89450]

After reading the link for this story, I was amused to see that things really haven't changed in a number of places. Management doesn't worry about Web site security until it hits them where it hurts, their liability insurance premium, or when the executives spend some time in the cooler.

The majority of defacements I've seen described involve little more than vandelism, electronic tagging by lower lifeforms of script kiddies, that do very little harm to the company whose site is defaced. You "wash the walls" and go on. End of story.

Except that it isn't the end of the story.

What happens when the defacer decides to use your Web site to store a couple hundred cracked credit card numbers? How about the 600 MB of MP3s of copyrighted music material that appears in its own directory of your Web server? The kiddie porn? Can you imagine what would happen if a terrorist cookbook were to be uploaded to your site, given today's paranoia caused by the November 11 terrorist attack?

IANAL, but I recall the Mogur-BBS debacle when a BBS system was used to traffic in telephone calling card numbers. Some facts are missing from the account the link points to, but it's sufficiently accurate to be useful. Here is another account of the incident. Here is a more thoughtful retrospective and analysis.

Shall I bring up the episode of Steve Jackson Games as an indication of the kind of risk that operators of public computer systems face when security is not a primary concern? Steve Jackson Games is apparently alive and well (and probably mad as hell about being mentioned in a Slashdot article) so the news isn't all bad, but the six months they were effectively out of business -- the publishing business -- must have hurt and hurt badly. Granted, the Secret Service has learned much since that 1990 fiasco, but can you imagine the long arm, and the long flatbed truck, coming and taking your computer systems because of the acts of some malicious script kiddie who does more than tagging?

Can your company afford to have its Web servers siezed and perhaps damaged because of the illegal acts of non-employees?

What you can do: tell your manager to contact your company's general legal counsel and request they research the legal liability, and the practical effects of law enforcement action, resulting from illegal acts committeed on public servers that have inadequate security controls. Emphasize that the research include short-term effects such as equipment seizure and forceable removal, damage inflicted during such action, and the expense of obtaining the timely return of the equipment.

If you run an e-commerce site, also be sure to ask about legal exposure in the event any web server containing crdit card records, customer information records, order histories, or credit search information is compromised and the information released to unauthorized people.

Steve Jackson Games was almost put out of business based on a bogus rumor. How would your company survive the legal onslaught from a script kiddie interested in more than just defacement?

His "solution" is wrong

[drew_kime]

There are two opposite sides to every debate. I am sure a middle ground is obtainable where everyone, well almost everyone, can meet and appease the majority of those concerned. Frankly, that's why it's called a "democracy". Without two opposing views, at an equal distance apart, a logical solution would be oppressed by the single minded behavior of an individual dominating force.

No. The reason it's called a democracy is because people get to vote. If there are in fact three sides to a debate, there is the distinct possibility that no one will be appeased. In fact, most compromise among reasonable people results in everyone being equally displeased, but willing to accept it.

Insisting on seeing every disagreement as a matter of two opposites is how we got the Republicans and the Democrats, with no (okay, little) room for third parties. I can't see how applying the same method to computer security will somehow suddenly work.

Re:Simple solution

[Hiro Antagonist]

Hiro, nice shredding!

Thank you; and double thanks for taking it well and coming up with a good rebuttal. So rare on /. these days...

you normally have 3 tiers in professional ebusiness configurations. web servers, business logic, and database servers.

This is true with JSP-based system (JSP+Web to EJB to DB), but often smaller setups are done with Perl or PHP in a two-tier system (Web+PHP/Perl to DB) that work quite well.

You are quite correct, however, in that most large installations use the three-tier model.

I think we're looking at it from two different angles. You appear to be approaching it from a datacenter admin point of view, like a Qwest rack monkey watching 1,000 servers. My approach imagines an admin with about 20 servers for one e-business/e-commerce solution. If it's one guy's job to keep maybe 8 web servers, three or four servlet engines, and four database backends running, then occasionally publishing a new CD for the web servers is not "completely unacceptable". Plus, with multiple servers, you design one clean layout, burn 8 CD's, and reboot the web servers one at a time so the site never goes down.

Speaking as a sysadmin, keeping one Unix admin around per twenty servers will get very expensive. One Unix admin can handle about fifty machines, assuming they were properly set up and documented to begin with.

Furthermore, one of the big advantages to running a Unix machine for things like this is that you don't need to physically interact with the hardware; for example, I can leave several "extra" Sun Netra X1 server appliances sitting in a rack, powered off, and if one of the production machines fails, I can remotely power the unit on, load an operating system on it (via Jumpstart, or just using dump and netcat), boot it, and configure it to take the place of the now-dead server (which I have powered off remotely). All without leaving my desk (or armchair if I'm telecommuting). I can then replace the dead server at my leisure.

Same goes for patching; I can bring a spare server online, bring the old server down to single user mode, and use the serial console to load patches and updates, all without having to drive over to the colocation facility.

if there is no IP connectivity from the web servers to the 2nd and 3rd tier, how are you going to get there? the web server would submit an ascii url to the servlet engine, and the servlet engine would reply with the content, also over serial. the web clients won't even have access to sending url requests over the serial line. even if they crack the box, LIDS will let you specify precisely which apps/binaries can use the serial port.

Point; but giving that serial links aren't sufficiently fast, it's a moot point at best.

you're right. I'm an idiot. Need more coffee. that makes the whole thing too slow for anything over 128k upstream.

Happens to all of us. And I think I'll get more mud myself...

that of course depends on how well you plan and implement.

Not really; a more complex solution offers more total points of failure; even a well thought-out and well implemented solution is subject to this simple fact.