Slashdot Mirror
Trend Micro Bug Hits Several Important Computers
dmarx writes "The Japan Times reports that a bug in Trend Micro's antivirus software has caused the CPUs of several important computers, including those at East Japan Railway, to grind to a halt. A bug free version was released on noon Saturday." From the article: "Kyodo News experienced LAN access failure from around 8:20 a.m. to shortly before noon. The Asahi Shimbun and Yomiuri Shimbun also had trouble with their LANs at their Tokyo and Osaka bureaus, but the problems did not affect editing or printing of their evening editions."
That was East Japan Railway. The crash was on Japan Rail West.
The buggy file slowed down computer performance substantially by making CPUs run at almost full capacity, the software company said.
Sounds like every interactively-scanning antivirus program I've ever installed. I wonder, when Microsoft releases server benchmarks, if they run them with antivirus software running in the background? I think this would give a 10%-15% edge to operating systems that don't require such measures of protection.
I'm a big tall mofo.
... but in case you're wondering if this may have caused the derailment at Amagasaki, apparently it didn't. Amagasaki is located in western Japan (covered by JR-West).
Still, the coincidence in time makes me wonder. I sure hope they don't use Windows in the train system I use... just read the EULA. My life is pretty "mission-critical" to me.
I suppose the manufacturer of the faulty software is not liable in any way. Would we buy say TV sets if their Terms of use said that they are in no way guaranteed to work for the purpose they were bought for, nor are they safe to use (like exploding randomly - It's time for the penguin on the top of the TV to explode).
I understand software is a tad more complex than your average TV, but cars are not exactly simple either and they seem to work quite well (most of the time). Will we ever get software that just works or will we always have to buy something in the good faith that it will work, but if it does not, it is our tough luck?
BTW, I hope slashdotting another japanese server won't cause much additional damage...
This has nothing to do with antivirus software. The driver was driving too fast. They don't have computers that run new software like this controlling the trains!
Shame on the testers who didn't catch this.
:(
No shit! I wasted several hours trying to get my computer running again. How come they didn't?!
though I had nothing better to do anyway
This is why sysadmins should never roll out updates without testing them first. And what's even worse than non-testing is letting individual stations update directly from a vendor's site on the internet. Just asking for trouble.
With Trend Micro, viruses are the least of your worries.
Was this the issue with LPT$VPN.594?
The large bookseller I work for (think "Stables and Lords") got hit with that on Friday. All the XP machines (basically, the Manager's computers in the stores) and even a few of the XP computers in the Helpdesk (where I work) would lock up and freeze during boot.
Deleting the offending file fixed the issue.
They can prove that there are no bugs can they? That would be a neat trick.
And what's "on noon"?
How about: A fixed version was released at noon on Saturday.
You want me to trust one of those finicky and new-fangled mainframes, when my slide-rule works perfectly reliably????? WTF?
... and then they built the supercollider.
Why a bug in Trend Micro's antivirus software would appear in Eastern Japanese LANs specifically?
Does it like sushi?
Oddly, my Solaris and/or Linux and/or OSX servers are able to get by without any sort of AV protection (other than promptly installing patches). And, oddly enough, they are more stable.
Go figure. :)
"He who would learn astronomy, and other recondite arts, let him go elsewhere. " -- John Calvin, commenting on Genesis 1
Yet another example of why critical computer systems should be stripped down to the barebones tried-and-true software and isolated from any potential source of interference. This goes doubly for a system like this on which the local infrastructure depends!
There was a discussion about auto update of both definitions and scan engines being a security risk some time ago on Full Disclosure (I think it started as a Windows Update thread). This event just goes to show that software which auto updates should be used with caution and controls are required if its going to be used on critical systems, ie any updates need to be tested prior to roll out. Whether or not this can be viewed as a security incident is debatable, but software which downloads updates that cause a DOS are usually viewed as malicious. I wonder about the cruft like Plaxo (and all that other supposedly safe stuff) which download updates all the time, I can't stop it (not for technical reasons ;) but I'm just waiting for the day an auto downloaded update craps out some VP's laptop.
What I want to know is why do the computers controlling the train system in Japan need antivirus. Are they attached to the internet? Do they have disk drives? This system should have neither, I can understand the reason for a seperated system to be connected to the net for reporting train schedules and problems. But connecting a control system like that? Running it on windows? Silly. Thats worse than having antivirus on an ATM.
0x100000 hlt
The train systems are becoming increasingly automated however. For example, the older lines have open platforms, but several of the newer lines have a wall at the edge of the platform, with elevator-style doors that align with the train doors. No way to fall off the platform in that situation. I'm pretty sure they use a computerized braking system to stop the trains precisely so that that the doors line up, and probably a computerized interlock system to synchronize the pairs of doors.
Freedom = (Meaningful - Coerced) Choice != (Speech | Beer^2), and sad sock puppets' bad mods avail them naught.
Um... I really have to wonder at the QA testing that goes on at Trend Micro. It seems that there have been some pretty big screwups there that made it into their enterprise software.
In case anyone forgot this one:
Trend Micro Quarantines Letter P
Ok, pop quiz:
Your SQL server is infected with a trojan. Nevermind how, it's not important. Your manager wants to know why it wasn't protected.
You are building yourself into a glass house. Mistakes happen. They are made by your or others on your staff. You should plan for those mistakes, life has a way of teaching these kind of lessons on it's own. Typically painfully.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Exactly. This is just part of the cost of running Windows. Any serious TCO-analysis should include the cost to purchase, install and update anti-virus software on Windows.
*They wanted me to give them my root password before they would turn on my network connection. I told the nice woman that if ITS expected me to trust them with my password, surely they would trust me with the password to one of the servers. She rolled her eyes and activated my connection.
This hosed all our work computers until the update appeared. 99% CPU usage on all of them. No helpfull info on the Trend site either. Cheers guys...
Some weeks ago there was a news here about using 1 CPU just to run housekeeping software (AV, anti-spyware, firewall, and so on) and let the other for user's taks.
It seems it is not so bad idea after all (at least, for Windows users).
DNA in your Linux: DNALinux
Don't be a retard, the point of preventing intrusion is that if you do get hacked, no damage can be done because the server is so locked down it can't do anything other then act as an SQL server.
You and I have differing definitions of "locked down", and in any case, I wasn't specifically referring to trojans, I was simply using them for the example.
Shit happens. As network admin, it's your job to limit the damage using every available mean. By not using AV on all machines ( yes, virginia, linux boxes too ), you are being negligent in your duties.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
It sounds like they are two different companies, which makes it somewhat likely that they run different AV products. But all of this is guesswork; let's wait for the facts.
Ah; you mean like rip-out the Microsoft OS and replace it with a minimalized Linux kernel? I'm all for that.... ;)
Regards;
"What kind of server do you have that requires [realtime AV]?"
File servers. You know, machines whose sole purpose is for end-users to stow files on them.
If your end users are keeping all of their critical files on their workstations you need to fire your admins and get some new ones who have a clue about disaster recovery.
They are starting to make the cars so complex that it drives the cost up significantly for initial purchase, and the repair costs get astronomical because it requires a specialist in most cases to *really* fix them, but they still only last a few years before they start to break down and become uneconomical for most people. Catch 22 now. Airplanes on the other hand have high initial cost, high repairs and maintenance costs, but are designed for decades of service, not just a few years. Where are the high tech safer cards with 20 year warranties? the cost has gone up tremendously compared to when I was a kid, yet they still seem to break as much and are much harder to work on for joe average.
No easy choices for joe consumer and land transportation. It's not like you can go buy a brand new cheap car that isn't infested with all sorts of electronic stuff that isn't really necessary. It may be useful, but it's not exactly necessary. You can get older cars of course, but even then it's a high cost to restore them and in a lot of cases they have to be modified to pass emissions, which lowers their actual practicality value by introducing complexity. More stuff bolted on = more stuff to break, simple as that. I mean, new cars now cost what houses used to cost not that long ago, and they still drop in value the same as they always did, drive off the lot, whoops, several thousand gone, then it goes downhill from there. It's a cost/benefits/practicality issue that's quite complex, I don't think it can be really stated that cars are that much more of a deal now just because of all the electronic controls, which are consistently the number #1 consumer complaint with cars and repairs, the electronic control systems nowadays. Blackbox voodoo stuff that even the dealer factory trained guys have a hard time dealing with once they develop bugs.
As an admin my job is to keep the servers running acceptable and cost effectively.
Real time virus protection hurts SQL server performance. Real time virus protection hurts web server performance. Real time virus protection costs money on print servers. If no damage can be done, then why spend the money or take the performance hit?
There was discussion on this on the Full-Disclosure mailing list when posters suspected that the 100% CPU usage on their computers was because of some new unknown virus.
A repesentative of Trend Micro Germany made a post to the thread where he explained the situation, apologized for it and offered pointers to their support database so that people could get the malfunctioning virus signatures uninstalled.
Guess everyone has a ton of money to throw around securing everything so perfectly. I WISH I could get that kind of funding for securing things. I've tried and tried to show the execs the importance of it all, and in the end it still gets shot down because the allocate the money to build an addition to our complex so more people will buyin and we'll have more money to play with, which of course will go towards improving that building or.....anything but improving the infrastructure already in place or funding security. Who do YOU work for? Not everyone gets that kind of money to play with Mr. Attitude.... *rolls eyes* sheesh.....
Yes I do know this.
Physical access is restricted with a key code that is changed every week as well as a physical dead bolt.
Remote access is restricted to 3 IP's that are within the building.
User access is restricted by me and my two coworkers. If you request access, we need to how, why, when, where, and on whose authority you have permission to access said data. When you can answer all of those to our satisfaction you will be granted access to that one dataset at your own box, validated with your own login information, and set to expire exactly at the time you no longer need it.
This is our job, it is what we do. I don't know what you other people are talking about with "finding time" We get 40 hours a week to thin about this shit and implement it.
I am absolutely 100% sure that no damage can be done. I stake my job on it. We have not had an intrusion yet, (despite 500+ daily attempts, mostly from Korea).
Yes, god herself told me. Every morning I pray and ask god to help me through another day. SO far she has delivered. I think she is speaking to me when I wake up suddenly in the middle of the night and say "Shit, If I change this to this and that to that then I can make the network that much more secure" (I know it is sad, but it is true).
hmmmm. maybe you're right....on both counts
wit Mc Afee? it works well enough for me
That would be the McAfee that caused *every* NT4 box here to hang on bootup after it downloaded a corrupted dat file from our local mirror?
I am absolutely 100% sure that no damage can be done.
This would frighten me, were I your manager. People who are this sure of anything have been, in my experience, zealots for that OS or so egotistical that I don't want them making decisions.
Crap breaks, people make mistakes. I believe this to the core of my being, and I plan on it. Sure, I lose some performance, but given I can throw more hardware at that particular problem, I don't worry about it.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
Crash appears, 14 hours after the event and therefore subject to modification AND to my interpretation, due to:
1) Train driver overshot the station, so backed up.
2) This put the train a couple of minutes behind schedule.
3) The driver ran faster than allowed through a descending 70 kph right-hand curve to catch up.
4) The train derailed and slammed into an apartment block.
5) The driver survived. Many others didn't. 12 hours after the crash at least 4 people were still trapped.
6) Trend's antivirus products had fuck all to do with this.
7) Supposedly "clueful" people can't help but mention Trend Antivirus and a random train accident in the same breath. Piss on such people who giggle over the deaths of dozens as long as they can make their silly little comments.
The operating system should really prevent this type of problem. The whole purpose of the OS is to mediate access to resources such as CPU. So if one process is able to monopolize the CPU and prevent other processes from getting CPU time, then the OS has failed to do its job. (I'm not sure Linux would do a better job or not -- I've seen cases where it had similar problems.)
Software sucks. Open Source sucks less.
Actually Japan Railway East and Japan Railway West were originally owned by the government until 1987, so the chances of them using the same system architecture and products is quite high. I wouldnt rule out a connection to the train wreck so quickly.
SniperX
Isn't it sad that a program specifically written to stop problems of this kind, is the cause of this problem?
I personally don't like the idea of having an extra add-on software package, designed to plug holes in the operating system, instead of fixing the operating system itself. And now MicroSoft is planning on including one of these in their OS, instead of actualy fixing the problems!
I think the virus writers and the spammers are trying to drive personal computers into the same oblivion as the CB radio. Take something useful, and fill it so full of crap that nobody can actually make use of it.
Who would win this election: Andrew Weiner vs Andrew Weiner's weiner.
I hope you aren't a linux man, saying "You get what you pay for". The irony.
-gjr
Since my office was so seriously affected by this problem, it would be great if people could post other embarassing Trend Micro stories too!
I want a new world. I think this one is broken.
Trend Micro make a product called PC-Cillin. What I have always wondered is, why on earth would anyone use an anti-virus tool named after a drug which is famous for not working against viruses?
Je fume. Tu fumes. Nous fûmes!
Friday night I experienced the same thing. All of a sudden, my CPU usage pegged at 99% and could barely do anything. Any programs/windows I launched either took a very long time to execute, if at all. It took me a while to figure out what went wrong. After messing around with the services (services.msc), I figured out it was pc-cillin. I just disabled all of the services associated with the program and rebooted. Everything came up fine afterwards and I just did a uninstall/reinstall and now my machine is happily chugging along!
Could this be related?
Train Rams Into Building in Japan; 50 Die
Executive ability is deciding quickly and getting someone else to do the work. --John G. Pollard
... is proof that you must be smoking crack. Either that or the machine must not be as "mission critical" as you'd like to think it is.
Windows O/S is only valid for machines that need to be up and running *some* of the time.
Crashed Computer Traps Thai Politician
z 12.html)
Updated 14 May 2003
http://aardvark.co.nz/daily/2003/n051301.shtml
Thailand's Finance Minister Suchart Jaovisidha had to be rescued today from inside his expensive BMW limousine after the onboard computer crashed, leaving the vehicle immobilized.
Once the computer failed, neither the door locks, power windows nor air conditioning systems would function, leaving the Minister and his driver trapped inside the rapidly heating vehicle.
Despite the pair's best efforts, it took a full ten minutes before they were able to summon the attention of a nearby guard who freed the two men by smashing one of the vehicle's windows with a sledgehammer.
A report (http://www.bangkokpost.com/Business/13May2003_bi
published in the Bangkok Post indicates that the vehicle was Mr Jaovisidha's own BMW 520 which was being used while his state-supplied Mercedes, was being repaired.
The revolution will NOT be televised.
... OfficeScan and ServerProtect on over 700 machines and did not experience any problems over the weekend. We used to be a McAfee shop and ditched them after two years of problems and then the company failing to honor our support contract with them. We tested Symantec's enterprise virus product and could not get the evals to do the "push" install and run correctly even after a couple hours on the phone with Symantec's support. Turned out that we'd have to manually touch each and every of the 700 desktop machine with a crew of support techs to clear out the old McAfee installation and reboot each one at least 3, possibly 4 times to get the Symantec product installed. Furthermore, the Symantec/Norton AV product felt like it just subtracted 200 MHz off the CPU speed of each machine once it was installed. We were not pleased with it at all. The Trend Micro eval install just simply worked right the frict time. The push installer removed the old McAfee and installed OfficeScan automatically with only a single reboot at the end of the installation. Of the 700 desktops on out network, we had to manually touch maybe 50 of them due to odd problems. Trend has been running fine for us for over 2 years now.
If you're all that and a bag of peanuts start a security consulting firm.
I want a new world. I think this one is broken.
Trend's had some cross-product bugs in virus software before
But then so has McAfee and CA, (though the last was a licensing component at fault).
There definitely does seem to be an increasing trend in vulnerable AV software at the moment.
Virus/anti-virus aside the car-computer bugs mentioned elsewhere in the thread have had a terrible impact. Also google for the dive computer (SCUBA) which had a bug in its handling of NitrOx divers, worked out really nasty for several people diving aggressive dive plans.
I want a new world. I think this one is broken.
We got hit with this on Friday at 3:30 PST. I work for a company in Los Angeles and I was one of the first hit in the company.
We thought it was a virus and it took us about an hour and half to figure out it was OfficeScan it self that was bad.
Bet this
link Checkmark labs recently gave out an award to the company for its spyware product. Spyware, as you know, slows down computers and makes them difficult to use. Oh the irony!!!
The latest gadget news and reviews. www.absolutegadget.com
Antivirus programs cause more problems than they fix. They cause significantly degraded performance. They cause unusual and unexpected problems with legitimate software. They give a false sense of security. In the end, though, they can only really protect against known malware, days or weeks after it's a problem. A combination of user training and regular software updates is more effective, in my opinion.
Two of my customers were hit with this at the same time on Friday around 4:50pm - the only good thing about it was that it hit at a time when many of the folks most affected by the bad update had gone for the weekend. They called, described the problem, and it hit almost completely in sync, all the machines that were running the latest XP with all the patches. We spent 3 hours that night troubleshooting and eventually figured out it was the AV software messing it up - and then about 20 minutes later on Trend Micro's site they had a "you gotta update from v594 to v596" to fix it. First off, lets face some reality here - it was only a matter of time before something this scale happened - AV software, if developed by a small group and not effectively tested, could be perhaps the least QA tested software on business PC's in the world today. Remember that response time is the major factor in AV protection - and getting your signatures out faster than the other guys, and faster than the virus spreads, is about the only success that these vendors know. For a long time now I've seen shoddy work from various AV vendors - Norton steals resources, Trend leaves stuff behind after an un-install and McAfee spams their own users after install. Thus far the only two that havn't bothered me that much are Zone Alarm and Grisoft's free AVG. For the last 2 years I've asked Trend Micro, Symantec and McAfee to add a single feature into their server-based email virus protection - and that is the smarts to know when to (and not to) respond to a message with a "this message contains a virus". Right now virus responses are a binary value - you either send them or you don't. Shouldn't the AV software be able to know from it's signature whether or not the senders email address is spoofed? Anyway, I digress. What it all boils down to is that AV vendors have a huge market penetration, and if some vendors aren't QA'ing their work (or if Microsoft is restricting updates by country) then it's inevitable that something nasty is going to be spread by the AV software. Also remember that it's not just the AV software - Microsoft's last round of updates seem to have broken more than just this.
You don't even have to read the article to know the answer to this... it's in the first post! (scroll up)
That was East Japan Railway. The crash was on Japan Rail West.
Almost as ironic as something you paying for screwing you over, and something you got for free being reliable. One might even say 'You get what you don't pay for'. :)
It is a simple matter of risk vs reward
Yes, it is.
Are you 100% sure you will not be targeted for assassination by a rouge government agency
I think we will both agree that a computer on a network with other computers is at higher risk of catching something than your statement.
The reward, in terms of dollars and performance, is worth the miniscule risk we take by not running the real time protection.
Do yourself a favor, and ask this question to any manager type:
"Would you prefer to have a high performance server, or a server with slightly less performance running an AV?"
Don't even mention that you can make up the performance difference with extra hardware.
I think most reasonable types ( and managers too ) would agree that the trade off is well worth it.
Mod me down with all of your hatred and your journey towards the dark side will be complete!
I spent half of the weekend trying to fix what I though was a virus. After a system restore etc, I managed to get things working again. But then Trend found a couple of virii that had been on my machine for months and never detected before (despite daily checks). Not impressed at all, will be looking for another anti-virus solution. Any suggestions?
Fear, Uncertainty and Doubt = [citation required]
MindStalker asks and states:
What I want to know is why do the computers controlling the train system in Japan need antivirus. ... connecting a control system like that? Running it on windows? Silly.
I agree and wonder if the ensuing chaos had anything to do with this unusual and fatal accident. The engineer, of course, is being blamed for speeding. You have to wonder what was making him speed. Japanese trains usually run like clockwork.
Fifty two people died and hundreds were injured. You can see the pictures here.
DMCA, Hollings, Palladium. What might have sounded like paranoia is now common sense.
Unfortunately some of us work in small businesses, where the server room doubles as a store room for asbestos waste (don't ask) and stationary - we had to fight tooth and claw to get even that little desk, and our last great victory was a £300 UPS. We don't have 40 hours to spend on this because we are expected to do other work as well.
"If he were a plant, people would roll him up and smoke him."
video cards are to the point where they contain HIGHLY SPECIALIZED computations a bazillion times faster than they could by sharing the CPU
people are looking at the new intel dual core setups for among other things, dedicating one core to their antivirus checker, as norton lately has been bogging down the CRAP outta pc's
how hard is it to make a PCI/ISA/ slot card that is the CPU for antivirus.. yes- I propose someone build an anti-virus processor, and mount it on a card.. let it do everything that gets loaded onto the processor...
every day http://en.wikipedia.org/wiki/Special:Random