Trend Micro Bug Hits Several Important Computers
dmarx writes "The Japan Times reports that a bug in Trend Micro's antivirus software has caused the CPUs of several important computers, including those at East Japan Railway, to grind to a halt. A bug free version was released on noon Saturday." From the article: "Kyodo News experienced LAN access failure from around 8:20 a.m. to shortly before noon. The Asahi Shimbun and Yomiuri Shimbun also had trouble with their LANs at their Tokyo and Osaka bureaus, but the problems did not affect editing or printing of their evening editions."
geeez just 1 bug? they should move to DC and try the metro... THEN they can complain.. ;-)
That was East Japan Railway. The crash was on Japan Rail West.
The buggy file slowed down computer performance substantially by making CPUs run at almost full capacity, the software company said.
Sounds like every interactively-scanning antivirus program I've ever installed. I wonder, when Microsoft releases server benchmarks, if they run them with antivirus software running in the background? I think this would give a 10%-15% edge to operating systems that don't require such measures of protection.
I'm a big tall mofo.
... but in case you're wondering if this may have caused the derailment at Amagasaki, apparently it didn't. Amagasaki is located in western Japan (covered by JR-West).
Still, the coincidence in time makes me wonder. I sure hope they don't use Windows in the train system I use... just read the EULA. My life is pretty "mission-critical" to me.
Yeah, I was wondering about that. Sounds like the driver was running late and speeding to meet the schedule, though...
I suppose the manufacturer of the faulty software is not liable in any way. Would we buy say TV sets if their Terms of use said that they are in no way guaranteed to work for the purpose they were bought for, nor are they safe to use (like exploding randomly - It's time for the penguin on the top of the TV to explode).
I understand software is a tad more complex than your average TV, but cars are not exactly simple either and they seem to work quite well (most of the time). Will we ever get software that just works or will we always have to buy something in the good faith that it will work, but if it does not, it is our tough luck?
BTW, I hope slashdotting another japanese server won't cause much additional damage...
This has nothing to do with antivirus software. The driver was driving too fast. They don't have computers that run new software like this controlling the trains!
Shame on the testers who didn't catch this.
:(
No shit! I wasted several hours trying to get my computer running again. How come they didn't?!
though I had nothing better to do anyway
This is why sysadmins should never roll out updates without testing them first. And what's even worse than non-testing is letting individual stations update directly from a vendor's site on the internet. Just asking for trouble.
Jokes? You are sick.
Anyways, what's to stop the bug from affecting their system at a different location? It's not like the East and West side of Japan are some huge distance apart.
Never trust any company with the word "Micro" in their name. Seriously, "micros" have a lower standard on everything compared to mainframes. You get what you pay for.
With Trend Micro, viruses are the least of your worries.
Was this the issue with LPT$VPN.594?
The large bookseller I work for (think "Stables and Lords") got hit with that on Friday. All the XP machines (basically, the Manager's computers in the stores) and even a few of the XP computers in the Helpdesk (where I work) would lock up and freeze during boot.
Deleting the offending file fixed the issue.
They can prove that there are no bugs can they? That would be a neat trick.
And what's "on noon"?
How about: A fixed version was released at noon on Saturday.
You want me to trust one of those finicky and new-fangled mainframes, when my slide-rule works perfectly reliably????? WTF?
... and then they built the supercollider.
Why a bug in Trend Micro's antivirus software would appear in Eastern Japanese LANs specifically?
Does it like sushi?
Oddly, my Solaris and/or Linux and/or OSX servers are able to get by without any sort of AV protection (other than promptly installing patches). And, oddly enough, they are more stable.
Go figure. :)
"He who would learn astronomy, and other recondite arts, let him go elsewhere. " -- John Calvin, commenting on Genesis 1
Yet another example of why critical computer systems should be stripped down to the barebones tried-and-true software and isolated from any potential source of interference. This goes doubly for a system like this on which the local infrastructure depends!
There was a discussion about auto update of both definitions and scan engines being a security risk some time ago on Full Disclosure (I think it started as a Windows Update thread). This event just goes to show that software which auto updates should be used with caution and controls are required if its going to be used on critical systems, ie any updates need to be tested prior to roll out. Whether or not this can be viewed as a security incident is debatable, but software which downloads updates that cause a DOS are usually viewed as malicious. I wonder about the cruft like Plaxo (and all that other supposedly safe stuff) which download updates all the time, I can't stop it (not for technical reasons ;) but I'm just waiting for the day an auto downloaded update craps out some VP's laptop.
Ummm. Okay. And your point is?
No, shame on you for implying that this bug had anything to do with that accident.
What I want to know is why do the computers controlling the train system in Japan need antivirus. Are they attached to the internet? Do they have disk drives? This system should have neither, I can understand the reason for a seperated system to be connected to the net for reporting train schedules and problems. But connecting a control system like that? Running it on windows? Silly. Thats worse than having antivirus on an ATM.
Servers do not need virus protection.
WTF are you doing running real time virus protection on a server anyway? What kind of server do you have that requires it? Our SQL servers are firewalled off with connections only happening for SSH and the SQL ports.
Our web servers are HTTP and SSL only.
Our print servers are so fucking locked down you need to be an admin to do anything other then submit a job.
Seeing as most windows viruses are email related then what are you doing checkign email on your server.
The ONLY place we have real-time virus protection installed is our EMAIL server. It autoupdates from Symantec, delete any email with a known virus attachment, deletes SPAM (while not virus protection per say it sure cuts down the malicious emails), and blocks any file with a windows executable extension.
If you are running real time protection on your servers you need to fire your admins and get some new ones who have a clue about computer security.
0x100000 hlt
The train systems are becoming increasingly automated however. For example, the older lines have open platforms, but several of the newer lines have a wall at the edge of the platform, with elevator-style doors that align with the train doors. No way to fall off the platform in that situation. I'm pretty sure they use a computerized braking system to stop the trains precisely so that that the doors line up, and probably a computerized interlock system to synchronize the pairs of doors.
Freedom = (Meaningful - Coerced) Choice != (Speech | Beer^2), and sad sock puppets' bad mods avail them naught.
Um... I really have to wonder at the QA testing that goes on at Trend Micro. It seems that there have been some pretty big screwups there that made it into their enterprise software.
In case anyone forgot this one:
Trend Micro Quarantines Letter P
Exactly. This is just part of the cost of running Windows. Any serious TCO-analysis should include the cost to purchase, install and update anti-virus software on Windows.
I've really been less than thrilled with Trend's software.
*They wanted me to give them my root password before they would turn on my network connection. I told the nice woman that if ITS expected me to trust them with my password, surely they would trust me with the password to one of the servers. She rolled her eyes and activated my connection.
wit Mc Afee? it works well enough for me
This hosed all our work computers until the update appeared. 99% CPU usage on all of them. No helpfull info on the Trend site either. Cheers guys...
Some weeks ago there was a news here about using 1 CPU just to run housekeeping software (AV, anti-spyware, firewall, and so on) and let the other for user's taks.
It seems it is not so bad idea after all (at least, for Windows users).
DNA in your Linux: DNALinux
It's different companies (and presumably, different computer infrastructure...)!
It sounds like they are two different companies, which makes it somewhat likely that they run different AV products. But all of this is guesswork; let's wait for the facts.
Ah; you mean like rip-out the Microsoft OS and replace it with a minimalized Linux kernel? I'm all for that.... ;)
Regards;
They are starting to make the cars so complex that it drives the cost up significantly for initial purchase, and the repair costs get astronomical because it requires a specialist in most cases to *really* fix them, but they still only last a few years before they start to break down and become uneconomical for most people. Catch 22 now. Airplanes on the other hand have high initial cost, high repairs and maintenance costs, but are designed for decades of service, not just a few years. Where are the high tech safer cards with 20 year warranties? the cost has gone up tremendously compared to when I was a kid, yet they still seem to break as much and are much harder to work on for joe average.
No easy choices for joe consumer and land transportation. It's not like you can go buy a brand new cheap car that isn't infested with all sorts of electronic stuff that isn't really necessary. It may be useful, but it's not exactly necessary. You can get older cars of course, but even then it's a high cost to restore them and in a lot of cases they have to be modified to pass emissions, which lowers their actual practicality value by introducing complexity. More stuff bolted on = more stuff to break, simple as that. I mean, new cars now cost what houses used to cost not that long ago, and they still drop in value the same as they always did, drive off the lot, whoops, several thousand gone, then it goes downhill from there. It's a cost/benefits/practicality issue that's quite complex, I don't think it can be really stated that cars are that much more of a deal now just because of all the electronic controls, which are consistently the number #1 consumer complaint with cars and repairs, the electronic control systems nowadays. Blackbox voodoo stuff that even the dealer factory trained guys have a hard time dealing with once they develop bugs.
There was discussion on this on the Full-Disclosure mailing list when posters suspected that the 100% CPU usage on their computers was because of some new unknown virus.
A repesentative of Trend Micro Germany made a post to the thread where he explained the situation, apologized for it and offered pointers to their support database so that people could get the malfunctioning virus signatures uninstalled.
Crash appears, 14 hours after the event and therefore subject to modification AND to my interpretation, due to:
1) Train driver overshot the station, so backed up.
2) This put the train a couple of minutes behind schedule.
3) The driver ran faster than allowed through a descending 70 kph right-hand curve to catch up.
4) The train derailed and slammed into an apartment block.
5) The driver survived. Many others didn't. 12 hours after the crash at least 4 people were still trapped.
6) Trend's antivirus products had fuck all to do with this.
7) Supposedly "clueful" people can't help but mention Trend Antivirus and a random train accident in the same breath. Piss on such people who giggle over the deaths of dozens as long as they can make their silly little comments.
The operating system should really prevent this type of problem. The whole purpose of the OS is to mediate access to resources such as CPU. So if one process is able to monopolize the CPU and prevent other processes from getting CPU time, then the OS has failed to do its job. (I'm not sure Linux would do a better job or not -- I've seen cases where it had similar problems.)
Software sucks. Open Source sucks less.
Actually Japan Railway East and Japan Railway West were originally owned by the government until 1987, so the chances of them using the same system architecture and products is quite high. I wouldnt rule out a connection to the train wreck so quickly.
SniperX
Isn't it sad that a program specifically written to stop problems of this kind, is the cause of this problem?
I personally don't like the idea of having an extra add-on software package, designed to plug holes in the operating system, instead of fixing the operating system itself. And now MicroSoft is planning on including one of these in their OS, instead of actualy fixing the problems!
I think the virus writers and the spammers are trying to drive personal computers into the same oblivion as the CB radio. Take something useful, and fill it so full of crap that nobody can actually make use of it.
Who would win this election: Andrew Weiner vs Andrew Weiner's weiner.
I hope you aren't a linux man, saying "You get what you pay for". The irony.
-gjr
Since my office was so seriously affected by this problem, it would be great if people could post other embarassing Trend Micro stories too!
I want a new world. I think this one is broken.
Trend Micro make a product called PC-Cillin. What I have always wondered is, why on earth would anyone use an anti-virus tool named after a drug which is famous for not working against viruses?
Je fume. Tu fumes. Nous fûmes!
Friday night I experienced the same thing. All of a sudden, my CPU usage pegged at 99% and could barely do anything. Any programs/windows I launched either took a very long time to execute, if at all. It took me a while to figure out what went wrong. After messing around with the services (services.msc), I figured out it was pc-cillin. I just disabled all of the services associated with the program and rebooted. Everything came up fine afterwards and I just did a uninstall/reinstall and now my machine is happily chugging along!
Could this be related?
Train Rams Into Building in Japan; 50 Die
Executive ability is deciding quickly and getting someone else to do the work. --John G. Pollard
... is proof that you must be smoking crack. Either that or the machine must not be as "mission critical" as you'd like to think it is.
Windows O/S is only valid for machines that need to be up and running *some* of the time.
His point is that he can use a thesaurus.
Crashed Computer Traps Thai Politician
z 12.html)
Updated 14 May 2003
http://aardvark.co.nz/daily/2003/n051301.shtml
Thailand's Finance Minister Suchart Jaovisidha had to be rescued today from inside his expensive BMW limousine after the onboard computer crashed, leaving the vehicle immobilized.
Once the computer failed, neither the door locks, power windows nor air conditioning systems would function, leaving the Minister and his driver trapped inside the rapidly heating vehicle.
Despite the pair's best efforts, it took a full ten minutes before they were able to summon the attention of a nearby guard who freed the two men by smashing one of the vehicle's windows with a sledgehammer.
A report (http://www.bangkokpost.com/Business/13May2003_bi
published in the Bangkok Post indicates that the vehicle was Mr Jaovisidha's own BMW 520 which was being used while his state-supplied Mercedes, was being repaired.
The revolution will NOT be televised.
... OfficeScan and ServerProtect on over 700 machines and did not experience any problems over the weekend. We used to be a McAfee shop and ditched them after two years of problems and then the company failing to honor our support contract with them. We tested Symantec's enterprise virus product and could not get the evals to do the "push" install and run correctly even after a couple hours on the phone with Symantec's support. Turned out that we'd have to manually touch each and every of the 700 desktop machine with a crew of support techs to clear out the old McAfee installation and reboot each one at least 3, possibly 4 times to get the Symantec product installed. Furthermore, the Symantec/Norton AV product felt like it just subtracted 200 MHz off the CPU speed of each machine once it was installed. We were not pleased with it at all. The Trend Micro eval install just simply worked right the frict time. The push installer removed the old McAfee and installed OfficeScan automatically with only a single reboot at the end of the installation. Of the 700 desktops on out network, we had to manually touch maybe 50 of them due to odd problems. Trend has been running fine for us for over 2 years now.
Trend's had some cross-product bugs in virus software before
But then so has McAfee and CA, (though the last was a licensing component at fault).
There definitely does seem to be an increasing trend in vulnerable AV software at the moment.
Virus/anti-virus aside the car-computer bugs mentioned elsewhere in the thread have had a terrible impact. Also google for the dive computer (SCUBA) which had a bug in its handling of NitrOx divers, worked out really nasty for several people diving aggressive dive plans.
I want a new world. I think this one is broken.
?!!!
Is this for real? Why in gods green earth would a non-idiot run billy bathgatesOS, (and 'antivirus software', norton utilities, etc...) to run a railroad. Just when I think I can't be more amazed about shit like this, the next day comes along and proves me wrong.
Seriously, Microsoft, in its entirety, needs to be be surgically removed from the earth. How much are we all collectively being forced to pay for bill gates to continue lord over his damn monopoly?
We got hit with this on Friday at 3:30 PST. I work for a company in Los Angeles and I was one of the first hit in the company.
We thought it was a virus and it took us about an hour and half to figure out it was OfficeScan it self that was bad.
Bet this
link Checkmark labs recently gave out an award to the company for its spyware product. Spyware, as you know, slows down computers and makes them difficult to use. Oh the irony!!!
The latest gadget news and reviews. www.absolutegadget.com
Antivirus programs cause more problems than they fix. They cause significantly degraded performance. They cause unusual and unexpected problems with legitimate software. They give a false sense of security. In the end, though, they can only really protect against known malware, days or weeks after it's a problem. A combination of user training and regular software updates is more effective, in my opinion.
Two of my customers were hit with this at the same time on Friday around 4:50pm - the only good thing about it was that it hit at a time when many of the folks most affected by the bad update had gone for the weekend. They called, described the problem, and it hit almost completely in sync, all the machines that were running the latest XP with all the patches. We spent 3 hours that night troubleshooting and eventually figured out it was the AV software messing it up - and then about 20 minutes later on Trend Micro's site they had a "you gotta update from v594 to v596" to fix it. First off, lets face some reality here - it was only a matter of time before something this scale happened - AV software, if developed by a small group and not effectively tested, could be perhaps the least QA tested software on business PC's in the world today. Remember that response time is the major factor in AV protection - and getting your signatures out faster than the other guys, and faster than the virus spreads, is about the only success that these vendors know. For a long time now I've seen shoddy work from various AV vendors - Norton steals resources, Trend leaves stuff behind after an un-install and McAfee spams their own users after install. Thus far the only two that havn't bothered me that much are Zone Alarm and Grisoft's free AVG. For the last 2 years I've asked Trend Micro, Symantec and McAfee to add a single feature into their server-based email virus protection - and that is the smarts to know when to (and not to) respond to a message with a "this message contains a virus". Right now virus responses are a binary value - you either send them or you don't. Shouldn't the AV software be able to know from it's signature whether or not the senders email address is spoofed? Anyway, I digress. What it all boils down to is that AV vendors have a huge market penetration, and if some vendors aren't QA'ing their work (or if Microsoft is restricting updates by country) then it's inevitable that something nasty is going to be spread by the AV software. Also remember that it's not just the AV software - Microsoft's last round of updates seem to have broken more than just this.
You don't even have to read the article to know the answer to this... it's in the first post! (scroll up)
That was East Japan Railway. The crash was on Japan Rail West.
Almost as ironic as something you paying for screwing you over, and something you got for free being reliable. One might even say 'You get what you don't pay for'. :)
The person who wrote that sysadmins should test 1st, really? And just how does one test for such lousy software engineering / development?
Specifically, many sysadmins have the pattern file deployed immediately upon it's release in order to keep up with the amount of virus' being released into the wild. I used to deploy pattern files manually, but couldn't keep up with it (in order to prevent outbreaks) because sometimes, I do have other work to do, toilets to visit, or get some actual sleep. It has the ability to "auto-update" for a REASON!!! I have never had an A/V update bring down a PC completely until now. My solution Friday nite was to send out instructions to all of my sites via FAX (to go into safe more, uninstall Trend, and temporarily install AVG) and my company WILL migrate to Sophos over the next 4 days (who apparently doesn't take their 24/7 customers for granted).
after it was released (on Friday nite) with version 2.596.00.
I spent half of the weekend trying to fix what I though was a virus. After a system restore etc, I managed to get things working again. But then Trend found a couple of virii that had been on my machine for months and never detected before (despite daily checks). Not impressed at all, will be looking for another anti-virus solution. Any suggestions?
Fear, Uncertainty and Doubt = [citation required]
MindStalker asks and states:
What I want to know is why do the computers controlling the train system in Japan need antivirus. ... connecting a control system like that? Running it on windows? Silly.
I agree and wonder if the ensuing chaos had anything to do with this unusual and fatal accident. The engineer, of course, is being blamed for speeding. You have to wonder what was making him speed. Japanese trains usually run like clockwork.
Fifty two people died and hundreds were injured. You can see the pictures here.
DMCA, Hollings, Palladium. What might have sounded like paranoia is now common sense.
I work in IT at a local credit union and we were hit by the bug at about 4:30 on Friday. At first we thought it was a virus and we narrowed it down to only affecting SP2 machines...which was both a good thing and a bad thing. Most of the PCs we run at the branches are on 2000 or XP SP1, but the IT department is running XP SP2...needless to say, that caused a few problems. Luckily, one of the guys in the department that has been here for a while never wanted to upgrade to XP because he was used to Win 2000 and doesn't like change, so we were able to run our nightly operations without any problems. Either way, it made for an interesting night.
video cards are to the point where they contain HIGHLY SPECIALIZED computations a bazillion times faster than they could by sharing the CPU
people are looking at the new intel dual core setups for among other things, dedicating one core to their antivirus checker, as norton lately has been bogging down the CRAP outta pc's
how hard is it to make a PCI/ISA/ slot card that is the CPU for antivirus.. yes- I propose someone build an anti-virus processor, and mount it on a card.. let it do everything that gets loaded onto the processor...
every day http://en.wikipedia.org/wiki/Special:Random