Trend Micro Bug Hits Several Important Computers

dmarx writes "The Japan Times reports that a bug in Trend Micro's antivirus software has caused the CPUs of several important computers, including those at East Japan Railway, to grind to a halt. A bug free version was released on noon Saturday." From the article: "Kyodo News experienced LAN access failure from around 8:20 a.m. to shortly before noon. The Asahi Shimbun and Yomiuri Shimbun also had trouble with their LANs at their Tokyo and Osaka bureaus, but the problems did not affect editing or printing of their evening editions."

Comments

Before the flury of obvios train crash jokes start

[Anonymous Coward]

That was East Japan Railway. The crash was on Japan Rail West.

Sounds familiar.

[bigtallmofo]

The buggy file slowed down computer performance substantially by making CPUs run at almost full capacity, the software company said.

Sounds like every interactively-scanning antivirus program I've ever installed. I wonder, when Microsoft releases server benchmarks, if they run them with antivirus software running in the background? I think this would give a 10%-15% edge to operating systems that don't require such measures of protection.

Re:Sounds familiar.

[bmalek]

This sounds like a study I recently read about the poor performance of Apache vs. IIS. If you read between the lines you find out that the reason why the Apache server performed so poorly is because it was using PHP as a module instead of being compiled into the server. Well duh, of course the Apache server is going to perform worse that way... As the saying goes: 'Lies, damn lies, and statistics' - Benjamin Disraeli

Re:Sounds familiar.

[biglig2]

No viruses on BeOS. Actually, no virus checkers either...

Re:Sounds familiar.

[Will2k_is_here]

No viruses on BeOS. Actually, no virus checkers either...

No users either...

Re:Sounds familiar.

[barzok]

Neither ASP nor ASP.NET are "compiled into" the web server itself - requests for ASP files are passed to ASP.DLL and ASPX is handled by the ASP.NET worker process. Both can be removed from the IIS configuration if desired, I'm pretty sure, using the same mechanism by which one installs the PHP processor (DLL) into IIS.

Re:Sounds familiar.

[cortana]

Overall, sounds like the ideal server platform to admin. :)

Re:Sounds familiar.

[Anonymous Coward]

The different he's talking about with PHP is using mod_php as opposed to php.exe. If Apache uses mod_php, it goes out and hits php4.dll just like your asp.dll. If it's not using mod_php, it's going out and executing "php.exe %1" every time you hit a PHP page, waiting for the result, then sending it to the browser. This is much slower than the DLL approach.

You just need mod_php compiled in to Apache (the equivilent of ISAPI), *not* all of PHP, for this to work.

Re:Sounds familiar.

[jeffmeden]

no virus checkers huh... how can you be remotely certain then, that there are no viruses?

Re:Sounds familiar.

[bluGill]

Lack of apps just makes it even easier to admin. None of the non-existant users will be asking for any of the non-existant applications to be upgraded. Nor will they be getting any of the non-existant viruses.

The big problem with being a BeOS admin is there is no money in it. Otherwise it is perfect.

Re:Sounds familiar.

[biglig2]

Because if anyone had written a BeOS virus it would be announced on BeBits - they need to get the application numbers up somehow.

I expect 100 posts like this.

[muyuubyou]

... but in case you're wondering if this may have caused the derailment at Amagasaki, apparently it didn't. Amagasaki is located in western Japan (covered by JR-West).

Still, the coincidence in time makes me wonder. I sure hope they don't use Windows in the train system I use... just read the EULA. My life is pretty "mission-critical" to me.

Re:I expect 100 posts like this.

[shanen]

I think it may be too early to rule out any connection to the fatal derailment. There is some preliminary evidence that the engineer may have been pushing in an attempt to get back on schedule--and the delays may have been indirectly related to the train delays mentioned in this article.

However, I admit that it was more likely due to his youth and inexperience. He was 23 and had less than a year handling the trains--but they also need to reconsider any external factors that may have helped pressure him to make the fatal mistake.

On the main topic, I'm not sure why Virus Buster is not being mentioned here. One of my Japanese co-workers said that was the affected product. I think they may have been acquired by Trend Micro, but it's still marketed under that name (written in katakana), and I think it is still the top anti-virus product in the Japanese market. I worked in Akihabara some years ago, and it was definitely quite dominant at that time.

Who's to blame

[janek78]

I suppose the manufacturer of the faulty software is not liable in any way. Would we buy say TV sets if their Terms of use said that they are in no way guaranteed to work for the purpose they were bought for, nor are they safe to use (like exploding randomly - It's time for the penguin on the top of the TV to explode).

I understand software is a tad more complex than your average TV, but cars are not exactly simple either and they seem to work quite well (most of the time). Will we ever get software that just works or will we always have to buy something in the good faith that it will work, but if it does not, it is our tough luck?

BTW, I hope slashdotting another japanese server won't cause much additional damage...

Re:Who's to blame

[Vo0k]

Let me wake you up.
Car manufacturers fight really hard to stop this from getting more of media attention, but modern cars are known to have SERIOUS software bugs. Just google car software bug or similar for stories and references - running 100MPH down a motorway and have the engine switched off, everything shut down (and even the steering wheel blocked), or having the central lock imprison you in the car, so you can't get out, or having random pieces of equipment (wipers, windows, chair adjustment) to start at random... These are real stories. Cars aren't what they used to be...

Re:Who's to blame

[Analogy Man]

Cars aren't what they used to be...

And that is a good thing...despite these software glitches cars are SIGNIFACTLY safer today due to computers:

• ABS Braking
• Structural Analysis software
• Vehicle dynamics / handling simulation
• CFD analysis for tires (they are quite efficient pumps really)

If cars are going to go fly by wire they need to be tested and maintained like airplanes instead of like disposable consumer electronics...but in balance computers have made cars safer.

Re:Who's to blame

[Patrik_AKA_RedX]

Software design is still a pretty young field of construction. Building construction has had more than 2 millenia to develop, while software design had about century (give or take a decade). In the early days (read: centuries) buildings were designed by rules of thumb. Only the last few centuries the real science of contruction was developed. (The metalurgical properties of steel wasn't researched until after WW2 when they figured out that welded ships couldn't handle the extreme cold of northern seas very well) In software design we're at the point where we're trying to come up with the science, but are still mostly using rules of thumb.

Given time software will reach a point where it's about as reliable as concrete buildings, but in the mean time we'll be stuck with the many kinds of blue screens.

Re:Who's to blame

[kfg]

RyanFenton, posting in the computerized cars for traffic control thread:

I'd MUCH rather trust a reasonably engineered computerized system than the thousands of other drivers around me on my way about town.

I didn't post there, but my very first reaction on reading was:

"And just where the hell do propose to find one of those?"

This story illustrates my reaction. Imagine thousands of cars around you on your way about town that have suddenly lost all control.

Without the introduction of computers cars are actually not that complicated. They consist of a relatively few number of parts mechanically linked in such a way that any child can intuitively grasp their operation. You can teach yourself a fair amount of auto mechanics through entirely empirical methods, just sitting down with the device, taking it apart, putting it back togehter, and grasping how the whole thing works by such observation.

Nobody's going to write a virus checker that way, or a car control system. The computer is too complicated, consisting of billions of invisible "parts" whose operation is entirely abstracted from their function.

To the extent that cars are complicated these days, to the further extent that even formally trained mechanics cannot figure out what's wrong with them without plugging them into a computer, it is because they now contain. . .computers.

So refering to cars as an example of something that's complicated but reliable is not factual ( and I myself have found myself sitting by the side of the road with a mechanically sound car that refused to run because a control chip died), but also begs the question.

KFG

Re:Who's to blame

[greed]

The surge results in a voltage drop on the +12 rail of the "good battery" car. It's trying to bring the dead battery up to the exact same voltage, within the current limiting effect of the jumper cables. Lead-acid batteries have a very low internal resistance, so they won't slow things down much. (And that's how you get 800 "cold cranking amps" out of 'em.) A dead battery will be between 11.8 and 12.2 volts, and the good system should be up around 13.2 to 14 or so, depending on the regulator.

Many computers need to have /RESET held low for a few would-be clock cycles after power-up, to allow the power rails to stabilize and the master oscillator to start. Usually this is done by a capacitor which slowly (comparatively) charges up to supply volatage; when it crosses a certain voltage, it releases /RESET (they're usually active-low), and the CPU can start.

All well and good...

If you've got a situation where the power rail drops suddenly, the capacitor on /RESET starts to discharge to the power rail. Enough, and it activates the /RESET line on the CPU. Even though the power drop wasn't enough to wipe out the CPU, it was able to trigger the power-on-RESET circuit. (The fix is to put a diode in the computer's power supply connection, so that the computer's power supply capacitors never try to bring the +12 rail back into spec.)

Another fun thing that can happen, though probably not in automotive circuits, is GND and Vcc inversion.

This used to happen a lot on Amigas with defective monitors; you'd get a high-voltage discharge in the monitor to the GND line, which would momentarily bring GND over Vcc, triggering a /RESET. The fix there is to separate shield ground from signal ground; or you could just go bankrupt.

Given the number of modern cars which, apparently, tell you not to jump-start, there is an awful lot lacking in modern automotive design. It's not hard to cope with a jump-start, you just have to not cut all those corners.

(My 1998 Subaru has no such warning; I've only heard about that warning from GM owners--I've never seen it myself.)

Re:Who's to blame

[jamesl]

Cars aren't what they used to be...

You never drove a 1967 Jaguar. Electrics by Lucas -- the Prince of Darkness.

Re:Tragic. That's the word to describe this

[commodoresloat]

This has nothing to do with antivirus software. The driver was driving too fast. They don't have computers that run new software like this controlling the trains!

A lesson here.

[Anonymous Coward]

This is why sysadmins should never roll out updates without testing them first. And what's even worse than non-testing is letting individual stations update directly from a vendor's site on the internet. Just asking for trouble.

New sales slogan

[Alien Being]

With Trend Micro, viruses are the least of your worries.

LPT$VPN.594?

[Anonymous Coward]

Was this the issue with LPT$VPN.594?

The large bookseller I work for (think "Stables and Lords") got hit with that on Friday. All the XP machines (basically, the Manager's computers in the stores) and even a few of the XP computers in the Helpdesk (where I work) would lock up and freeze during boot.

Deleting the offending file fixed the issue.

Bug free?

[taobill]

A bug free version was released on noon Saturday.

They can prove that there are no bugs can they? That would be a neat trick.

And what's "on noon"?

How about: A fixed version was released at noon on Saturday.

Can anyone explain?

[0olong]

Why a bug in Trend Micro's antivirus software would appear in Eastern Japanese LANs specifically?

Does it like sushi?

The problem with AV

[Fished]

Antivirus checking is, by nature, an invasive procedure. Is it really surprising that these products have such a lousy reputation for impacting system stability?

Oddly, my Solaris and/or Linux and/or OSX servers are able to get by without any sort of AV protection (other than promptly installing patches). And, oddly enough, they are more stable.

Go figure. :)

Re:The problem with AV

[mikeumass]

Less market share. Windows is a much more apetizing market. Especially since most users wouldn't know if they had a trojan in the first place. How many people actually renew thier subscriptions with Norton or NA?

Re:The problem with AV

[Deffexor]

I actually ran into this problem at a customer's site this weekend. They had Trend Micro AV and the computer was utterly crippled. It was like it had some utterly malicious virus on it gobbling up all the cpu time.

Using SysInternal's Process Explorer, I was ultimately able to see that a module (running as a part of the "system" process) called "TmXPflt.sys" was running 4 simultaneous threads each using about 25% of the CPU. Since the "system" process is given higher priority than all other processes, the system naturally slowed to a crawl.

I rebooted into safe mode and renamed this file and restarted. The system behaved like normal again. The file said it was a Trend Micro "XP Post Filter" (mail filter?) - After all that, I thought that it was particularly weird that I hadn't read about some problem from Trend Micro on a major news outlet (like Slashdot) :-)

Antivirus software on mission critical computers?

[mferrier]

Yet another example of why critical computer systems should be stripped down to the barebones tried-and-true software and isolated from any potential source of interference. This goes doubly for a system like this on which the local infrastructure depends!

Auto Update of Antivirus IS a secuirty risk

[csk_1975]

There was a discussion about auto update of both definitions and scan engines being a security risk some time ago on Full Disclosure (I think it started as a Windows Update thread). This event just goes to show that software which auto updates should be used with caution and controls are required if its going to be used on critical systems, ie any updates need to be tested prior to roll out. Whether or not this can be viewed as a security incident is debatable, but software which downloads updates that cause a DOS are usually viewed as malicious. I wonder about the cruft like Plaxo (and all that other supposedly safe stuff) which download updates all the time, I can't stop it (not for technical reasons ;) but I'm just waiting for the day an auto downloaded update craps out some VP's laptop.

Why AntiVirus?

[MindStalker]

What I want to know is why do the computers controlling the train system in Japan need antivirus. Are they attached to the internet? Do they have disk drives? This system should have neither, I can understand the reason for a seperated system to be connected to the net for reporting train schedules and problems. But connecting a control system like that? Running it on windows? Silly. Thats worse than having antivirus on an ATM.

Re:Why AntiVirus?

[guy-in-corner]

Even if a computer system isn't connected to the Internet, you can guarantee that -- if it's connected to any kind of network infrastructure -- some idiot is going to jack their laptop into it, or plug a USB key into one of the PCs.

This is how viruses can get onto supposedly 'private' networks.

It takes a significant amount of effort from the IT guys to harden a system against this -- managed switches, Windows group policy. They're guaranteed to forget something.

The right thing to do is to disable the AV updates over the Internet, and use internal update servers (assuming that your AV solution supports it).

This means that you can validate the AV software on a test rig before it ends up on mission-critical production kit.

A disassemble of this virus

[WetCat]

0x100000 hlt

It should be part of the TCO

[RoLi]

Exactly. This is just part of the cost of running Windows. Any serious TCO-analysis should include the cost to purchase, install and update anti-virus software on Windows.

We had the same problem

[Xerxes1729]

The same thing happened at my school this weekend. At the beginning of the year, ITS required that anyone with a Windows machine install this Trend Micro program and give them the password to an administrator account*. By "securing" all the Windows machines, network outages would be prevented. Ironic, eh? Those of us who use other OSs, of course, were unaffected. And best of all, when they sent out a notice about fixing the problem, they didn't explain what had happened - we had to wait for one of the students who works there to tell us.

*They wanted me to give them my root password before they would turn on my network connection. I told the nice woman that if ITS expected me to trust them with my password, surely they would trust me with the password to one of the servers. She rolled her eyes and activated my connection.

Re:We had the same problem

[Ruprecht the Monkeyb]

The problem is with your IT department, then, not with Trend Micro. The TM client software can be deployed in a number of ways that don't require client interaction, much less giving them the admin password.

I use TM's enterprise stuff at a number of clients, and I've found it to be far more reliable than anything else. Most of my clients were using other products before I moved them over to TM, and nearly all of them were having problems with client interaction, updates not working, etc. And despite updating regularly, I've never been hit by any of the bugs reported.

Helpful, NOT...

[timbo1234]

This hosed all our work computers until the update appeared. 99% CPU usage on all of them. No helpfull info on the Trend site either. Cheers guys...

So dual CPU makes sense...

[stm2]

Some weeks ago there was a news here about using 1 CPU just to run housekeeping software (AV, anti-spyware, firewall, and so on) and let the other for user's taks.
It seems it is not so bad idea after all (at least, for Windows users).

that's the problem

[zogger]

They are starting to make the cars so complex that it drives the cost up significantly for initial purchase, and the repair costs get astronomical because it requires a specialist in most cases to *really* fix them, but they still only last a few years before they start to break down and become uneconomical for most people. Catch 22 now. Airplanes on the other hand have high initial cost, high repairs and maintenance costs, but are designed for decades of service, not just a few years. Where are the high tech safer cards with 20 year warranties? the cost has gone up tremendously compared to when I was a kid, yet they still seem to break as much and are much harder to work on for joe average.

No easy choices for joe consumer and land transportation. It's not like you can go buy a brand new cheap car that isn't infested with all sorts of electronic stuff that isn't really necessary. It may be useful, but it's not exactly necessary. You can get older cars of course, but even then it's a high cost to restore them and in a lot of cases they have to be modified to pass emissions, which lowers their actual practicality value by introducing complexity. More stuff bolted on = more stuff to break, simple as that. I mean, new cars now cost what houses used to cost not that long ago, and they still drop in value the same as they always did, drive off the lot, whoops, several thousand gone, then it goes downhill from there. It's a cost/benefits/practicality issue that's quite complex, I don't think it can be really stated that cars are that much more of a deal now just because of all the electronic controls, which are consistently the number #1 consumer complaint with cars and repairs, the electronic control systems nowadays. Blackbox voodoo stuff that even the dealer factory trained guys have a hard time dealing with once they develop bugs.

Info on Full-Disclosure list

[tsvk]

There was discussion on this on the Full-Disclosure mailing list when posters suspected that the 100% CPU usage on their computers was because of some new unknown virus.

A repesentative of Trend Micro Germany made a post to the thread where he explained the situation, apologized for it and offered pointers to their support database so that people could get the malfunctioning virus signatures uninstalled.

Re:Servers do not need real time virus protection.

[grasshoppa]

I am absolutely 100% sure that no damage can be done.

This would frighten me, were I your manager. People who are this sure of anything have been, in my experience, zealots for that OS or so egotistical that I don't want them making decisions.

Crap breaks, people make mistakes. I believe this to the core of my being, and I plan on it. Sure, I lose some performance, but given I can throw more hardware at that particular problem, I don't worry about it.

OS should provide protection

[booch]

The operating system should really prevent this type of problem. The whole purpose of the OS is to mediate access to resources such as CPU. So if one process is able to monopolize the CPU and prevent other processes from getting CPU time, then the OS has failed to do its job. (I'm not sure Linux would do a better job or not -- I've seen cases where it had similar problems.)

Trend Micro

[Fjornir]

So -- this is the same Trend Micro that decided to quarantine Cygwin a month or so back, took out our entire development team. A couple of years back Trend Micro decided to quarantine all emails containing the letter 'p'.

Since my office was so seriously affected by this problem, it would be great if people could post other embarassing Trend Micro stories too!

ANd these guys got a certification recently

[Madas]

link Checkmark labs recently gave out an award to the company for its spyware product. Spyware, as you know, slows down computers and makes them difficult to use. Oh the irony!!!

This was bound to happen, and it will happen again

[js9kv]

Two of my customers were hit with this at the same time on Friday around 4:50pm - the only good thing about it was that it hit at a time when many of the folks most affected by the bad update had gone for the weekend. They called, described the problem, and it hit almost completely in sync, all the machines that were running the latest XP with all the patches. We spent 3 hours that night troubleshooting and eventually figured out it was the AV software messing it up - and then about 20 minutes later on Trend Micro's site they had a "you gotta update from v594 to v596" to fix it. First off, lets face some reality here - it was only a matter of time before something this scale happened - AV software, if developed by a small group and not effectively tested, could be perhaps the least QA tested software on business PC's in the world today. Remember that response time is the major factor in AV protection - and getting your signatures out faster than the other guys, and faster than the virus spreads, is about the only success that these vendors know. For a long time now I've seen shoddy work from various AV vendors - Norton steals resources, Trend leaves stuff behind after an un-install and McAfee spams their own users after install. Thus far the only two that havn't bothered me that much are Zone Alarm and Grisoft's free AVG. For the last 2 years I've asked Trend Micro, Symantec and McAfee to add a single feature into their server-based email virus protection - and that is the smarts to know when to (and not to) respond to a message with a "this message contains a virus". Right now virus responses are a binary value - you either send them or you don't. Shouldn't the AV software be able to know from it's signature whether or not the senders email address is spoofed? Anyway, I digress. What it all boils down to is that AV vendors have a huge market penetration, and if some vendors aren't QA'ing their work (or if Microsoft is restricting updates by country) then it's inevitable that something nasty is going to be spread by the AV software. Also remember that it's not just the AV software - Microsoft's last round of updates seem to have broken more than just this.