Trend Micro Bug Hits Several Important Computers

dmarx writes "The Japan Times reports that a bug in Trend Micro's antivirus software has caused the CPUs of several important computers, including those at East Japan Railway, to grind to a halt. A bug free version was released on noon Saturday." From the article: "Kyodo News experienced LAN access failure from around 8:20 a.m. to shortly before noon. The Asahi Shimbun and Yomiuri Shimbun also had trouble with their LANs at their Tokyo and Osaka bureaus, but the problems did not affect editing or printing of their evening editions."

Comments

Sounds familiar.

[bigtallmofo]

The buggy file slowed down computer performance substantially by making CPUs run at almost full capacity, the software company said.

Sounds like every interactively-scanning antivirus program I've ever installed. I wonder, when Microsoft releases server benchmarks, if they run them with antivirus software running in the background? I think this would give a 10%-15% edge to operating systems that don't require such measures of protection.

LPT$VPN.594?

[Anonymous Coward]

Was this the issue with LPT$VPN.594?

The large bookseller I work for (think "Stables and Lords") got hit with that on Friday. All the XP machines (basically, the Manager's computers in the stores) and even a few of the XP computers in the Helpdesk (where I work) would lock up and freeze during boot.

Deleting the offending file fixed the issue.

Re:The problem with AV

[mikeumass]

Less market share. Windows is a much more apetizing market. Especially since most users wouldn't know if they had a trojan in the first place. How many people actually renew thier subscriptions with Norton or NA?

Why AntiVirus?

[MindStalker]

What I want to know is why do the computers controlling the train system in Japan need antivirus. Are they attached to the internet? Do they have disk drives? This system should have neither, I can understand the reason for a seperated system to be connected to the net for reporting train schedules and problems. But connecting a control system like that? Running it on windows? Silly. Thats worse than having antivirus on an ATM.

Re:Why AntiVirus?

[guy-in-corner]

Even if a computer system isn't connected to the Internet, you can guarantee that -- if it's connected to any kind of network infrastructure -- some idiot is going to jack their laptop into it, or plug a USB key into one of the PCs.

This is how viruses can get onto supposedly 'private' networks.

It takes a significant amount of effort from the IT guys to harden a system against this -- managed switches, Windows group policy. They're guaranteed to forget something.

The right thing to do is to disable the AV updates over the Internet, and use internal update servers (assuming that your AV solution supports it).

This means that you can validate the AV software on a test rig before it ends up on mission-critical production kit.

Re:Who's to blame

[Patrik_AKA_RedX]

Software design is still a pretty young field of construction. Building construction has had more than 2 millenia to develop, while software design had about century (give or take a decade). In the early days (read: centuries) buildings were designed by rules of thumb. Only the last few centuries the real science of contruction was developed. (The metalurgical properties of steel wasn't researched until after WW2 when they figured out that welded ships couldn't handle the extreme cold of northern seas very well) In software design we're at the point where we're trying to come up with the science, but are still mostly using rules of thumb.

Given time software will reach a point where it's about as reliable as concrete buildings, but in the mean time we'll be stuck with the many kinds of blue screens.

Re:Who's to blame

[kfg]

RyanFenton, posting in the computerized cars for traffic control thread:

I'd MUCH rather trust a reasonably engineered computerized system than the thousands of other drivers around me on my way about town.

I didn't post there, but my very first reaction on reading was:

"And just where the hell do propose to find one of those?"

This story illustrates my reaction. Imagine thousands of cars around you on your way about town that have suddenly lost all control.

Without the introduction of computers cars are actually not that complicated. They consist of a relatively few number of parts mechanically linked in such a way that any child can intuitively grasp their operation. You can teach yourself a fair amount of auto mechanics through entirely empirical methods, just sitting down with the device, taking it apart, putting it back togehter, and grasping how the whole thing works by such observation.

Nobody's going to write a virus checker that way, or a car control system. The computer is too complicated, consisting of billions of invisible "parts" whose operation is entirely abstracted from their function.

To the extent that cars are complicated these days, to the further extent that even formally trained mechanics cannot figure out what's wrong with them without plugging them into a computer, it is because they now contain. . .computers.

So refering to cars as an example of something that's complicated but reliable is not factual ( and I myself have found myself sitting by the side of the road with a mechanically sound car that refused to run because a control chip died), but also begs the question.

KFG

It should be part of the TCO

[RoLi]

Exactly. This is just part of the cost of running Windows. Any serious TCO-analysis should include the cost to purchase, install and update anti-virus software on Windows.

So dual CPU makes sense...

[stm2]

Some weeks ago there was a news here about using 1 CPU just to run housekeeping software (AV, anti-spyware, firewall, and so on) and let the other for user's taks.
It seems it is not so bad idea after all (at least, for Windows users).

Re:The problem with AV

[Deffexor]

I actually ran into this problem at a customer's site this weekend. They had Trend Micro AV and the computer was utterly crippled. It was like it had some utterly malicious virus on it gobbling up all the cpu time.

Using SysInternal's Process Explorer, I was ultimately able to see that a module (running as a part of the "system" process) called "TmXPflt.sys" was running 4 simultaneous threads each using about 25% of the CPU. Since the "system" process is given higher priority than all other processes, the system naturally slowed to a crawl.

I rebooted into safe mode and renamed this file and restarted. The system behaved like normal again. The file said it was a Trend Micro "XP Post Filter" (mail filter?) - After all that, I thought that it was particularly weird that I hadn't read about some problem from Trend Micro on a major news outlet (like Slashdot) :-)

OS should provide protection

[booch]

The operating system should really prevent this type of problem. The whole purpose of the OS is to mediate access to resources such as CPU. So if one process is able to monopolize the CPU and prevent other processes from getting CPU time, then the OS has failed to do its job. (I'm not sure Linux would do a better job or not -- I've seen cases where it had similar problems.)

Re:Who's to blame

[greed]

The surge results in a voltage drop on the +12 rail of the "good battery" car. It's trying to bring the dead battery up to the exact same voltage, within the current limiting effect of the jumper cables. Lead-acid batteries have a very low internal resistance, so they won't slow things down much. (And that's how you get 800 "cold cranking amps" out of 'em.) A dead battery will be between 11.8 and 12.2 volts, and the good system should be up around 13.2 to 14 or so, depending on the regulator.

Many computers need to have /RESET held low for a few would-be clock cycles after power-up, to allow the power rails to stabilize and the master oscillator to start. Usually this is done by a capacitor which slowly (comparatively) charges up to supply volatage; when it crosses a certain voltage, it releases /RESET (they're usually active-low), and the CPU can start.

All well and good...

If you've got a situation where the power rail drops suddenly, the capacitor on /RESET starts to discharge to the power rail. Enough, and it activates the /RESET line on the CPU. Even though the power drop wasn't enough to wipe out the CPU, it was able to trigger the power-on-RESET circuit. (The fix is to put a diode in the computer's power supply connection, so that the computer's power supply capacitors never try to bring the +12 rail back into spec.)

Another fun thing that can happen, though probably not in automotive circuits, is GND and Vcc inversion.

This used to happen a lot on Amigas with defective monitors; you'd get a high-voltage discharge in the monitor to the GND line, which would momentarily bring GND over Vcc, triggering a /RESET. The fix there is to separate shield ground from signal ground; or you could just go bankrupt.

Given the number of modern cars which, apparently, tell you not to jump-start, there is an awful lot lacking in modern automotive design. It's not hard to cope with a jump-start, you just have to not cut all those corners.

(My 1998 Subaru has no such warning; I've only heard about that warning from GM owners--I've never seen it myself.)

We too use Trend Micro...

[Anonymous Coward]

... OfficeScan and ServerProtect on over 700 machines and did not experience any problems over the weekend. We used to be a McAfee shop and ditched them after two years of problems and then the company failing to honor our support contract with them. We tested Symantec's enterprise virus product and could not get the evals to do the "push" install and run correctly even after a couple hours on the phone with Symantec's support. Turned out that we'd have to manually touch each and every of the 700 desktop machine with a crew of support techs to clear out the old McAfee installation and reboot each one at least 3, possibly 4 times to get the Symantec product installed. Furthermore, the Symantec/Norton AV product felt like it just subtracted 200 MHz off the CPU speed of each machine once it was installed. We were not pleased with it at all. The Trend Micro eval install just simply worked right the frict time. The push installer removed the old McAfee and installed OfficeScan automatically with only a single reboot at the end of the installation. Of the 700 desktops on out network, we had to manually touch maybe 50 of them due to odd problems. Trend has been running fine for us for over 2 years now.

Re:We had the same problem

[Anonymous Coward]

Sorry I think you don't understand the situation the pattern file caused. Officescan (the corporate version of the software that was effected) has a remote admin/update/rollback feature built in. Normally this would be the way that issues are handled. The issue though was that the trend product runs at a very low level on the system and every-time it hit DLL and a few other file types it brought the kernel/system utilization up to 100%. The problem is that the real time scanner is almost always hitting DLL and those other types of files. When this was happening you could not even telnet to port 135 on the problem host. The only solution that seemed to work was either their emergency update tool (mind you they released this like 36+ hours after the networks were down) which bashed the host with commands to fix the issue until it responded (but needed the admin password), rebooting the machine in safe mode, or writing your own script/program to do the same thing (what I ended up doing).

The frustrating thing about this situation was that Trend would not even tell me if it was a bug or if their update server was hacked and this was a malicious code injection Friday (I spent 9 hours on the phone with them). They just gave me the runaround that the support managers and project managers were all on a conference call trying to work through the PR announcement. Very poor form. When the announcement was finally made it plays down the fact that most of their large install base clients use a product called control center that checks and pushes out pat updated in minute intervals. So the "only released for 1.5 hours" junk in the PR was meaningless -- all of their XP install base at large corporations were down because of this. One more frustrating part of the night was the 30 minutes or so it took me to discover that Trend was responsible for the issue -- I then called support and the person on the initial call made believe that he had not heard of an issue until I stated that I knew it was the 594 pattern file -- then he went into the rehearsed line "I can't comment on this issue as we are still investigating it and coming up with a press release." argh.

ANd these guys got a certification recently

[Madas]

link Checkmark labs recently gave out an award to the company for its spyware product. Spyware, as you know, slows down computers and makes them difficult to use. Oh the irony!!!