Poison Attacks Against Machine Learning

mikejuk writes "Support Vector Machines (SVMs) are fairly simple but powerful machine learning systems. They learn from data and are usually trained before being deployed. SVMs are used in security to detect abnormal behavior such as fraud, credit card use anomalies and even to weed out spam. In many cases they need to continue to learn as they do the job and this raised the possibility of feeding it with data that causes it to make bad decisions. Three researchers have recently demonstrated how to do this with the minimum poisoned data to maximum effect. What they discovered is that their method was capable of having a surprisingly large impact on the performance of the SVMs tested. They also point out that it could be possible to direct the induced errors so as to produce particular types of error. For example, a spammer could send some poisoned data so as to evade detection for a while. AI based systems may be no more secure than dumb ones."

Comments

Propaganda

[mbone]

On this side of the human / AI line, we call this propaganda. It has historically proved very effective, specially if you can control all of the "training data."

Re:Propaganda

[betterunixthanunix]

Drugs with side effects trigger attachments. Caffeine is just as dangerous as Alcohol in that respect

Except that "attachments" are not dangerous. Coma and death are dangerous, brain damage is dangerous, liver damage is dangerous, and the typical doses of alcohol are frighteningly close to such adverse effects -- whereas the typical dose of caffeine is nowhere near that point.

Go to a coffee stand (or at work) and watch some people with their hands shaking so hard they can't hold the coffee in the cup.

Which may be scary, but is not a sign of any permanent damage to that person's mind or body. Caffeine withdrawal is tough, but it is not life threatening, and a person who is committed to it can get through the symptoms at home (maybe with the help of close friend) in less than a week. Alcohol withdrawal, on the other hand, can be so dangerous that it requires medical supervision.

That is a sign of a drug addiction beyond the persons ability to control.

Yet the drug abuse and dependence treatment programs that emerged from clinical psychology (read: science) are based on teaching people how to take control and avoid harmful behaviors.

Prescribed drugs can be abused but at least someone is trying to limit the effects

Really? A typical Adderall prescription (d,l-amphetamine salts) is for 10-20mg, two-three times per day, for a month. That is well above a lethal quantity, and a person could easily give themselves brain damage by taking a large fraction of their month's supply. People who abuse Adderall and related medicines (other amphetamines, Ritalin, etc.) can have psychotic episodes; see, for example, this recent NY Times article (sorry for paywall) about prescription stimulant abuse among high school and college students:

https://www.nytimes.com/2012/06/10/education/seeking-academic-edge-teenagers-abuse-stimulants.html?_r=1&hp

It's not just psychiatric drugs; prescription opiates are also readily abused, and people get high by using the prescribed amount of those drugs. Some pharmaceutical opiates are more potent than heroin, and abuse is an ever-present concern with those drugs; Rush Limbaugh abused prescription opiates:

http://www.cbsnews.com/2100-201_162-1561324.html

Here is the problem with the war on drugs: recreational drugs need not be any more dangerous than prescription drugs. Pharmaceutical methamphetamine is safer than "truck stop" methamphetamine, not because it is a different drug, but because the production is much better controlled. Many of the dangerous of recreational methamphetamine stem from the adulterants that are left over from poor production techniques.

So in a sense, I agree with you: we need better regulation. That means legalizing recreational drugs, and requiring that legal sources adhere to standardized and regulation production and distribution methods (I do not think anyone can argue that a 14 year old should be buying recreational drugs). When someone buys cocaine, they should not have to worry about what is mixed into the drug; when someone buys MDMA (ecstasy), they should not worry about having actually received methamphetamine mixed with caffeine (a well known trick on the black market). There will still be problems with abuse, but when someone visits their doctor, they should be able to tell their doctor what drugs they have been taking, and in what doses -- which is basically impossible if you are buying some mystery powder in an alley somewhere.

Not very practical

[ceoyoyo]

So if you know the algorithm and training data, and you can feed the system new data with manipulated labels then you can confuse it. It's a little early to panic about your spam filter. Hopefully everyone realizes that if you let the spammers tell your computer what is and is not spam, they can cause it to let their spam through.

Re:Try this on humans

[sg_oneill]

When you think about it, whats going on here is inducing mental illness in "thinking" machines.

We already know how to induce mental illness in humans. Religion and war.

Re:Try this on humans

[marcosdumay]

You mean propaganda and social pressure.

Religion and war are just consequences of those.