Erasing Objects From Video In Real Time

Smoothly interpolating away objects in still pictures is impressive enough, but reader geoffbrecker writes with a stunning demonstration from Germany's Technical University of Ilmenau of on-the-fly erasure of selected objects in video. Quoting: "The effect is achieved by an image synthesizer that reduces the image quality, removes the object, and then increases the image quality back up. This all happens within 40 milliseconds, fast enough that the viewer doesn't notice any delay."

Comments

Cool, but probably still has a ways to go.

[suso]

Pretty good, but take note that all the examples where objects sitting on pretty flat colored backgrounds. I'd like to see what happens when you try to remove an object in a complex environment. Like removing a single person standing in a crowd.

Re:Cool, but probably still has a ways to go.

[DrXym]

Pretty good, but take note that all the examples where objects sitting on pretty flat colored backgrounds. I'd like to see what happens when you try to remove an object in a complex environment. Like removing a single person standing in a crowd.

There were some examples of that in the clip if you watched closely. The removal drain on a pebbly asphalt caused a weird swirly pattern to occur as the camera moved. I expect the same would be true for live attempts at the same. It probably works best on static things on solid backgrounds that nobody is likely to be walking over. I expect it will be used a lot in live broadcasting, especially sports events.

Re:Cool, but probably still has a ways to go.

[stewbacca]

All of their samples (except the brick one) use solid/high contrast surfaces that are somewhat evenly lit. Still kind of cool though, but they should have waited until it's more mature to impress us old timer motion graphics guys.

Re:Cool, but probably still has a ways to go.

[RandCraw]

Right. I'm pretty sure the underlying technology is based on Seam Carving , where a continuous background region is collapsed 'seamlessly' (or an object within such a region is removed). This doesn't work so well when the background is discontinuous, so it's not going to remove logos from clothing. It's also not going to work well on live video, since the object to be removed needs to be identified manually before the excision can occur. But it works nicely on prerecorded media. I'm impressed at how well it works on a brick background.

Journalistic Integrity

[Defenestrar]

Great - it'll start off by making eyesore real estate disappear from "live coverage," then be required as a precondition for live celebrity interviews (not just makeup to cover that acne), moving on to inconvenient points to the story that would take too much time and effort to explain, then images which might "disturb the children" (number of student bodies in Tienanmen Square?), and finally develop to ubiquitous studio-in-a-cameras such that we'll have little assurance of whether live coverage is fact or fiction.

Of course that's just pessimism speaking. Really I'm looking forward to watching live reports without those obnoxious people waving at their mothers, or holding up witty slogans about taxation.

Re:I thought what I'd do is...

[mister_playboy]

That phrase was familiar to me, but I wasn't sure where I had seen it... now I remember:

https://secure.wikimedia.org/wikipedia/en/wiki/Laughing_Man_(Ghost_in_the_Shell)

Re:I thought what I'd do is...

[EdZ]

Not really. There are plenty of webcams that come with free software that can overlay an image (including the requisite spinning-text-around-face logo of the Laughing Man) over a tracked face in real time, but this software instead edits out a tracked area using surrounding data. I wish they gave more explanation, or any explanation at all, rather than the nebulous magical 'increase the image quality back up'.

The Running Man (Film version)

[Shivetya]

comes to life then?

Its bad enough people believe lines said by comedians are the actual lines of some high profile people, how can we hope that people will care enough to know if the video they are seeing is not edited? Hollywood doesn't need the tech to make movies, maybe to "fix" reality shows, but I figure politics is where the mileage comes in.

Suspected limitations

[mattaw]

Obvious limitations from the demo:

1) Objects must be sitting on a consistent(ish) surface with a low rate of change compared to the object. Desk, Chair, Bathroom, Wall, Hubcap, etc.

2) It doesn't handle strong shadows (or they are not showing us it doing so).

3) It makes the greatest amount of mistakes with the shadows anyway.

Please add anything I missed to future posts.

I would like to see it erase a boat from a choppy sea where there are 5-7 waves for the length of the boat as I expect that to be a pathological case. I would also like to see it erase a discolouration rather than a very different object to see its behaviour. Cool technology though!

Re:Perfect Application

[John Hasler]

A simpler version of this has already been used to edit billboards visible in broadcasts of baseball games.

Re:Perfect Application

[gstoddart]

We need this built into our televisions to automagically remove those network logo "bugs" and other crap they have started putting on the screen during the shows.

First off, I don't think we'll get control over this on our TVs. The networks aren't gonna let us delete their "bugs".

I'm actually more concerned over something like Running Man where you can't trust the news reports you see because someone selectively tweaked the image to hide/alter the bits they don't want you to see.

Now, of course, the technology isn't evil ... it will be humans doing that. But, you can imagine government run media stripping out protesters or burning cars to tell everybody that everything is just sunshine and bunnies.

Re:Do we still believe what we see?

[houghi]

The 'Live on scene' is pretty rare already. Almost everything is taped and edited. And even now many people believe what they see as they will have only one (if that) source of information. How many people will actually look at what others have to say? http://aljazeera.com/ as an alternative? Nah, because that is propaganda from the enemy. Better just watch Fox News.

People do not want to be informed. They want to be entertained.

Cheep and easy counter measures

[one cup of coffee]

If you wish to be seen, but are being spliced out. You could wear several different colored t-shirts over each other or something similar and then take them off to trick the camera at least temporarily. If you want people to see something that is being blocked out, you would have to probably spray them or it with some kind of colorant, or a bright flash of light might also do the trick, maybe some kind of a portable strobe light. This is just off the top of my head.

Somehow l feel like like I shouldn't be giving away these ideas, maybe my tinfoil hat is just making my head itchy...

Re:Perfect Application

[sjames]

Or stripping out the thousands of peaceful middle aged protesters and the hundreds of uninvolved pedestrians being tear gassed to show only the one or two violent people who actually have nothing to do with the protest.

Next up, witnesses will disappear from police video to discredit them in court.

This is just step one.

[HiThere]

In addition to dealing with reflections, which I consider just a part of polishing step one, step two will use the position of something in the video as an anchor and substitute the image of something else.

How far off do you think *that* is? I give it two years to the the lab demo with problems.

Prisoner Zero has escaped...

[rlseaman]

Invisibility is an ancient notion and tampering with video as old as the Lumière brothers. What is new here is the trend toward placing these capabilities closer and closer to the camera. Combine such effects with the face detection algorithm that is already in your phone's camera and the original picture can remove or replace individuals from the scene of the crime. "Ground truth" will be ever more difficult to establish.